What is updated when publishing a remote file?

Samuel Dauzon Updated by Samuel Dauzon

When a remote dataset is updated using an automated publishing schedule, the behavior is different depending on the source file's type and location.

This article describes what happens when such a file is stored on an FTP server, is an ArcGIS file, or is available via HTTPS.

Files stored on FTP

The FTP source is a folder

  • If an existing file is modified, we will reprocess the entire file.
  • If a file is added (without modifying the other files), only this file will be processed. In other words, the update is incremental.
  • If there are no changes in the folder, the dataset is not reprocessed on publication.

The FTP source is a file

  • If there is a change in the file, the entire file will be reprocessed on publication (because the FTP source is a file and not a folder, it is not possible to use incremental mode).
  • If no change, the dataset is not reprocessed on publication.

ArcGIS files

In the case of an ArcGIS source, we are unable to detect whether the file has been modified since the last publication. As such, the entire file will be systematically be processed and published.

Files available via HTTPS

Similar to the FTP case, we need to detect a change in the file's modification date. This can be checked via the "Etag" and "Last-modified" headers, and the "If-None-Match" and "If-Modified-Since" conditional requests.

  • If the "Last-modified" header returned by the HTTP source changes with each request, for example, the detection will not work, resulting in the entire file being reprocessed.
  • The "If-Modified-Since" conditional request must provide a response with a HTTP 200 status, only if the resource has been modified since the given date. If the resource has not been modified since then, the response must have an HTTP 304 status with no content.
  • With "If-None-Match", the server returns the requested resource, with a status of 200, only if it does not have an ETag corresponding to those provided. For the other methods, the request will only be processed if the ETag of any existing resource does not match any of the listed values. When the condition fails for GET and HEAD methods, the server returns an HTTP 304 (Not Modified) status code.
    For more information: https://developer.mozilla.org/fr/docs/Web/HTTP/Headers/If-None-Match

How did we do?

Promote mobility data thanks to GTFS and other formats

Table of Contents

Contact

Powered by HelpDocs (opens in a new tab)