OLCI: How can I retrieve a complete archive of Sentinel-3 OLCI data?

Retrieving an archive of OLCI data

Retrieving an archive of OLCI data

Please note that EUMETSAT is currently transitioning the provision of Sentinel-3 data to our new data services. More information is available on our website.

The archive will only be fully consistent at the time a reprocessing is completed and the processing baseline used matches the current operational baseline. However, reprocessings are conducted when a major change happens in the processor, so it is typically fine to use data from multiple baselines if there hasn’t been a major change. Users working on applications that require long-term consistent processing for trend analysis, should pay additional attention to changes associated with processing.

Carefully consider when to gather an archive - it is often worth waiting for the next reprocessing before conducting a large piece of analysis or downstream product generation, to minimise the need for updating your archive. Please contact ops@eumetsat.int for more information.

To create a complete archive/time series of data (for a region or global coverage) you will need to access data from several sources to obtain the most up-to-date collection possible, this includes:

  1. Data from the current rolling archive of data from the current operational baseline:

    1. This is the latest version of the processor and would be the ideal version you would want to have been used for the full archive you collect.

    2. This data is available from the EUMETSAT Data Store (see our Data Store knowledge base and EUMETSAT Gitlab for Command line and Python tools available for accessing this).

  2. Data from the current operational baseline that is older than 01/01/2021, but not yet reprocessed

    1. If more than a year has passed without a reprocessing, or if the reprocessing doesn’t cover the full mission, you will need data from the from the data centre.

    2. This data must be ordered.

  3. Data that has been reprocessed.

    1. Data are periodically reprocessed. You will want to use the latest reprocessed versions of the data of interest to form your archive/time series beyond what is available on the current rolling archive under the latest processing baseline.

    2. This data is available from the EUMETSAT Data Store and the relevant collectiionscan be accessed via GUI and API.

  4. Data from past operational baselines that are older than 01/01/2021 and not yet reprocessed.

    1. This data is available from the data centre and must be ordered.

To see the history of baseline changes, please visit the OLCI processing baselines page.

Current status

  • For Level-1 data please follow the guidance above to obtain a current best quality archive from Data Store, and Data Centre.

  • For Level-2 data the full archive of OLCI level-2 data at both full and reduced resolution is now available on the Data Store, with a consistent processing under baseline collection 3. See further details on this below and on the OLCI processing baselines page.

 

Accessing a time series of OLCI L2 data from the Data Store

  • OLCI L2 full resolution reprocessed products are held in collection EO:EUM:DAT:0556

  • OLCI L2 full resolution operational products are held in collection EO:EUM:DAT:0407

There are further collections for the L2 reduced resolution products, and Level-1 (for which there are operational, not reprocessed collections at present).

To acquire the best quality time series across a given region, it is necessary to assess the contents of both reprocessed and operational collections, and determine where to ‘switch’ between the two (e.g. at the end of the reprocessed archive). This can be done using the eumdac CLI or Python library (EUMDAC)

Using the eumdac CLI

To determine the last available product in the reprocessing collection, we can run the following command:

1 eumdac search -c EO:EUM:DAT:0556 --limit 1 --bbox 6.76 37.47 12.11 43.31

 *Note that a bounding box has been set to a region in the Mediterranean as an example to reduce the search results, but the same line can be run without specifying a bbox, which would search globally.

 

1 eumdac search -c EO:EUM:DAT:0407 --limit 1000000 --bbox 6.76 37.47 12.11 43.31 -s 2021-04-28T09:53:03 > OPE.txt

 

Setting --limit 1 constrains the search to only return one result. As the Data Store always returns products in data order of latest to earliest, this will give only the most recent product, which is as below;

S3A_OL_2_WFR____20210428T095003_20210428T095303_20220308T090710_0180_071_136______MAR_R_NT_003.SEN3

 

Now we have the date of the last product in the reprocessing, we can use this to specify our search limits in the operational collection.

To retrieve all reprocessed products that match the search criteria, the following command can be run:

 

1 eumdac search -c EO:EUM:DAT:0556 --limit 10000 --bbox 6.76 37.47 12.11 43.31 > REP.txt

 **Note that limit is set to its maximum. If you are retrieving a very large portion of the archive you may need to split your temporal/regional search ranges in to multiple calls.

 

To retrieve all operational products that match the search criteria, the following command can be run:

1 eumdac search -c EO:EUM:DAT:0407 --limit 1000000 --bbox 6.76 37.47 12.11 43.31 -s 2021-04-28T09:53:03 > OPE.txt

*Note that the start date of the search is set to correspond to the end of the reprocessing.

 

In the above cases, the product list is added to two text files “REP.txt” and “OPE.txt”. The products listed in these files can then be downloaded using:

1 2 3 eumdac download -c EO:EUM:DAT:0556 -p @REP.txt eumdac download -c EO:EUM:DAT:0407 -p @OPE.txt

If required, the NRT files can be removed from the OPE.txt file prior to downloading. Future versions of eumdac will support filtering by timeliness at the searching stage.

Using the eumdac Python library

You can find an example of using the EUMDAC Python library to access operational and reprocessed collections for OLCI in the EUMETSAT Gitlab.