Switching the data access from the Data Centre to the Data Store for faster delivery times.
In October 2020 EUMETSAT introduced the Data Store, an online repository for delivering satellite data to the users, with no waiting times. The basic data catalogue offered back then for the the pilot phase has been growing with more and more EUMETSAT data and meteorological, climate and ocean products, to the point that the EUMETSAT Data Store and Data Centre are now offering in parallel a very significant part of the EUMETSAT data catalogue. This has allowed the majority of our users to move their data retrieval pipelines from the Data Centre Ordering to the Data Store & Data Tailor data access services.
Multiple users have attended the regular training activities organised by EUMETSAT on the new data services and hundreds of emails have been exchanged to provide support to specific use cases. It is estimated that the Data Store & Data Tailor are able to support almost 90% of the data downloads registered in EUMETSAT's Data Centre. Finally, time has come for the Data Centre to start slowing down its activity as a data access service in favour of the consolidated Data Store. The Data Centre - EUMETSAT's long term satellite data archive - will focus on its key task, which is securely archiving and preserving the data from the many existing and new satellite missions of EUMETSAT.
The transition from the Data Centre to the Data Store is organised in a staged approach, involving reductions on the catalogue, the customisation offer and the service level of the Data Centre, as follows:
Reduction of the Data Centre products customisation offer
as of 10th of July 2023: The media deliveries are discontinued and the following customisations are no longer supported for the MSG and Metop orders. Users are encouraged to customise their data deliveries on the Data Tailor:
Region of Interest / Geographical Area subsetting
Channel / Band / Variable subsetting
Reformatting to image formats (PNG, JPEG, TIFF)
Reformatting to HDF5
Reformatting to McIdas AREA files - no longer supported
Reduction of the Data Centre catalogue
as of 17th of April 2023: access to historical data (sensing time before 28/04/2022) for the following Sentinel-6 collections is removed from the Data Centre, in favour of the reprocessed datasets (with higher quality)* that are available on the Data Store.
Product Name | Product Type | Data Centre | Data Store |
|---|---|---|---|
Poseidon-4 Altimetry Level 1A High Resolution | P4_1A_HR_____ | ||
Poseidon-4 Altimetry Level 1B Low Resolution | P4_1B_LR_____ | ||
Poseidon-4 Altimetry Level 1B High Resolution | P4_1B_HR_____ | ||
Poseidon-4 Altimetry Level 2 Low Resolution | P4_2__LR_____ | ||
Poseidon-4 Altimetry Level 2 High Resolution | P4_2__HR_____ | ||
Climate-quality Advanced Microwave Radiometer Level 2 Products | MW_2__AMR____ |
|
planned for 2023 Q4: access to MFG, MSG, Metop, Sentinel-3 and Sentinel-6 mission data is removed from the Data Centre for those collections which are available on the Data Store, where the "best quality data"* is available for each orbit / slot.
* Please note that due to storage limit considerations, unlike the Data Centre (tape based archive), the Data Store (online disk based repository) in general offers only the best quality data available for each product type / orbit. This means that the reprocessed datasets are offered by default and gaps are filled with NTC data, or STC/NRT when no NTC is available. The NRT data is typically only available for the most recent orbits, where no NTC data has been produced yet. Further details - on which collections are offering which product types - are available on the Data Store User Knowledge base:
Reduction of the Data Centre service levels
planned for 2023 Q4: users will be able to submit a maximum of 5 orders (reduced from 10) in parallel. New orders won´t be accepted until previous ones are delivered.
planned for 2023 Q4: order delivery times might increase due to the reduction of the computational resources dedicated to the processing of Data Centre user orders.
Users unduly impacted by these changes, in need of more time to adapt their data retrieval chains, are encouraged to contact the EUMETSAT Helpdesk. For more information on our training, including understanding the transition from Data Centre to Data Store, visit our page
Improved and Simplified Data Access
Users will benefit from the following new features when accessing EUMETSAT data from the Data Store & Data Tailor:
Faster access: downloading data from the Data Store is faster than ordering it from the Data Centre where the data has to be retrieved from tape in most cases.
Programmatic access: ready to download and run from command line, EUMDAC (EUMETSAT´s Data Access Client) lets users automate their data retrievals from the Data Store with simple commands. Also available as a Python library, it comes with methods and helpers to use the Data Store & Data Tailor APIs.
Data Tailoring: formatting and processing the data is possible prior it´s download thanks to the integration with the Data Tailor Web Service. This is also possible after the data downloads for users who prefer to run heavy customisations on their local environment (see the Data Tailor standalone software)
Connected to EUMETView: users are now able to visualize the images on EUMETView prior their download on Data Store.
More information about the Data Store is available here.
MEMBER STATES
