esget

Don't try to use this! It is unfinished and has no support.

To mirror parts of the Earth System Grid Federation (ESGF) archive check out Synda (https://github.com/Prodiguer/synda) That is also what ETHZ-IAC uses now for all its mirroring-needs.

esget

Earth System Grid gETer

esget is a tool to mirror part of the Earth System Grid Federation (ESGF) distributed database. The output of many climate-related simulation projects is distributed via ESGF, e.g. CORDEX, CMIP5, GeoMIP, or PMIP3.

esget is designed to deal with very large datasets. Its features include:

• tracking of local files
• accepts arbitrary dataset-definitions
• uses multiprocessing for ESGF-queries and downloading
• uses an arbitrary large number of hosts for parallel downloading
• state is kept in a SQLITE-database
• mostly Python