{"id":18819038,"url":"https://github.com/geoscienceaustralia/fetch","last_synced_at":"2025-04-13T23:32:57.136Z","repository":{"id":53090198,"uuid":"47670683","full_name":"GeoscienceAustralia/fetch","owner":"GeoscienceAustralia","description":"Ancillary fetch daemon","archived":false,"fork":false,"pushed_at":"2024-12-17T00:00:54.000Z","size":429,"stargazers_count":3,"open_issues_count":2,"forks_count":8,"subscribers_count":10,"default_branch":"develop","last_synced_at":"2025-03-27T13:46:13.492Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GeoscienceAustralia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-12-09T05:27:36.000Z","updated_at":"2023-02-10T16:59:17.000Z","dependencies_parsed_at":"2023-01-24T08:15:57.555Z","dependency_job_id":null,"html_url":"https://github.com/GeoscienceAustralia/fetch","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Ffetch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Ffetch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Ffetch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Ffetch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GeoscienceAustralia","download_url":"https://codeload.github.com/GeoscienceAustralia/fetch/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248797027,"owners_count":21163068,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T00:19:56.139Z","updated_at":"2025-04-13T23:32:56.613Z","avatar_url":"https://github.com/GeoscienceAustralia.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ancillary fetch daemon [![Build Status](https://travis-ci.org/GeoscienceAustralia/fetch.svg?branch=develop)](https://travis-ci.org/GeoscienceAustralia/fetch)\n\nDownload ancillary data automatically.\n\nIt is run with one argument: a config file location. This will run endlessly,\ndownloading according to schedules in the config file:\n\n    fetch-service config.yaml\n\n(and is typically run from an init script)\n\nAdditionally, you can run a single rule from the config file, ignoring any\nschedules. It will run the rule once immediately and exit:\n\n    fetch-now config.yaml LS7_CPF\n\nFetch uses file locks in its work directory to ensure that only one instance of each rule is running at a time. You\ncan safely use `fetch-now` while a service is running without risking multiple instances\ninterfering.\n\n### Development\n\nIf not installed to the system, such as during development, they can\nalternatively be run directly from modules:\n\nService:\n\n    python -m fetch.scripts.service config.yaml\n\nNow:\n\n    python -m fetch.scripts.now config.yaml LS7_CPF\n\nDevelopers should refer to the ``docs`` directory and the [README](./docs/README.md) file therein.\n\n## Configuration file\n\nConfiguration files are loaded in [YAML](https://en.wikipedia.org/wiki/YAML) format\n(essentially nested lists and dictionaries: YAML is a superset of JSON).\n\nAn example configuration file:\n\n    # Work directory:\n    directory: /data/ancillary-fetch\n\n    # Notification settings (for errors):\n    notify:\n      email: ['jeremy.hooke@ga.gov.au']\n\n    # Download rules:\n    rules:\n\n      Modis utcpole-leapsec:\n        schedule: '0 7 * * mon'\n        source: !http-files\n          urls:\n          - http://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/utcpole.dat\n          - http://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/leapsec.dat\n          target_dir: /eoancillarydata/sensor-specific/MODIS/\n\n      LS8 CPF:\n        schedule: '*/30 * 1 1,4,7,10 *'\n        source: !rss\n          url: http://landsat.usgs.gov/cpf.rss\n          target_dir: /eoancillarydata/sensor-specific/LANDSAT8/CalibrationParameterFile\n\n`directory:` specifies the work directory for the daemon lock and log files.\n\n`notify:` allows configuration of error notification.\n\nThe third option contains download rules (`rules:`).\n\n- In this case there are two rules specified: one http download of utcpole/leapsec files,\nand an RSS feed download of CPF files.\n\n- Rules are prefixed by a name: in the above example they are named `Modis utcpole-leapsec` and\n`LS8 CPF`.\n\n- Names are used as an ID for the rule.\n\n- The `source:` property is our download source for the rule. It is tagged with a YAML type (`!rss` or `!http-files` in this example)\nto specify the type of downloader.\n\n- Each downloader has properties: Usually the URL to download from, and a target directory to put the files.\n\n- `schedule:` uses standard cron syntax for the download schedule.\n\n### Download sources\n\nTypes of downloaders:\n\n#### !http-files\n\nFetch static HTTP URLs.\n\nThis is useful for unchanging URLs that need to be repeatedly updated.\n\nExample:\n\n    source: !http-files\n      urls:\n      - http://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/utcpole.dat\n      - http://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modis/leapsec.dat\n      target_dir: /eoancillarydata/sensor-specific/MODIS/\n\nAll http rules have a `connection_timeout` option, defaulting to 100 (seconds).\n\n#### !ftp-files\n\nLike http-files, but for FTP.\n\n    source: !ftp-files\n      hostname: is.sci.gsfc.nasa.gov\n      paths:\n      - /ancillary/ephemeris/tle/drl.tle\n      - /ancillary/ephemeris/tle/norad.tle\n      target_dir: /eoancillarydata/sensor-specific/MODIS/tle\n\n\n#### !http-directory\n\nFetch files from a HTTP listing page.\n\nA ([regexp](https://docs.python.org/2/howto/regex.html#regex-howto)) pattern can be specified to only download certain filenames.\n\n    source: !http-directory\n        url: http://rhe-neo-dev03/ancillary/gdas\n        # Download only files beginning with 'gdas'\n        name_pattern: gdas.*\n        target_dir: '/tmp/gdas-files'\n\n#### !ftp-directory\n\nLike http-directory, but for FTP\n\n    source: !ftp-directory\n      hostname: ftp.cdc.noaa.gov\n      source_dir: /Datasets/ncep.reanalysis/surface\n      # Match filesnames such as \"pr_wtr.eatm.2014.nc\"\n      name_pattern: pr_wtr.eatm.[0-9]{4}.nc\n      target_dir: /eoancillarydata/water_vapour/source\n\n#### !rss\n\nDownload files from an RSS feed.\n\n    source: !rss\n      url: http://landsat.usgs.gov/cpf.rss\n      target_dir: /eoancillarydata/sensor-specific/LANDSAT8/CalibrationParameterFile\n\n#### !ecmwf-api\n\nFetch now allows access to the batch data servers of the European Centre for Medium-term Weather Forecasts. The data archive is accessed via\nthe [Python ECMWF API](https://software.ecmwf.int/wiki/display/WEBAPI/Accessing+ECMWF+data+servers+in+batch).\n\nThe ECMWF API required properties to be specfied as follows:\n\n    source: !ecmwf-api\n        cls: ei\n        dataset: interim\n        date: 2005-01-03/to/2005-01-05\n        area: 0/100/-50/160\n        expver: 1\n        grid: 0.125/0.125\n        levtype: sfc\n        param: 134.128\n        stream: oper\n        time: 00:00:00\n        step: 0\n        typ: an\n        target: /home/547/smr547/ecmwf_data/sp_20050103_to_20050105.grib\n        override_existing: True\n\nThe keys (dataset, date, area, etc) are [MARS keywords](https://software.ecmwf.int/wiki/display/UDOC/MARS+keywords)\n used to specify various aspects of the data retrieval. Please note that the ``class`` and\n``type`` keywords have different spelling (``cls`` and ``typ``) to avoid Python compiler name clashes.\n\nRequest parameter are complex. ECMWF recommend using the ``View Request Parameters`` feature as you get familiar with\nthe [avaiable ECMWF data sets](http://apps.ecmwf.int/datasets/). This\nwill assist you in preparing error-free requests.\n\nThe ``!ecmwf-api`` datasource supports [Transformers](#transformers) and the ``override_existing`` option (defaults to ``False``).\n``!ecmwf-api`` datasources can also be used with the [!date-range](#!date-range) datasource.\n\n### Transformers\n\nTransformers allow for dynamic folder and file names (both sources and destinations).\n\nDownloaders supporting them have a `filename-transform:` property.\n\n#### !date-pattern\n\nPut the current date/time in the filename.\n\nThis takes a [format](https://docs.python.org/2/library/string.html#formatstrings) string with properties 'year', 'month', 'day', 'julday' (Julian day) and 'filename' (the original filename)\n\nExample of an FTP download\n\n    source: !ftp-files\n      hostname: is.sci.gsfc.nasa.gov\n      paths:\n      - /ancillary/ephemeris/tle/noaa/noaa.tle\n      target_dir: /eoancillarydata/sensor-specific/NOAA/tle\n      # Prepend the current date to the output filename (eg. '20141024.noaa.tle')\n      filename_transform: !date-pattern '{year}{month}{day}.{filename}'\n\n\n#### !regexp-extract\n\nExtract fields from a filename, and use them in the destination directory.\n\n(This requires knowledge of [regular expressions](https://docs.python.org/2/howto/regex.html#regex-howto) including named groups)\n\nSupply a regexp pattern with named groups. Those group names can then be used in the target folder name.\n\nIn this example, we have a pattern with three regexp groups: 'year', 'month' and 'day'. We use\nyear and month in the `target_dir`.\n\n    LS8 BPF:\n    schedule: '*/15 * * * *'\n    source: !rss\n      url: http://landsat.usgs.gov/bpf.rss\n      # Extract year and month from filenames using regexp groups\n      #    Example filename: 'LT8BPF20141028232827_20141029015842.01'\n      filename_transform: !regexp-extract 'L[TO]8BPF(?P\u003cyear\u003e[0-9]{4})(?P\u003cmonth\u003e[0-9]{2})(?P\u003cday\u003e[0-9]{2}).*'\n      # Use these group names ('year' and 'month') in the output location:\n      target_dir: /eoancillarydata/sensor-specific/LANDSAT8/BiasParameterFile/{year}/{month}\n\n\n#### !date-range\n\nA `!date-range` is a pseudo-source that repeats a source multiple times over a date range.\n\nIt takes a `start_day` number and an `end_day` number. These are relative to the current\nday: ie. A start day of -3 means three (UTC) days ago .\n\nIt then overrides properties on the embedded source using each date.\n\nExample:\n\n    Modis Att-Ephem:\n    schedule: '20 */2 * * *'\n    source: !date-range\n      start_day: -3\n      end_day: 0\n      overridden_properties:\n        url: http://oceandata.sci.gsfc.nasa.gov/Ancillary/Attitude-Ephemeris/{year}/{julday}\n        target_dir: /eoancillarydata/sensor-specific/MODIS/ancillary/{year}/{julday}\n      using: !http-directory\n        name_pattern: '[AP]M1(ATT|EPH).*'\n        # Overridden by the property above\n        url: ''\n        # Overridden by the property above\n        target_dir: ''\n\nThis expands to four `!http-directory` downloaders. Three days ago, two days ago, one day ago and today.\n\nThe properties in `overridden_properties:` are formatted with the given date and set on each `!http-directory` downloader.\n\n### Post-download file processing\n\nPost-download processing can be done with the `process:` field.\n\nCurrently only shell commands are supported, using the `!shell` processor.\n\nFor example, use gdal to convert each downloaded file from NetCDF (`*.nc`) to Tiff (`*.tiff`):\n\n    Water vapour:\n      schedule: '30 12 * * *'\n      source: !ftp-directory\n        hostname: ftp.cdc.noaa.gov\n        source_dir: /Datasets/ncep.reanalysis/surface\n        # Match filenames such as \"pr_wtr.eatm.2014.nc\"\n        name_pattern: pr_wtr.eatm.[0-9]{4}.nc\n        target_dir: /data/fetch/eoancil-test/water_vapour/source\n      # Convert files to tiff (from netCDF)\n     process: !shell\n        command: 'gdal_translate -a_srs \"+proj=latlong +datum=WGS84\" {parent_dir}/{filename} {parent_dir}/{file_stem}.tif'\n        expect_file: '{parent_dir}/{file_stem}.tif'\n\nWhere:\n\n- `command:` is the shell command to run\n- `expect_file:` is the full path to an output file. (To allow fetch daemon to track newly added files)\n\n\nBoth `command:`, the list of files in `input_files:` and `expect_file:` are evaluated with [python string formatting](https://docs.python.org/3/library/string.html#formatstrings),\n supporting the following fields as well as being evaluated with the named groups found in the `input_files:` pattern:\n\n    # Full name of file (eg. 'pr_wtr.eatm.2014.nc')\n    {filename}\n    # Suffix of filename (eg. '.nc')\n    {file_suffix}\n    # Filename without suffix (eg. 'pr_wtr.eatm.2014')\n    {file_stem}\n    # Directory ('/data/fetch/eoancil-test/water_vapour/source')\n    {parent_dir}\n\n\nA more complex example involving sidecar files requiring download but to be treated as a single group when a post-download process is to take place:\n\n    MODIS BRDF:\n      schedule: '10/10 * * * *'\n      source: !date-range\n        # Download from one day ago (-1) to tomorrow (1):\n        start_day: -20\n        end_day: -10\n        overridden_properties:\n          url: https://e4ftl01.cr.usgs.gov/MOTA/MCD43A1.006/{year}.{month}.{day}\n          target_dir: /tmp/data/BRDF/MCD43A1.006/{year}.{month}.{day}\n        using: !http-directory\n          url: ''\n          target_dir: ''\n          name_pattern: 'MCD43A1\\.A[0-9]{7}\\.h(2[7-9]|3[0-2])v(09|1[0-3])\\.006\\.[0-9]{13}\\.hdf'\n          beforehand: !http-auth\n            url: https://urs.earthdata.nasa.gov\n            username: \u003cusername\u003e\n            password: \u003cpassword\u003e\n      process: !shell\n        command: 'swfo-convert mcd43a1 h5-md --fname {brdf_base}/{collection}/{ymd}/{basename}{hdf_ext} --outdir /tmp/data/conversion/BRDF/{collection}/{ymd}/ --filter-opts ''{{\"aggression\": 6}}'' --compression BLOSC_ZSTANDARD'\n        input_files: ['^(?P\u003cbrdf_base\u003e.*BRDF)/(?P\u003ccollection\u003e.*)/(?P\u003cymd\u003e[0-9]{4}\\.[0-9]{2}\\.[0-9]{2})/(?P\u003cbasename\u003e.*)(?P\u003chdf_ext\u003e.hdf)(?P\u003cxml_ext\u003e.xml)?', ['{brdf_base}/{collection}/{ymd}/{basename}{hdf_ext}', '{brdf_base}/{collection}/{ymd}/{basename}{hdf_ext}.xml']]\n        expect_file: '/tmp/data/conversion/BRDF/{collection}/{ymd}/{basename}.h5'\n\nWhere:\n\n- `command:` is the shell command to run\n- `input_files:` contains a regex pattern, and list of expected files that are to be checked before running the post-process command\n- `expect_file:` is the full path to an output file. (To allow fetch daemon to track newly added files)\nThis is useful when there are sidecar files.\nThe value format is a list where the first element is a regx pattern.  e.g.  `'^(?P\u003cbase\u003e.*hdf)'`\nThis is applied to the full name of the downloaded file and used to create named groups used in the second element.\nThe second element is a list of files that must be present before the shell command is executed.\n\n\n## Signals:\n\nSend a `SIGHUP` signal to reload the config file without interrupting existing downloads:\n\n    kill -1 \u003cpid\u003e\n\nSend a `SIGINT` or `SIGTERM` signal to start a graceful shutdown (any active\ndownloads will be completed before exiting).\n\n    kill \u003cpid\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeoscienceaustralia%2Ffetch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeoscienceaustralia%2Ffetch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeoscienceaustralia%2Ffetch/lists"}