{"id":32174956,"url":"https://github.com/podaac/data-subscriber","last_synced_at":"2025-10-21T19:18:04.173Z","repository":{"id":37849054,"uuid":"366596558","full_name":"podaac/data-subscriber","owner":"podaac","description":"Subscribe and bulk download collections of data at PO.DAAC","archived":false,"fork":false,"pushed_at":"2025-03-15T00:18:06.000Z","size":1698,"stargazers_count":107,"open_issues_count":51,"forks_count":34,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-09-25T04:44:05.287Z","etag":null,"topics":["data-access","development","python","tva"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/podaac.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-12T04:55:41.000Z","updated_at":"2025-09-24T11:20:05.000Z","dependencies_parsed_at":"2023-02-18T09:45:56.006Z","dependency_job_id":"90aa1fd9-2dfb-4f27-bfb0-4e60c0e0adf1","html_url":"https://github.com/podaac/data-subscriber","commit_stats":{"total_commits":135,"total_committers":14,"mean_commits":9.642857142857142,"dds":0.7259259259259259,"last_synced_commit":"aebd81893f22a2c06f93b670ce7929bb33f28243"},"previous_names":[],"tags_count":31,"template":false,"template_full_name":null,"purl":"pkg:github/podaac/data-subscriber","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/podaac%2Fdata-subscriber","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/podaac%2Fdata-subscriber/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/podaac%2Fdata-subscriber/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/podaac%2Fdata-subscriber/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/podaac","download_url":"https://codeload.github.com/podaac/data-subscriber/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/podaac%2Fdata-subscriber/sbom","scorecard":{"id":739421,"data":{"date":"2025-08-11","repo":{"name":"github.com/podaac/data-subscriber","commit":"7c3895d973c0d9840476220aab7dc50ad526b130"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.3,"checks":[{"name":"Code-Review","score":4,"reason":"Found 5/11 approved changesets -- score normalized to 4","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/python-app.yml:1","Warn: no topLevel permission defined: .github/workflows/release.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-app.yml:23: update your workflow using https://app.stepsecurity.io/secureworkflow/podaac/data-subscriber/python-app.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-app.yml:25: update your workflow using https://app.stepsecurity.io/secureworkflow/podaac/data-subscriber/python-app.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/python-app.yml:29: update your workflow using https://app.stepsecurity.io/secureworkflow/podaac/data-subscriber/python-app.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/python-app.yml:47: update your workflow using https://app.stepsecurity.io/secureworkflow/podaac/data-subscriber/python-app.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-app.yml:70: update your workflow using https://app.stepsecurity.io/secureworkflow/podaac/data-subscriber/python-app.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/python-app.yml:72: update your workflow using https://app.stepsecurity.io/secureworkflow/podaac/data-subscriber/python-app.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/python-app.yml:76: update your workflow using https://app.stepsecurity.io/secureworkflow/podaac/data-subscriber/python-app.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/python-app.yml:83: update your workflow using https://app.stepsecurity.io/secureworkflow/podaac/data-subscriber/python-app.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/podaac/data-subscriber/release.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:17: update your workflow using https://app.stepsecurity.io/secureworkflow/podaac/data-subscriber/release.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release.yml:21: update your workflow using https://app.stepsecurity.io/secureworkflow/podaac/data-subscriber/release.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release.yml:35: update your workflow using https://app.stepsecurity.io/secureworkflow/podaac/data-subscriber/release.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:51: update your workflow using https://app.stepsecurity.io/secureworkflow/podaac/data-subscriber/release.yml/main?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/release.yml:57","Warn: pipCommand not pinned by hash: .github/workflows/release.yml:60","Warn: pipCommand not pinned by hash: .github/workflows/release.yml:61","Warn: pipCommand not pinned by hash: .github/workflows/release.yml:62","Info:   0 out of   7 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   6 third-party GitHubAction dependencies pinned","Info:   0 out of   4 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 27 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":1,"reason":"9 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2024-230 / GHSA-248v-346w-9cwc","Warn: Project is vulnerable to: PYSEC-2024-60 / GHSA-jjg7-2v4v-x38h","Warn: Project is vulnerable to: GHSA-9hjg-9r4m-mvj7","Warn: Project is vulnerable to: GHSA-9wx4-h78v-vm56","Warn: Project is vulnerable to: GHSA-34jh-p97f-mpxf","Warn: Project is vulnerable to: PYSEC-2023-212 / GHSA-g4mx-q9vg-27p4","Warn: Project is vulnerable to: GHSA-pq67-6m6q-mj2v","Warn: Project is vulnerable to: PYSEC-2023-192 / GHSA-v845-jxx5-vc9f","Warn: Project is vulnerable to: GHSA-jfmj-5v4g-7637"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-22T16:59:09.351Z","repository_id":37849054,"created_at":"2025-08-22T16:59:09.351Z","updated_at":"2025-08-22T16:59:09.351Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280319046,"owners_count":26310346,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-21T02:00:06.614Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-access","development","python","tva"],"created_at":"2025-10-21T19:17:59.349Z","updated_at":"2025-10-21T19:18:04.168Z","avatar_url":"https://github.com/podaac.png","language":"Python","readme":"[![Python Build](https://github.com/podaac/data-subscriber/actions/workflows/python-app.yml/badge.svg?branch=main)](https://github.com/podaac/data-subscriber/actions/workflows/python-app.yml)\r\n[![PyPi release](https://github.com/podaac/data-subscriber/actions/workflows/release.yml/badge.svg)](https://github.com/podaac/data-subscriber/actions/workflows/release.yml)\r\n\r\n\r\n# Scripted Access to PODAAC data\r\n\r\n ----\r\n\r\n![N|Solid](https://podaac.jpl.nasa.gov/sites/default/files/image/custom_thumbs/podaac_logo.png)\r\n\r\n\r\n## Subscriber or Bulk Download?\r\n\r\nThere are 2 tools in this repository, the data subscriber and the data downloader. Which you use depends on your use case. If you're not sure, we'd recommend starting with the downloader.\r\n\r\n![Download or Subscribe?](/img/PO.DAAC%20Tools.png)\r\n\r\n**Downloader** - [Documentation](Downloader.md)\r\n\r\nThe Downloader is useful if you need to download PO.DAAC data once in a while or prefer to do it \"on-demand\". The Downloader makes no assumptions about the last time run or what is new in the archive, it simply uses the provided requests and downloads all matching data.\r\n\r\n**Subscriber** - [Documentation](Subscriber.md)\r\n\r\nThe subscriber is useful for users who need to continuously pull the latest data from the PO.DAAC archive. If you feed data into a model or real time process, the subscriber allows you to repeatedly run the script and only download the latest data.\r\n\r\n\r\n## Installation\r\n\r\nBoth subscriber and downloader require Python \u003e= 3.7.\r\n\r\nThe subscriber and downloader scripts are available in the [pypi python repository](https://pypi.org/project/podaac-data-subscriber/), it can be installed via pip:\r\n\r\n```\r\npip install podaac-data-subscriber\r\n```\r\n\r\nyou should now have access to the downloader and subscriber Command line interfaces:\r\n\r\n```\r\n$\u003e usage: PO.DAAC data subscriber [-h] -c COLLECTION -d OUTPUTDIRECTORY [-f] [-sd STARTDATE] [-ed ENDDATE] [-b BBOX] [-dc] [-dydoy] [-dymd] [-dy] [--offset OFFSET] [-m MINUTES]\r\n                               [-e EXTENSIONS] [--process PROCESS_CMD] [--version] [--verbose] [-p PROVIDER] [--dry-run]\r\n\r\n...\r\n```\r\n\r\n```\r\n$\u003e usage: PO.DAAC bulk-data downloader [-h] -c COLLECTION -d OUTPUTDIRECTORY [--cycle SEARCH_CYCLES] [-sd STARTDATE] [-ed ENDDATE] [-f] [-b BBOX] [-dc] [-dydoy] [-dymd] [-dy]\r\n                                    [--offset OFFSET] [-e EXTENSIONS] [-gr GRANULENAME] [--process PROCESS_CMD] [--version] [--verbose] [-p PROVIDER] [--limit LIMIT] [--dry-run]\r\n\r\n\r\n...\r\n```\r\n\r\n**Note:** If after installation, the `podaac-data-subscriber` or `podaac-data-downloader` commands are not available, you may need to add the script location to the PATH. This could be due to a *User Install* of the python package, which is common on shared systems where python packages are installed for the user (not the system). See [Installing to the User Site](https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-to-the-user-site) and [User Installs](https://pip.pypa.io/en/latest/user_guide/#user-installs) for more information on finding the location of installed scripts and adding them to the PATH.\r\n\r\n## Step 1:  Get Earthdata Login     \r\nThis step is needed only if you dont have an Earthdata login already.\r\nhttps://urs.earthdata.nasa.gov/\r\n\u003e The Earthdata Login provides a single mechanism for user registration and profile  management for all EOSDIS system components (DAACs, Tools, Services). Your Earthdata login   also helps the EOSDIS program better understand the usage of EOSDIS services to improve  user experience through customization of tools and improvement of services. EOSDIS data are  openly available to all and free of charge except where governed by international  agreements.\r\n\r\nFor setting up your authentication, see the notes on the `netrc` file below.\r\n\r\n## Step 2: Setup your Earthdata Login\r\nThe netrc used within the script  will allow Python scripts to log into any Earthdata Login without being prompted for\r\ncredentials every time you run. The netrc file should be placed in your HOME directory.\r\nTo find the location of your HOME directory\r\n\r\nOn UNIX you can use\r\n```\r\necho $HOME\r\n```\r\nOn Windows you can use\r\n```\r\necho %HOMEDRIVE%%HOMEPATH%\r\n```\r\n\r\nThe output location from the command above should be the location of the `.netrc` (`_netrc` on Windows) file.\r\n\r\nThe format of the `netrc` file is as follows:\r\n\r\n```\r\nmachine urs.earthdata.nasa.gov\r\n    login \u003cyour username\u003e\r\n    password \u003cyour password\u003e\r\n```\r\nfor example:\r\n\r\n```\r\nmachine urs.earthdata.nasa.gov\r\n    login podaacUser\r\n    password podaacIsAwesome\r\n```\r\n\r\n**If the script cannot find the netrc file, you will be prompted to enter the username and password and the script wont be able to generate the CMR token**\r\n\r\n\r\n## Advanced Usage\r\n\r\n### Request data from another DAAC...\r\n\r\nUse the 'provider' flag to point at a non-PO.DAAC provider. Be aware, the default data types (--extensions) may need to be specified if the desired data are not in the defaults.\r\n\r\n```\r\npodaac-data-subscriber -c SENTINEL-1A_SLC -d myData  -p ASF -sd 2014-06-01T00:46:02Z\r\n```\r\n\r\n### Logging\r\n\r\nFor error troubleshooting, one can set an environment variable to gain more insight into errors:\r\n\r\n```\r\nexport PODAAC_LOGLEVEL=DEBUG\r\n```\r\n\r\nAnd then run the script. This should give you more verbose output on URL requests to CMR, tokens, etc.\r\n\r\n### OTHER OPTIONS\r\n\r\nThe podaac downloader and subscriber make calls to github for checking recent releases. Unauthenticated requests are limited to 60 per hour. If you start seeing errors like:\r\n```\r\nreleases_json = {'documentation_url': 'https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting', 'message': \"API... here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)\"}\r\n```\r\nYou'll want to set the environment variable GITHUB_TOKEN to a github personal access token- this allows for up to 5000 calls per hour. This requires a free github account. Most users will not run in to this issue.\r\n\r\n\r\n### In need of Help?\r\nThe PO.DAAC User Services Office is the primary point of contact for answering your questions concerning data and information held by the PO.DAAC. User Services staff members are knowledgeable about both the data ordering system and the data products themselves. We answer questions about data, route requests to other DAACs, and direct questions we cannot answer to the appropriate information source.\r\n\r\nPlease contact us via email at podaac@podaac.jpl.nasa.gov\r\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpodaac%2Fdata-subscriber","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpodaac%2Fdata-subscriber","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpodaac%2Fdata-subscriber/lists"}