{"id":31956023,"url":"https://github.com/cisagov/cyhy-feeds","last_synced_at":"2025-10-23T19:53:31.795Z","repository":{"id":37848340,"uuid":"144632857","full_name":"cisagov/cyhy-feeds","owner":"cisagov","description":"Tools to create and retrieve Cyber Hygiene (CyHy) data extracts","archived":false,"fork":false,"pushed_at":"2025-09-09T19:26:19.000Z","size":411,"stargazers_count":1,"open_issues_count":6,"forks_count":4,"subscribers_count":4,"default_branch":"develop","last_synced_at":"2025-10-14T14:47:31.455Z","etag":null,"topics":["cyhy","data","export"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cisagov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-08-13T20:54:54.000Z","updated_at":"2025-09-09T19:26:22.000Z","dependencies_parsed_at":"2025-09-09T21:22:02.790Z","dependency_job_id":"7a93ffa4-e147-4bcb-acef-8bd8bfd37e9e","html_url":"https://github.com/cisagov/cyhy-feeds","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cisagov/cyhy-feeds","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cisagov%2Fcyhy-feeds","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cisagov%2Fcyhy-feeds/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cisagov%2Fcyhy-feeds/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cisagov%2Fcyhy-feeds/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cisagov","download_url":"https://codeload.github.com/cisagov/cyhy-feeds/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cisagov%2Fcyhy-feeds/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280683815,"owners_count":26372970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-23T02:00:06.710Z","response_time":142,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cyhy","data","export"],"created_at":"2025-10-14T14:47:16.592Z","updated_at":"2025-10-23T19:53:31.747Z","avatar_url":"https://github.com/cisagov.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cyber Hygiene Feeds 📥 📤 #\n\n[![GitHub Build Status](https://github.com/cisagov/cyhy-feeds/workflows/build/badge.svg)](https://github.com/cisagov/cyhy-feeds/actions)\n[![Coverage Status](https://coveralls.io/repos/github/cisagov/cyhy-feeds/badge.svg?branch=develop)](https://coveralls.io/github/cisagov/cyhy-feeds?branch=develop)\n[![Total alerts](https://img.shields.io/lgtm/alerts/g/cisagov/cyhy-feeds.svg?logo=lgtm\u0026logoWidth=18)](https://lgtm.com/projects/g/cisagov/cyhy-feeds/alerts/)\n[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/cisagov/cyhy-feeds.svg?logo=lgtm\u0026logoWidth=18)](https://lgtm.com/projects/g/cisagov/cyhy-feeds/context:python)\n[![Known Vulnerabilities](https://snyk.io/test/github/cisagov/cyhy-feeds/develop/badge.svg)](https://snyk.io/test/github/cisagov/cyhy-feeds)\n\ncyhy-feeds consists of two parts: the extractor and the retriever.\n\n`cyhy-data-extract` retrieve and compress the specified data, sign the compressed\nfile, encrypt the file, and optionally push the encrypted, compressed file to an\nS3 bucket using provided AWS credentials.\n\n`cyhy-data-retriever` take a provided file (optionally stored in an S3 bucket),\ndecrypt it, and then decompress it to local storage.\n\n## Getting Started ##\n\nBoth `cyhy-data-extract` and `cyhy-data-retriever` require Python 3.\n\nTo run the tool locally first install the requirements:\n\n```console\npip install -r requirements.txt\n```\n\n### cyhy-data-extract Usage and Examples ###\n\n#### cyhy-data-extract Usage ####\n\n```console\nCreate compressed, encrypted, signed extract file with Federal CyHy data for\nintegration with the Weathermap project.\n\nUsage:\n  COMMAND_NAME --config CONFIG_FILE [--cyhy-config CYHY_CONFIG] [--scan-config SCAN_CONFIG] [--assessment-config ASSESSMENT_CONFIG] [-v | --verbose] [-a | --aws ] [--cleanup-aws] [--date DATE] [--debug]\n  COMMAND_NAME (-h | --help)\n  COMMAND_NAME --version\n\nOptions:\n  -h --help                                                         Show this screen\n  --version                                                         Show version\n  -x CYHY_CONFIG --cyhy-config=CYHY_CONFIG                          CyHy MongoDB configuration to use\n  -y SCAN_CONFIG --scan-config=SCAN_CONFIG                          Scan MongoDB configuration to use\n  -z ASSESSMENT_CONFIG --assessment-config=ASSESSMENT_CONFIG        Assessment MongoDB configuration to use\n  -v --verbose                                                      Show verbose output\n  -a --aws                                                          Output results to S3 bucket\n  --cleanup-aws                                                     Delete old files from the S3 bucket\n  -c CONFIG_FILE --config=CONFIG_FILE                               Configuration file for this script\n  -d DATE --date=DATE                                               Specific date to export data from in form: %Y-%m-%d (eg. 2018-12-31) NOTE that this date is in UTC\n  --debug                                                           Enable debug logging\n```\n\n#### cyhy-data-extract Examples ####\n\nExtract CyHy data for the current day using the MongoDB configuration in `cyhy.yml`\nand the runtime configuration in `cyhy-data-extract.cfg`.\n\n```console\npython3 cyhy-data-extract.py --cyhy-config cyhy.yml --config cyhy-data-extract.cfg\n```\n\nExtract scan data for the current day using the MongoDB configuration in 'scan.yml'\nand the runtime configuration in `cyhy-data-extract.cfg`.\n\n```console\npython3 cyhy-data-extract.py --scan-config scan.yml --config cyhy-data-extract.cfg\n```\n\nExtract assessment data for the current day using the MongoDB configuration in\n`assessment.yml` and the runtime configuration in `cyhy-data-extract.cfg`.\n\n```console\npython3 cyhy-data-extract.py --assessment-config assessment.yml --config cyhy-data-extract.cfg\n```\n\nExtract CyHy and scan data for the current day using the MongoDB configurations\nin `cyhy.yml` and `scan.yml`, respectively, and use the runtime configuration in\n`cyhy-data-extract.cfg`.\n\n```console\npython3 cyhy-data-extract.py --cyhy-config cyhy.yml --scan-config scan.yml\n  --config cyhy-data-extract.cfg\n```\n\nExtract CyHy and scan data for the current day using the MongoDB configurations\nin `cyhy.yml` and `scan.yml`, upload the results to AWS, and use the runtime\nconfiguration in `cyhy-data-extract.cfg`.\n\n```console\npython3 cyhy-data-extract.py --cyhy-config cyhy.yml --scan-config scan.yml\n  --aws --config cyhy-data-extract.cfg\n```\n\nExtract CyHy and scan data for January 25th, 2019 using the MongoDB configurations\nin `cyhy.yml` and `scan.yml`, upload the results to AWS, and use the runtime\nconfiguration in `cyhy-data-extract.cfg`.\n\n```console\npython3 cyhy-data-extract.py --cyhy-config cyhy.yml --scan-config scan.yml\n  --aws --config cyhy-data-extract.cfg --date 2019-01-25\n```\n\nExtract CyHy, scan, and assessment data for January 25th, 2019 using the MongoDB\nconfigurations in `cyhy.yml`, `scan.yml`, and `assessment.yml`, upload the results\nto AWS, and use the runtime configuration in `cyhy-data-extract.cfg`.\n\n```console\npython3 cyhy-data-extract.py --cyhy-config cyhy.yml --scan-config scan.yml\n  --assessment-config assessment.yml --aws --config cyhy-data-extract.cfg\n  --date 2019-01-25\n```\n\n### cyhy-data-retriever Usage and Examples ###\n\n#### cyhy-data-retriever Usage ####\n\n```console\nRetrieve a compressed, encrypted, signed extract file and\nverify/decrypt/uncompress it.\n\n   NOTES:\n   * the python modules below must be installed for the script to work\n   * This script expects to operate on a GPG-encrypted bzip2 tar file\n      e.g. filename.tbz.gpg\n\nUsage:\n  COMMAND_NAME [-v | --verbose] [--filename EXTRACT_FILENAME] [-a | --aws]\n    --config CONFIG_FILE\n  COMMAND_NAME (-h | --help)\n  COMMAND_NAME --version\n\nOptions:\n  -h --help                                         Show this screen\n  --version                                         Show version\n  -f EXTRACT_FILENAME --filename=EXTRACT_FILENAME   Name of extract file to retrieve\n  -v --verbose                                      Show verbose output\n  -c CONFIG_FILE --config=CONFIG_FILE               Configuration file for this script\n  -a --aws                                          Output results to S3 bucket\n\n```\n\n#### cyhy-data-retriever Examples ####\n\nRetrieve the data stored in file `cyhy_extract_2019-01-25T000000+0000.tbz.gpg`\nresiding on AWS using the runtime configuration in `cyhy-data-retriever.cfg`.\n\n```console\n  cyhy-data-retriever --filename cyhy_extract_2019-01-25T000000+0000.tbz.gpg --aws\n    --config cyhy-data-retriever.cfg\n```\n\n### Extract Config File Parameters ###\n\n- `FED_ORGS_EXCLUDED` - Orgs to exclude from extract\n- `GNUPG_HOME` - Location of GNUPG database (eg. /Users/bob/.gnupg)\n- `RECIPIENTS` - Names on the gpg public key(s)\n- `SIGNER` - Gpg signer to ensure integrity\n- `SIGNER_PASSPHRASE` - Passphrase for signer gpg key\n- `OUTPUT_DIR` - Directory to output extract to\n- `FILE_RETENTION_NUM_DAYS` - Number of days to hold extract\n- `ES_AWS_CONFIG_SECTION_NAME` - Name of the AWS config file section\n  containing the configuration to be used when accessing the\n  Elasticsearch data\n- `ES_REGION` - Region for DMARC bucket\n- `ES_URL` - Elasticsearch URL\n- `ES_RETRIEVE_SIZE` - Elasticsearch size\n\n### Retriever Config File Parameters ###\n\n- `CLIENT_PRIVATE_KEY_FILE` - Path to gpg private key\n- `GNUPG_HOME` - Location of GPG database (eg. /Users/bob/.gnupg)\n- `GPG_DECRYPTION_PASSPHRASE` - Passphrase for private gpg key\n- `AWS_ACCESS_KEY_ID` - User ID used for AWS S3 bucket read access\n- `AWS_SECRET_ACCESS_KEY` - Key for AWS S3 bucket read access\n- `PROXY_CONFIG` - Only needed when proxy is present\n\n## Contributing ##\n\nWe welcome contributions!  Please see [here](CONTRIBUTING.md) for\ndetails.\n\n## License ##\n\nThis project is in the worldwide [public domain](LICENSE).\n\nThis project is in the public domain within the United States, and\ncopyright and related rights in the work worldwide are waived through\nthe [CC0 1.0 Universal public domain\ndedication](https://creativecommons.org/publicdomain/zero/1.0/).\n\nAll contributions to this project will be released under the CC0\ndedication. By submitting a pull request, you are agreeing to comply\nwith this waiver of copyright interest.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcisagov%2Fcyhy-feeds","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcisagov%2Fcyhy-feeds","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcisagov%2Fcyhy-feeds/lists"}