{"id":18913530,"url":"https://github.com/complianceascode/auditree-harvest","last_synced_at":"2025-04-15T08:30:50.851Z","repository":{"id":55968344,"uuid":"280180704","full_name":"ComplianceAsCode/auditree-harvest","owner":"ComplianceAsCode","description":"The Auditree data gathering and reporting tool.","archived":false,"fork":false,"pushed_at":"2024-08-23T14:47:01.000Z","size":74,"stargazers_count":13,"open_issues_count":12,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-10T17:04:48.218Z","etag":null,"topics":["compliance","compliance-as-code","compliance-automation","continuous-compliance","devops","devsecops","python"],"latest_commit_sha":null,"homepage":"https://auditree.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ComplianceAsCode.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-16T14:49:54.000Z","updated_at":"2024-10-30T17:59:52.000Z","dependencies_parsed_at":"2024-08-23T16:13:44.612Z","dependency_job_id":"37a795c2-4851-43bd-b7d9-4f70f9d05a41","html_url":"https://github.com/ComplianceAsCode/auditree-harvest","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ComplianceAsCode%2Fauditree-harvest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ComplianceAsCode%2Fauditree-harvest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ComplianceAsCode%2Fauditree-harvest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ComplianceAsCode%2Fauditree-harvest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ComplianceAsCode","download_url":"https://codeload.github.com/ComplianceAsCode/auditree-harvest/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249035294,"owners_count":21202050,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compliance","compliance-as-code","compliance-automation","continuous-compliance","devops","devsecops","python"],"created_at":"2024-11-08T10:08:11.695Z","updated_at":"2025-04-15T08:30:50.608Z","avatar_url":"https://github.com/ComplianceAsCode.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![OS Compatibility][platform-badge]](#prerequisites)\n[![Python Compatibility][python-badge]][python-dl]\n[![pre-commit][pre-commit-badge]][pre-commit]\n[![Code validation](https://github.com/ComplianceAsCode/auditree-harvest/workflows/format%20%7C%20lint%20%7C%20test/badge.svg)][lint-test]\n[![Upload Python Package](https://github.com/ComplianceAsCode/auditree-harvest/workflows/PyPI%20upload/badge.svg)][pypi-upload]\n\n# auditree-harvest\n\nThe Auditree data gathering and reporting tool.\n\n## Introduction\n\nAuditree `harvest` is a command line tool that assists with the gathering and\nformatting of data into human readable reports.  Auditree `harvest` allows\na user to easily retrieve historical raw data, in bulk, from a Git repository\nand optionally format that raw data to meet reporting needs.  Auditree\n`harvest` is meant to retrieve and report on historical evidence from an\nevidence locker.  It is, however, not limited to just processing evidence.  Any\nfile found in a Git repository hosting service can be processed by Auditree `harvest`.\n\n## Prerequisites\n\n- Supported for execution on OSX and LINUX.\n- Supported for execution with Python 3.6 and above.\n\nPython 3 must be installed, it can be downloaded from the [Python][python-dl]\nsite or installed using your package manager.\n\nPython version can be checked with:\n\n```sh\npython --version\n```\n\nor\n\n```sh\npython3 --version\n```\n\nThe `harvest` tool is available for download from [PyPI](https://pypi.org/project/auditree-harvest/).\n\n## Installation\n\nIt is best practice, but not mandatory, to run `harvest` from a dedicated Python\nvirtual environment.  Assuming that you have the Python [virtualenv][virtual-env]\npackage already installed, you can create a virtual environment named `venv` by\nexecuting `virtualenv venv` which will create a `venv` folder at the location of\nwhere you executed the command.  Alternatively you can use the python `venv` module\nto do the same.\n\n```sh\npython3 -m venv venv\n```\n\nAssuming that you have a virtual environment and that virtual environment is in\nthe current directory then to install a new instance of `harvest`, activate\nyour virtual environment and use `pip` to install `harvest` like so:\n\n```sh\n. ./venv/bin/activate\npip install auditree-harvest\n```\n\nAs we add functionality to `harvest` users will want to upgrade their `harvest`\npackage regularly.  To upgrade `harvest` to the most recent version do:\n\n```sh\n. ./venv/bin/activate\npip install auditree-harvest --upgrade\n```\n\nSee [pip documentation][pip-docs] for additional options for using `pip`.\n\n## Configuration\n\nSince Auditree `harvest` interacts with Git repositories, it requires Git remote\nhosting service credentials in order to do its thing.  Auditree `harvest` will by\ndefault look for a `username` and `token` in a `~/.credentials` file.  You can\noverride the credentials file location by using the `--creds` option on a `harvest`\nCLI execution. Valid section headings include `github`, `github_enterprise`, `bitbucket`,\nand `gitlab`.  Below is an example of the expected credentials entry.\n\n```ini\n[github]\nusername=your-gh-username\ntoken=your-gh-token\n```\n\n## Execution\n\n### Collate data\n\nTo collate historical versions of a file from a Git repository hosting service\nlike Github, provide the repository URL (`repo` positional argument), the\nrelative path to the file within the remote repository including the file name\n(`filepath` positional argument) and an optional date range (`--start` and `--end`\narguments).  You can also, optionally, provide the local Git repository path\n(`--repo-path` argument), if the repository already exists locally and you wish\nto override the remote repository download behavior.\n\n```sh\nharvest collate https://github.com/org-foo/repo-bar /raw/baz/baz.json --start 20191201 --end 20191212 --repo-path ./bar-repo\n```\n\n- File versions are written to the current local directory where `harvest` was\nexecuted from.\n- File versions are prefixed by the commit date in `YYYYMMDD` format.\n- File versions are gathered with daily granularity.\n   - Only the latest version of a file for a given day is retrieved.\n   - If a file did not change on a date then no file version is written for that\n   date.  Instead the latest version prior to that date serves as the version of\n   that file for that date.\n- If you don't provide a `--start` and an `--end` then the latest version of a\nfile is retrieved.\n- If you only provide a `--start` date file versions from the start date to the\ncurrent date are retrieved.\n- If you only provide an `--end` date the latest version of a file for the end\ndate is retrieved.\n\n### Generate report(s)\n\nTo run a report using content contained in a Git repository hosting service\nlike Github, provide the repository URL (`repo` positional argument), the report\npackage (`package`), the report name (`name` positional argument) and include\nany configuration that the report requires (`--config`) as a JSON string.  You\ncan also, optionally, provide the local Git repository path (`--repo-path`\nargument), if the repository already exists locally and you wish to override\nthe remote repository download behavior.\n\n```sh\nharvest report https://github.com/org-foo/repo-bar auditree_arboretum check_results_summary --config '{\"start\":\"20191212\",\"end\":\"20191221\"}'\n```\n\n#### Getting report details\n\nTo see a full summary of available reports within any package (like `auditree-arboretum`) do:\n\n```sh\nharvest reports auditree_arboretum --list\n```\n\nTo see details on a specific report that include usage example do something like:\n\n```sh\nharvest reports auditree_arboretum --detail check_results_summary\n```\n\n## Report development\n\nReports should be hosted with the fetchers/checks that collect the evidence for\nthe reports process. Within `auditree-arboretum` this means the code lives in the\nappropriate provider directory.  Contributing common harvest reports are as follows:\n\n1.  Adhere to the auditree-arboretum contribution guidelines - **TODO add link**.\n2.  Reports go in the \"reports\" folder by provider.\n3.  Create a python module with a class that extends the [BaseReporter][base-reporter]\nclass.\n    - The `harvest` CLI will use the report module name as the name of the\n    report (_sans the .py extention)._\n    - **Only one report class per report module is permitted.**\n4.  In the new report class the expectations are as follows:\n    - Provide a module level docstring that contains:\n       - A single line summary\n       - A detailed description of the report that includes evidence/files being\n       processed and expected configuration\n       - At least one usage example\n       - Use the [check results summary report docstring][crs-rpt] as an example/template.\n       - `harvest` uses this docstring to display available reports and their\n       details to the user.\n    - Provide/Override the `report_filename` property to return the name of the\n    report (including extension).  `harvest` uses this property to apply a report\n    template (if desired) and to determine which writer function to use when writing\n    the report to a file.  Use the [check results summary report report_filename property][crs-rpt]\n    and the [Python packages summary report report_filename property][pps-rpt] as examples.\n    - Provide/Override the `generate_report` method.  This is where you put your\n    evidence processing and report formatting logic.  Use the\n    [check results summary report generate_report][crs-rpt] method as an example.\n       - `harvest` takes the optional `--config` command line argument as a JSON\n       string when executing a report, converts it to a dictionary and attaches\n       it as the `config` attribute to your report object.  Use the report object's\n       `config` attribute in the `generate_report` method if you plan to have report\n       specific configuration options.\n       - Your report object also has a method that retrieves an evidence file for\n       a given date. Use the report object's `get_file_content` method when\n       retrieving evidence from an evidence locker.\n       - Generating CSV reports:\n          - `harvest` uses the Python [CSV writer][python-csv] to write out the\n          report file. So be sure that your `generate_report` method returns a\n          list of dictionaries that adheres to the expectations of the Python\n          [CSV writer][python-csv].\n       - Generating reports from a Jinja2 template:\n          - Add a report template named the same as your `report_filename`\n          property with a `.tmpl` extension.  `harvest` will start to look for\n          the template in the same directory as the report module.  So as long as\n          it exists within that directory structure, `harvest` will find it.\n          Use [python_packages_summary.md.tmpl][pps-rpt-tmpl] as an example.\n          - `harvest` will look for this template file as part of your report\n          processing and, if found, will pass your `generate_report` returned\n          content through the template logic.\n          - Your `generate_report` returned content should be a dictionary with\n          everything necessary for your report template to render the desired\n          report appropriately.\n          - The report template can access the \"raw\" content generated by\n          `generate_report` through a dictionary named `data` and also has\n          access to the report's attributes through the `report` object.\n          Use [python_packages_summary.md.tmpl][pps-rpt-tmpl] as an example.\n       - Generating reports without templates:\n          - You just want to generate report content directly from `generate_report`?\n          No problem.  Just generate a string as the report content or a list of\n          strings as the rows of the report content and `harvest` will do the rest.\n\n### Custom report development\n\nIf you find that you have a specific reporting need that does not fit in as a common\n`harvest` report, no problem.  Just develop the report in a separate repo/project\nfollowing the same guidelines as above.  As long as the package is importable by\npython and you tell `harvest` what package to look for your report(s) in via the CLI,\nit will handle the rest.\n\n\n[changes]: https://github.com/ComplianceAsCode/auditree-harvest/blob/main/CHANGES.md\n[platform-badge]: https://img.shields.io/badge/platform-osx%20|%20linux-orange.svg\n[python-badge]: https://img.shields.io/badge/python-v3.6+-blue.svg\n[python-dl]: https://www.python.org/downloads/\n[pip-docs]: https://pip.pypa.io/en/stable/reference/pip/\n[virtual-env]: https://pypi.org/project/virtualenv/\n[contributing]: https://github.com/ComplianceAsCode/auditree-harvest/blob/main/CONTRIBUTING.md\n[base-reporter]: https://github.com/ComplianceAsCode/auditree-harvest/blob/main/harvest/reporter.py\n[crs-rpt]: https://github.com/ComplianceAsCode/auditree-harvest/blob/main/auditree_arboretum/provider/auditree/reports/check_results_summary.py\n[pps-rpt]: https://github.com/ComplianceAsCode/auditree-harvest/blob/main/auditree_arboretum/provider/auditree/reports/python_packages_summary.py\n[python-csv]: https://docs.python.org/3/library/csv.html#csv.writer\n[python-io]: https://docs.python.org/3/tutorial/inputoutput.html\n[pps-rpt-tmpl]: https://github.com/ComplianceAsCode/auditree-harvest/blob/main/auditree_arboretum/provider/auditree/reports/report_templates/python_packages_summary.md.tmpl\n[pre-commit-badge]: https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit\u0026logoColor=white\n[pre-commit]: https://github.com/pre-commit/pre-commit\n[lint-test]: https://github.com/ComplianceAsCode/auditree-harvest/actions?query=workflow%3A%22format+%7C+lint+%7C+test%22\n[pypi-upload]: https://github.com/ComplianceAsCode/auditree-harvest/actions?query=workflow%3A%22PyPI+upload%22\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcomplianceascode%2Fauditree-harvest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcomplianceascode%2Fauditree-harvest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcomplianceascode%2Fauditree-harvest/lists"}