{"id":41698220,"url":"https://github.com/grip-on-software/data-gathering","last_synced_at":"2026-01-24T20:55:07.323Z","repository":{"id":171609868,"uuid":"648147761","full_name":"grip-on-software/data-gathering","owner":"grip-on-software","description":"Modules used to gather data from different data sources in software development processes","archived":false,"fork":false,"pushed_at":"2024-07-19T10:05:01.000Z","size":2259,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-24T22:00:07.347Z","etag":null,"topics":["data-gathering","software-development-process"],"latest_commit_sha":null,"homepage":"https://gros.liacs.nl/data-gathering/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grip-on-software.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-06-01T10:08:22.000Z","updated_at":"2024-07-19T10:05:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"61a6214c-129e-452a-9066-ff96a1c1c5e2","html_url":"https://github.com/grip-on-software/data-gathering","commit_stats":null,"previous_names":["grip-on-software/data-gathering"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/grip-on-software/data-gathering","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grip-on-software%2Fdata-gathering","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grip-on-software%2Fdata-gathering/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grip-on-software%2Fdata-gathering/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grip-on-software%2Fdata-gathering/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grip-on-software","download_url":"https://codeload.github.com/grip-on-software/data-gathering/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grip-on-software%2Fdata-gathering/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28736791,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-24T19:23:36.361Z","status":"ssl_error","status_checked_at":"2026-01-24T19:23:28.966Z","response_time":89,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-gathering","software-development-process"],"created_at":"2026-01-24T20:55:06.569Z","updated_at":"2026-01-24T20:55:07.308Z","avatar_url":"https://github.com/grip-on-software.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Software development process data gathering\n===========================================\n\n[![PyPI](https://img.shields.io/pypi/v/gros-gatherer.svg)](https://pypi.python.org/pypi/gros-gatherer)\n[![Build \nstatus](https://github.com/grip-on-software/data-gathering/actions/workflows/gatherer-tests.yml/badge.svg)](https://github.com/grip-on-software/data-gathering/actions/workflows/gatherer-tests.yml)\n[![Coverage \nStatus](https://coveralls.io/repos/github/grip-on-software/data-gathering/badge.svg?branch=master)](https://coveralls.io/github/grip-on-software/data-gathering?branch=master)\n[![Quality Gate\nStatus](https://sonarcloud.io/api/project_badges/measure?project=grip-on-software_data-gathering\u0026metric=alert_status)](https://sonarcloud.io/project/overview?id=grip-on-software_data-gathering)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.10911862.svg)](https://doi.org/10.5281/zenodo.10911862)\n\nThe Python modules in this repository gather data from different sources that \nare used by software development teams and projects, as well as control \na distributed setup of data gathering. The data gathering modules are part of \nGrip on Software, a research project involving a larger pipeline where the \ngathered data is made available for analysis purposes through a MonetDB \ndatabase setup.\n\nThe following systems from software development processes are able to be \ninteracted with using the GROS gatherer modules, focusing on data acquisition:\n\n- Jira\n- Git, including additional repository data from GitHub and GitLab\n- Azure DevOps/VSTS/TFS, including Git-based data\n- Subversion\n- Jenkins\n- Quality-time\n- SonarQube\n- BigBoat\n\nThere are many ways to use the GROS gatherer, such as manual script usage, \nDocker images, Jenkins jobs, agent-based Docker compose network isolation, \ncentral controller instances and usage in other applications. However, this \nREADME.md document focuses on the module installation and development. More \nthorough documentation on compatibility with versions of data sources, \nconfiguration, script overviews and agent-controller APIs is found in the \n[online data-gathering documentation](https://gros.liacs.nl/data-gathering/).\n\n## Installation\n\nThe data gathering modules require Python version 3.8 and higher.\n\nTo obtain the latest release version of the module and its dependencies from \nPyPI, use the following command:\n\n```\npip install gros-gatherer\n```\n\nWe recommend creating a virtual environment to manage your dependencies. Make \nsure that `python` runs the Python version in the virtual environment. \nOtherwise, the dependencies are installed to the system libraries path or the \nuser's Python libraries path if you do not have access to the system libraries. \n\n## Configuration\n\nSome modules require the existence of settings and credentials files in the \ndirectory from which the script importing the module is run. This path is \nadjustable with environment variables. For details on configuration, view the \n[documentation](https://gros.liacs.nl/data-gathering/configuration.html).\n\n## Development and testing\n\nMost of the modules come with unit tests, while also depending on the \ncorrectness of dependencies to provide accurate data from sources (i.e. our \nunit tests often use mocks in place of the dependencies) and testing the actual \nsystem in non-production settings. To run unit tests in this repository, first \ninstall the test dependencies with `make setup_test` which also installs all \ndependencies for the modules. Then `coverage run tests.py` provides test \nresults in the output, with XML versions compatible with, e.g., JUnit and \nSonarQube available in the `test-reports/` directory. Detailed information on \ntest coverage is also obtainable after a test run in various report formats, \nfor example:\n\n- `coverage report -m` for a report on (counts of) statements and branches that \n  were hit and missed in the modules in the output.\n- `coverage html` for a HTML report in the `htmlcov/` directory.\n- `coverage xml -i` for an XML output suitable for, e.g., SonarQube.\n\nTo perform all the steps except the HTML report, run `make coverage`. If you do \nnot need XML outputs (each test class writes an XML file by default), then run \n`make test` to just report on test successes and failures or `make cover` to \nalso have the terminal report on statement/branch hits/misses.\n\n[GitHub Actions](https://github.com/grip-on-software/data-gathering/actions) is \nused to run the unit tests and report on coverage on commits and pull requests. \nThis includes quality gate scans tracked by \n[SonarCloud](https://sonarcloud.io/project/overview?id=grip-on-software_data-gathering) \nand [Coveralls](https://coveralls.io/github/grip-on-software/data-gathering) \nfor coverage history.\n\nThe Python scripts and modules conform to code style and typing standards which \nmay be checked using Pylint with `make pylint` and mypy with `make mypy`, \nrespectively, after running `make setup_analysis` to install static code \nanalysis tools. The command for mypy provides potential errors in the output \nand typing coverage reports in several formats, including XML (compatible with \nJUnit and  SonarQube) in the `mypy-report/` directory. To also receive the HTML \nreport, use `make mypy_html` instead.\n\nFinally, the schemas in the `schema/` directory allow validation of certain \nconfiguration files as well as all the exported artifacts against the schema. \nFor example, the Jira and Azure DevOps field mapping specifications are able to \nbe checked; see the [issue \ntrackers](https://gros.liacs.nl/data-gathering/configuration.html#issue-trackers-jira-and-azure-devops) \ndocumentation section for an example.\n\nWe publish releases to [PyPI](https://pypi.org/project/gros-gatherer/) using \n`make setup_release` to install dependencies from `requirements-release.txt` \nand `make release` which performs multiple checks: unit tests, typing, lint and \nversion number consistency. The release files are also published on \n[GitHub](https://github.com/grip-on-software/data-gathering/releases) and from \nthere are archived on [Zenodo](https://zenodo.org/doi/10.5281/zenodo.10911861). \nNoteworthy changes to the modules are added to the \n[changelog](https://gros.liacs.nl/data-gathering/changelog.html).\n\n## License\n\nData gathering scripts and modules are licensed under the Apache 2.0 License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrip-on-software%2Fdata-gathering","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrip-on-software%2Fdata-gathering","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrip-on-software%2Fdata-gathering/lists"}