{"id":15520718,"url":"https://github.com/proycon/codemeta-harvester","last_synced_at":"2025-08-22T23:21:50.801Z","repository":{"id":45668886,"uuid":"444871269","full_name":"proycon/codemeta-harvester","owner":"proycon","description":"Harvest and aggregate codemeta/schema.org software metadata from source repositories and service endpoints, automatically converting from known metadata schemes in the process","archived":false,"fork":false,"pushed_at":"2025-03-10T12:33:36.000Z","size":123,"stargazers_count":9,"open_issues_count":9,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-10T16:04:02.689Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/proycon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":"codemeta.json","zenodo":null}},"created_at":"2022-01-05T16:21:48.000Z","updated_at":"2025-03-07T15:18:24.000Z","dependencies_parsed_at":"2025-03-07T12:27:03.694Z","dependency_job_id":"a751362e-4493-4f20-9ed5-7fa05fbf0033","html_url":"https://github.com/proycon/codemeta-harvester","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fcodemeta-harvester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fcodemeta-harvester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fcodemeta-harvester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proycon%2Fcodemeta-harvester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/proycon","download_url":"https://codeload.github.com/proycon/codemeta-harvester/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249110320,"owners_count":21214308,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T10:29:01.996Z","updated_at":"2025-04-15T16:41:43.921Z","avatar_url":"https://github.com/proycon.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Codemeta Harvester\n \n[![Project Status: Active -- The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)\n\nThis is a harvester for software metadata. It actively attempts to detect and\nconvert software metadata in source code repositories and converts this to a\nunified [codemeta](https://codemeta.github.io) representation.\n \nThe tool is implemented as a simple POSIX shell script that in turn invokes a number of tools to\ndo the actual work:\n\n* [codemetapy](https://github.com/proycon/codemetapy) - Conversion to codemeta from various other metadata formats\n* [cffconvert](https://github.com/citation-file-format/cff-converter-python) - Conversion from CITATION.cff to codemeta\n* [Software Metadata Extraction Framework (SOMEF)](https://github.com/KnowledgeCaptureAndDiscovery/somef) - Analyzes and extracts metadata from README.md and some other sources (optional dependency)\n\nA few simple additional metadata extractions methods, as simple shell scripts, have been implemented alongside the main script.\n\nThis harvester can be used for two purposes:\n\n1. to harvest a possibly large number of software projects, for instance to make them available in some kind of search portal.\n2. as a means to produce a `codemeta.json` file for your own project\n\n## Installation\n\nA docker container can be build as follows:\n\n``make docker``\n\nA pre-built container image can also be pulled from Docker Hub:\n\n``docker pull proycon/codemeta-harvester``\n\nAlternatively if you prefer not to use containers, you can also install the\nsoftware as follows:\n\n* Run ``make env`` to build a Python virtual environment in the `env` directory with the needed dependencies. This assumes you have a Python installation on your system.\n* Activate the environment with `. env/bin/activate` whenever you want to use it.\n* You will need to also ensure to install the following dependencies using your system's package manager\n    * git\n    * curl\n    * [dasel](https://github.com/TomWright/dasel)\n    * [recode](https://github.com/rrthomas/recode/)\n    * coreutils or busybox\n    * GNU Make\n    * GNU awk\n\nYou can use ``make devenv`` if you want to rely on the latest development release of codemetapy, rather than the latest\nstable version (this will create a `devenv/` dir instead of `env/`)\n\n## Usage: producing codemeta for your project\n\nIn your project directory, which ideally should be a git clone, you can just run codemeta-harvester to create a `codemeta.json`\nfile based on the files in your repository:\n\n`codemeta-harvester`\n\nYou probably use the docker container, then the syntax is as follows:\n\n`docker run -v $(pwd):/data proycon/codemeta-harvester`\n\nThe `-v` argument mounts your current working directory in the container, you may adapt it according to your needs.\n\nIf you want to regenerate an existing ``codemeta.json``, rather than use it as input which would be the default\nbehaviour, then add the ``--regen`` parameter. This overwrites any existing `codemeta.json`.\n\nThe harvester can make use of the Github/GitLab API to query metdata from GitHub/GitLab, but this allows only limited anonymous requests. Please set the\nenvironment variable `$GITHUB_TOKEN`/`$GITLAB_TOKEN` to a [github personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) / [gitlab personal access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html), if you use Docker you should pass it to the container using `--env-arg GITHUB_TOKEN=$GITHUB_TOKEN`/`--env-arg GITLAB_TOKEN=$GITLAB_TOKEN`.\n\n## Usage: harvesting metadata for various projects\n\nTo harvest and collect metadata from various projects, you need to create configuration files that tells the harvester\nwhere to look. These are simple ``yaml`` configuration files, one for each tool to harvest. They are put into a directory of your choice, and take the following format:\n\n```yaml\nsource: https://github.com/user/repo\nservices:\n    - https://example.org\n```\n\nThe ``source`` property specifies a single source code repository where the source code of the tool lives. This must be  *git* repository that is publicly accessible.  Note that you can specify only one repository here, choose the one that is representative for the software as a whole.\n\nThe ``services`` property lists zero or more URLs where the tool can be accessed as a service. This may be a web application, simple webpage, or some other form of webservice. For webservices, rather than enumerate all service endpoints individually, this should be pointed to a URL that provides itself provides a specification of endpoints, for example a URL serving a [OpenAPI](https://www.openapis.org/) specification. The information provided here will be expressed in the resulting `codemeta.json` through the ``isSourceCodeOf`` codemeta v3 property as described in issue [codemeta/codemeta#271](https://github.com/codemeta/codemeta/issues/271). This links the source code to specific instantiations of the software.\n\nAdditional properties you may specify:\n\n* ``root`` - The root path in the source code repository where to look for metadata. This can be set if the tool lives\n    as a sub-part of a larger repository. Defaults to the repository root.\n* `scandirs` - Sub directories to scan for metadata, in case not everything lives in the root directory.\n* `ref` - The git reference (a branch name of tag name) to use. You can set this if you want to harvest one particular\n    version. If not set, codemeta-harvester will check out the latest\n    version tag by default (this assumes you use some kind of semantic versioning for your tags). Only if no tags are present at all, it\n    falls back to using the `master` or `main` branch directly.\n* `tagprefix` - A prefix used for the git tags (only applicable in edge cases), the last part of the tag must still comply to a semantic versioning scheme.\n* `tagignore` - A regular expression (`grep -E`) of git tags to ignore (only applicable in edge cases), by default tags with \"alpha\", \"beta\" and like \"rc1\" will be ignored.\n\n\nPass the directory where you put your configurations (or a single configuration file) to codemeta-harvester as follows:\n\n`codemeta-harvester /path/to/your/configdir/`\n\nOr for Docker:\n\n`docker run -v /path/to/your/configdir/:/config -v $(pwd):/data proycon/codemeta-harvester /config`\n\n## Composition and precedence\n\nCodemeta-harvester relies [codemetapy](https://github.com/proycon/codemetapy) to combine different input sources into one `codemeta.json`, we call this *composition*. \n\nWhen a certain input source defines a property (on `schema:SoftwareSourceCode`), it will *overwrite* any values that were set earlier by previous sources. This entails that there is a certain order of precedence in which sources codemeta-harvester considers more important than others. The priority is roughly the following:\n\n1. ``codemeta.json``, if this file is provided, the harvest won't look at anything else (aside from the three exceptions mentioned at the end).\n2. ``codemeta-harvest.json``\n3. ``CITATION.cff``\n3. Language specific metadata from ``setup.py``, ``pyproject.toml``, ``pom.xml``, ``package.json`` and similar.\n4. files such as `LICENSE`,  `CONTRIBUTORS`, `AUTHORS`, `README`\n5. Information from git (e.g. contributors, git tag for version, date of first/last commit)\n6. Information from the github or gitlab API (e.g. project name/description)\n\nThree notable exceptions are:\n\n1. For development status, repostatus badge in the `README.md` *in the git master/main branch* takes precendence over all else (overriding whatever is in codemeta.json!)\n2. For maintainers, the parsing of `MAINTAINERS` *in the git master/main branch* is always taken into account (merged with anything in codemeta.json)\n3. If the harvester finds a version-specific DOI at [Zenodo](https://zenodo.org) for your software, it will always use that (overriding whatever is in codemeta.json)\n\n## Acknowledgement\n\nThis software was funded in the scope of the [CLARIAH-PLUS project](https://clariah.nl).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproycon%2Fcodemeta-harvester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproycon%2Fcodemeta-harvester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproycon%2Fcodemeta-harvester/lists"}