{"id":13650947,"url":"https://github.com/italia/publiccode-crawler","last_synced_at":"2026-02-10T10:04:01.595Z","repository":{"id":37734754,"uuid":"126154269","full_name":"italia/publiccode-crawler","owner":"italia","description":"publiccode.yml crawler for the Open Source software catalog of Developers Italia","archived":false,"fork":false,"pushed_at":"2024-07-05T09:48:06.000Z","size":16431,"stargazers_count":28,"open_issues_count":16,"forks_count":52,"subscribers_count":13,"default_branch":"main","last_synced_at":"2024-08-03T02:05:11.458Z","etag":null,"topics":["crawler","developers-italia","hacktoberfest","publiccode","publiccodeyml"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/italia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-21T09:27:28.000Z","updated_at":"2024-07-24T08:57:42.000Z","dependencies_parsed_at":"2024-03-14T08:25:40.919Z","dependency_job_id":"940d7fb3-8f08-44b0-b5da-41fddb91afc4","html_url":"https://github.com/italia/publiccode-crawler","commit_stats":{"total_commits":402,"total_committers":53,"mean_commits":7.584905660377358,"dds":0.6940298507462687,"last_synced_commit":"a91e213b9538bf6a530d00cc11cad2a1b43653d8"},"previous_names":["italia/developers-italia-backend"],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/italia%2Fpubliccode-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/italia%2Fpubliccode-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/italia%2Fpubliccode-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/italia%2Fpubliccode-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/italia","download_url":"https://codeload.github.com/italia/publiccode-crawler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223906066,"owners_count":17223003,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","developers-italia","hacktoberfest","publiccode","publiccodeyml"],"created_at":"2024-08-02T02:00:42.908Z","updated_at":"2025-12-14T20:42:26.644Z","avatar_url":"https://github.com/italia.png","language":"Go","funding_links":[],"categories":["💻 Publiccode"],"sub_categories":[],"readme":"# publiccode.yml crawler for the software catalog of Developers Italia\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/italia/publiccode-crawler/v4)](https://goreportcard.com/report/github.com/italia/publiccode-crawler/v4)\n[![Join the #publiccode channel](https://img.shields.io/badge/Slack%20channel-%23publiccode-blue.svg?logo=slack)](https://developersitalia.slack.com/messages/CAM3F785T)\n[![Get invited](https://slack.developers.italia.it/badge.svg)](https://slack.developers.italia.it/)\n\n## Description\n\nDevelopers Italia provides [a catalog of Free and Open Source](https://developers.italia.it/en/search)\nsoftware aimed to Public Administrations.\n\n`publiccode-crawler` retrieves the `publiccode.yml` files from the\nrepositories of publishers found in the [Developers Italia API](https://github.com/italia/developers-italia-api).\n\n## Setup and deployment processes\n\n`publiccode-crawler` can either run manually on the target machine or it can be deployed\nfrom a Docker container.\n\n### Manually configure and build\n\n1. Rename `config.toml.example` to `config.toml` and set the variables\n\n   \u003e **NOTE**: The application also supports environment variables in substitution\n   \u003e to config.toml file. Remember: \"environment variables get higher priority than\n   \u003e the ones in configuration file\"\n\n2. Build the binary with `go build`\n\n### Docker\n\nYou can build the Docker image using\n\n```console\ndocker build .\n```\n\nor use the image published to DockerHub:\n\n```console\ndocker run -it italia/publiccode-crawler\n```\n\n## Commands\n\n### `publiccode-crawler crawl`\n\nGets the list of publishers from `https://api.developers.italia.it/v1/publishers`\nand starts to crawl their repositories.\n\n### `publiccode-crawler crawl publishers*.yml`\n\nGets the list of publishers in `publishers*.yml` and starts to crawl\ntheir repositories.\n\n### `publiccode-crawler crawl-software \u003csoftware\u003e \u003cpublisher\u003e`\n\nCrawl just the software specified as parameter.\nIt takes the software URL and its publisher id as parameters.\n\nEx. `publiccode-crawler crawl-software https://api.developers.italia.it/v1/software/a2ea59b0-87cd-4419-b93f-00bed8a7b859 edb66b3d-3e36-4b69-aba9-b7c4661b3fdd`\n\n### Other commands\n\n* `crawler download-publishers` downloads organizations and repositories from\n  the [onboarding portal repository](https://github.com/italia/developers-italia-onboarding)\n  and saves them to a publishers YAML file.\n\n## See also\n\n* [developers-italia-api](https://github.com/italia/developers-italia-api): the API\n  used to store the results of the crawling\n* [publiccode-parser-go](https://github.com/italia/publiccode-parser-go): the Go\n  package for parsing publiccode.yml files\n\n## Authors\n\n[Developers Italia](https://developers.italia.it) is a project by\n[AgID](https://www.agid.gov.it/) and the\n[Italian Digital Team](https://teamdigitale.governo.it/), which developed the\ncrawler and maintains this repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitalia%2Fpubliccode-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fitalia%2Fpubliccode-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitalia%2Fpubliccode-crawler/lists"}