{"id":13501596,"url":"https://github.com/repology/repology-updater","last_synced_at":"2025-03-29T09:30:55.544Z","repository":{"id":11019299,"uuid":"68022568","full_name":"repology/repology-updater","owner":"repology","description":"Repology backend service to update repository and package data","archived":false,"fork":false,"pushed_at":"2024-10-29T21:13:34.000Z","size":4568,"stargazers_count":500,"open_issues_count":119,"forks_count":175,"subscribers_count":15,"default_branch":"master","last_synced_at":"2024-10-29T23:38:55.447Z","etag":null,"topics":["package","packages","repository","version"],"latest_commit_sha":null,"homepage":"https://repology.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/repology.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-09-12T15:36:56.000Z","updated_at":"2024-10-29T21:13:38.000Z","dependencies_parsed_at":"2023-10-03T00:49:07.170Z","dependency_job_id":"5b13e672-c43e-459d-8aaf-073f3910c3b3","html_url":"https://github.com/repology/repology-updater","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/repology%2Frepology-updater","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/repology%2Frepology-updater/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/repology%2Frepology-updater/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/repology%2Frepology-updater/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/repology","download_url":"https://codeload.github.com/repology/repology-updater/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246167035,"owners_count":20734377,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["package","packages","repository","version"],"created_at":"2024-07-31T22:01:42.942Z","updated_at":"2025-03-29T09:30:55.537Z","avatar_url":"https://github.com/repology.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Repology\n\n[![CI](https://github.com/repology/repology-updater/workflows/CI/badge.svg)](https://github.com/repology/repology-updater/actions/workflows/ci.yml)\n\nRepology is a service which monitors *a lot* of package repositories\nand other sources and aggregates data on software package versions,\nreporting new releases and packaging problems.\n\nThis repository contains Repology updater code, a backend service\nwhich updates the repository information. See also the\n[web application](https://github.com/repology/repology-webapp) code.\n\n## Dependencies\n\n  - [Python](https://www.python.org/) 3.11+\n  - Python module [Jinja2](http://jinja.pocoo.org/)\n  - Python module [libversion](https://pypi.python.org/pypi/libversion) (also requires [libversion](https://github.com/repology/libversion) C library)\n  - Python module [psycopg2](http://initd.org/psycopg/)\n  - Python module [pyyaml](http://pyyaml.org/)\n  - Python module [xxhash](https://github.com/ifduyue/python-xxhash)\n  - Python module [pydantic](https://pydantic-docs.helpmanual.io/)\n  - [PostgreSQL](https://www.postgresql.org/) 16.0+\n  - PostgreSQL extension [libversion](https://github.com/repology/postgresql-libversion)\n\nNeeded for fetching/parsing repository data:\n\n  - Python module [jsonslicer](https://pypi.org/project/jsonslicer/)\n  - Python module [lxml](http://lxml.de/)\n  - Python module [protobuf](https://github.com/protocolbuffers/protobuf)\n  - Python module [pyparsing](https://github.com/pyparsing/pyparsing)\n  - Python module [requests](http://python-requests.org/)\n  - Python module [rpm](http://rpm.org/) (comes with RPM package manager)\n  - Python module [rubymarshal](https://github.com/d9pouces/RubyMarshal)\n  - Python module sqlite3 (part of Python, sometimes packaged separately)\n  - Python module [tomli](https://pypi.org/project/tomli/)\n  - Python module [yarl](https://github.com/aio-libs/yarl)\n  - Python module [zstandard](https://pypi.org/project/zstandard/)\n  - [git](https://git-scm.com/)\n  - [rsync](https://rsync.samba.org/)\n\n### Development dependencies\n\nOptional, for doing HTML validation when running tests:\n  - Python module [pytidylib](https://pypi.python.org/pypi/pytidylib) and [tidy-html5](http://www.html-tidy.org/) library\n\nOptional, for checking schemas of configuration files:\n  - Python module [voluptuous](https://pypi.python.org/pypi/voluptuous)\n\nOptional, for python code linting:\n  - Python module [flake8](https://pypi.python.org/pypi/flake8)\n  - Python module [flake8-builtins](https://pypi.python.org/pypi/flake8-builtins)\n  - Python module [flake8-import-order](https://pypi.python.org/pypi/flake8-import-order)\n  - Python module [flake8-quotes](https://pypi.python.org/pypi/flake8-quotes)\n  - Python module [mypy](http://mypy-lang.org/)\n\n## Running\n\n### Preparing\n\nSince repology rules live in separate repository you'll need to\nclone it first. The location may be arbitrary, but `rules.d`\nsubdirectory is what default configuration file points to, so\nusing it is the most simple way.\n\n```shell\ngit clone https://github.com/repology/repology-rules.git rules.d\n```\n\n### Configuration\n\nFirst, you may need to tune settings which are shared by all repology\nutilities, such as directory for storing downloaded repository state\nor DSN (string which specifies how to connect to PostgreSQL database).\nSee `repology.conf.default` for default values, create `repology.conf`\nin the same directory to override them (don't edit `repology.conf.default`!)\nor specify path to alternative config via `REPOLOGY_SETTINGS`\nenvironment variable, or override settings via command line.\n\nBy default, repology uses `./_state` directory for storing raw and parsed\nrepository data and `repology/repology/repology` database/user/password\non localhost.\n\n### Creating the database\n\nFor the following steps you'll need to set up the database. Ensure\nPostgreSQL server is up and running, and execute the following\ncommands to create the database for repology:\n\n```shell\npsql --username postgres -c \"CREATE DATABASE repology\"\npsql --username postgres -c \"CREATE USER repology WITH PASSWORD 'repology'\"\npsql --username postgres -c \"GRANT ALL ON DATABASE repology TO repology\"\npsql --username postgres --dbname repology -c \"GRANT CREATE ON SCHEMA public TO PUBLIC\"\npsql --username postgres --dbname repology -c \"CREATE EXTENSION pg_trgm\"\npsql --username postgres --dbname repology -c \"CREATE EXTENSION libversion\"\n```\n\nin the case you want to change the credentials, don't forget to add\nthe actual ones to `repology.conf`.\n\n### Populating the database\n\nNote that you need more than 11GiB of disk space for Repology\nPostgreSQL database and additionally more than 11GiB space for raw\nand parsed repository data if you decide to run a complete update\nprocess.\n\n#### Option 1: use dump\n\nThe fastest and most simple way to fill the database would be to\nuse a database [dump](https://dumps.repology.org/) of main Repology\ninstance:\n\n```shell\ncurl -s https://dumps.repology.org/repology-database-dump-latest.sql.zst | unzstd | psql -U repology\n```\n\n#### Option 2: complete update\n\nAnother option would be to go through complete update process which\nincludes fetching and parsing all repository data from scratch and\npushing it to the database.\n\nFirst, init the database schema:\n\n```shell\n./repology-update.py --initdb\n```\n\nNote that this command drops all existing data in Repology database,\nif any. You only need to run this command once.\n\nNext, run the update process:\n\n```shell\n./repology-update.py --fetch --fetch --parse --database --postupdate\n```\n\nExpect it to take several hours the first time, subsequent updates\nwill be faster. You can use the same command to updated. Brief\nexplanation of options used here:\n\n  - `--fetch` tells the utility to fetch raw repository data\n    (download files, scrape websites, clone git repos) into state\n    directory. Note that it needs to be specified twice to allow\n\tupdating.\n  - `--parse` enables parsing downloaded data into internal format\n    which is also saved into state directory.\n  - `--database` pushes processed package data into the database.\n  - `--postupdate` runs optional cleanup tasks.\n\n## Documentation\n\n  - How to extend or fix [rules](https://github.com/repology/repology-rules/blob/master/README.md) for package matching\n  - How repology [compares versions](https://github.com/repology/libversion/blob/master/doc/ALGORITHM.md)\n\n## Author\n\n  - [Dmitry Marakasov](https://github.com/AMDmi3) \u003camdmi3@amdmi3.ru\u003e\n\n## License\n\nGPLv3 or later, see [COPYING](COPYING).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frepology%2Frepology-updater","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frepology%2Frepology-updater","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frepology%2Frepology-updater/lists"}