{"id":24976584,"url":"https://github.com/datagouv/hydra","last_synced_at":"2026-01-30T11:46:06.153Z","repository":{"id":38305965,"uuid":"471059941","full_name":"datagouv/hydra","owner":"datagouv","description":"Async metadata crawler for data.gouv.fr","archived":false,"fork":false,"pushed_at":"2026-01-19T15:59:48.000Z","size":16116,"stargazers_count":7,"open_issues_count":17,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-01-19T21:41:54.168Z","etag":null,"topics":["government","government-data","open-data"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datagouv.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-03-17T16:32:12.000Z","updated_at":"2026-01-19T15:14:46.000Z","dependencies_parsed_at":"2023-11-24T14:29:20.218Z","dependency_job_id":"9c22b7e7-da31-489e-9c07-2a67b420a9d4","html_url":"https://github.com/datagouv/hydra","commit_stats":null,"previous_names":["datagouv/hydra","etalab/udata-hydra"],"tags_count":21,"template":false,"template_full_name":null,"purl":"pkg:github/datagouv/hydra","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datagouv%2Fhydra","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datagouv%2Fhydra/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datagouv%2Fhydra/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datagouv%2Fhydra/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datagouv","download_url":"https://codeload.github.com/datagouv/hydra/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datagouv%2Fhydra/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28911981,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-30T08:15:08.179Z","status":"ssl_error","status_checked_at":"2026-01-30T08:14:31.507Z","response_time":66,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["government","government-data","open-data"],"created_at":"2025-02-03T22:01:35.031Z","updated_at":"2026-01-30T11:46:06.146Z","avatar_url":"https://github.com/datagouv.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![udata-hydra](banner.png)\n\n# udata-hydra\n\n[![CircleCI](https://circleci.com/gh/datagouv/hydra.svg?style=svg)](https://circleci.com/gh/datagouv/hydra)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n`udata-hydra` is an async metadata crawler for [data.gouv.fr](https://www.data.gouv.fr).\n\nURLs are crawled via _aiohttp_, catalog and crawled metadata are stored in a _PostgreSQL_ database.\n\nSince it's called _hydra_, it also has mythical powers embedded:\n- analyse remote resource metadata over time to detect changes in the smartest way possible\n- if the remote resource is tabular (csv or excel-like), convert it to a PostgreSQL table, ready for APIfication, and to parquet to offer another distribution of the data\n- if the remote resource is a geojson, convert it to PMTiles to offer another distribution of the data\n- send crawl and analysis info to a udata instance\n\n## 🏗️ Architecture schema\n\nThe architecture for the full workflow is the following:\n\n![Full workflow architecture](docs/archi-idd-IDD.drawio.png)\n\n\nThe hydra crawler is one of the components of the architecture. It will check if resource is available, analyse the type of file if the resource has been modified, and analyse the CSV content. It will also convert CSV resources to database tables and send the data to a udata instance.\n\n![Crawler architecture](docs/hydra.drawio.png)\n\n## 📦 Dependencies\n\nThis project uses `libmagic`, which needs to be installed on your system, e.g.:\n\n`brew install libmagic` on MacOS, or `sudo apt-get install libmagic-dev` on Linux.\n\nThis project uses Python \u003e=3.11 and [uv](https://docs.astral.sh/uv/) to manage dependencies.\n\n## 🚀 Installation\n\n### With uv (recommended)\n```bash\nuv sync\n```\n\n### With pip\n```bash\npip3 install -e .\n```\n\n## 🖥️ CLI\n\n### Create database structure\n\nInstall udata-hydra dependencies and cli (see Installation section above), then migrate the DB with:\n\n`uv run udata-hydra migrate`\n\n### Load (UPSERT) latest catalog version from data.gouv.fr\n\n`uv run udata-hydra load-catalog`\n\n## 🕷️ Crawler\n\n`uv run udata-hydra-crawl`\n\nIt will crawl (forever) the catalog according to the config set in `config.toml`, with a default config in `udata_hydra/config_default.toml`.\n\n`BATCH_SIZE` URLs are queued at each loop run.\n\nThe crawler will start with URLs never checked and then proceed with URLs crawled before `CHECK_DELAYS` interval. It will then wait until something changes (catalog or time).\n\nThere's a by-domain backoff mechanism. The crawler will wait when, for a given domain in a given batch, `BACKOFF_NB_REQ` is exceeded in a period of `BACKOFF_PERIOD` seconds. It will retry until the backoff is lifted.\n\nIf an URL matches one of the `EXCLUDED_PATTERNS`, it will never be checked.\n\n## ⚙️ Worker\n\nA job queuing system is used to process long-running tasks. Launch the worker with the following command:\n\n`uv run rq worker -c udata_hydra.worker`\n\nTo monitor worker status:\n\n`uv run rq info -c udata_hydra.worker --interval 1`\n\nTo empty all the queues:\n\n`uv run rq empty -c udata_hydra.worker low default high`\n\n## 📊 CSV conversion to database\n\nConverted CSV tables will be stored in the database specified via `config.DATABASE_URL_CSV`. For tests it's the same database as for the catalog. Locally, `docker compose` will launch two distinct database containers.\n\n## 🧪 Tests\n\nTo run the tests, you need to launch the test database with `docker compose --profile test up -d`.\n\nMake sure the dependencies are installed (including dev dependencies) with `uv sync` (see Installation section above).\n\nThen you can run the tests with `uv run pytest`.\n\nTo run a specific test file, you can pass the path to the file to pytest, like this: `uv run pytest tests/test_file.py`.\n\nTo run a specific test function, you can pass the path to the file and the name of the function to pytest, like this: `uv run pytest tests/test_api/test_api_checks.py::test_get_latest_check`.\n\nIf you would like to see print statements as they are executed, you can pass the -s flag to pytest (`uv run pytest -s`). However, note that this can sometimes be difficult to parse.\n\n### 🎯 Tests coverage\n\nPytest automatically uses the `coverage` package to generate a coverage report, which is displayed at the end of the test run in the terminal.\nThe coverage is configured in the `pyproject.toml` file, in the `[tool.pytest.ini_options]` section.\nYou can also override the coverage report configuration when running the tests by passing some flags like `--cov-report` to pytest. See [the pytest-cov documentation](https://pytest-cov.readthedocs.io/en/latest/config.html) for more information.\n\n### 📈 Performance benchmarking\n\nHydra includes performance benchmarks to track and compare the performance of different operations on large files.\nThese benchmarks help identify performance regressions and improvements across different commits.\n\n#### How it works\n\nPerformance benchmarks are automatically executed on CI runners when pushing to the `benchmarks` branch. The benchmarks test three key operations:\n\n1. **CSV analysis** on large files using integrated test data\n2. **CSV to GeoJSON conversion** on large files using the `TEST_GEOCSV_URL` configured in CI\n3. **GeoJSON to PMTiles conversion** on large files using the `TEST_GEOCSV_URL` configured in CI\n\n#### Benchmark execution\n\nBenchmarks run on:\n- **[CircleCI](https://app.circleci.com/pipelines/github/datagouv/hydra)** ([workflow file](https://github.com/datagouv/hydra/blob/main/.circleci/config.yml)) - available as a manually triggerable pipeline after a push to `benchmarks` branch\n- **[GitHub Actions](https://github.com/datagouv/hydra/actions/workflows/benchmark.yml)** ([workflow file](https://github.com/datagouv/hydra/blob/main/.github/workflows/benchmark.yml)) - triggered automatically on pushes to `benchmarks` branch\n\nUsing two different CI systems allows for performance comparison across different environments and gives a way to avoid exhausting CI time limits.\n\n#### Metrics collected\n\nEach benchmark run collects **execution time** in seconds, **commit information** (hash, author) and **runner specifications** (CPU cores, memory, Python version, runner class), which are stored in [`.benchmarks/benchmarks.csv`](https://github.com/datagouv/hydra/blob/benchmarks/.benchmarks/benchmarks.csv).\n\nMore specifically:\n- `datetime` - when the test was run\n- `test_name` - which test was executed\n- `input_file` - URL or path of the input test data file used\n- `ci` - which CI system ran the test (github or circleci)\n- `execution_time_seconds` - performance measurement\n- `commit_author` - who made the commit\n- `commit_id` - the commit hash (7 characters)\n- `runner_class` - CircleCI/GitHub Actions runner type\n- `runner_cpu` - number of CPU cores\n- `runner_memory` - available memory in MB\n- `python_version` - Python version used\n\nResults are committed and pushed back to the `benchmarks` branch, creating a historical performance tracking dataset.\n\n#### Viewing results\n\nYou can view the current benchmark results at: [benchmarks.csv](https://github.com/datagouv/hydra/blob/benchmarks/.benchmarks/benchmarks.csv)\n\n#### Running benchmarks locally\n\nTo run performance benchmarks locally, you can use the CLI commands:\n\n```bash\n# Convert CSV to GeoJSON\nuv run udata-hydra convert-csv-to-geojson /path/to/large/file.csv\n\n# Convert GeoJSON to PMTiles\nuv run udata-hydra convert-geojson-to-pmtiles /path/to/large/file.geojson\n```\n\nThese commands allow you to test performance improvements locally before pushing to the benchmarks branch.\n\n## 🔌 API\n\nThe API will need a Bearer token for each request on protected endpoints (any endpoint that isn't a `GET`).\nThe token is configured in the `config.toml` file as `API_KEY`, and has a default value set in the `udata_hydra/config_default.toml` file.\n\nIf you're using hydra as an external service to receive resource events from [udata](https://github.com/opendatateam/udata), then udata needs to also configure this\nAPI key in its `udata.cfg` file:\n\n```python\n# Whether udata should publish the resource events\nPUBLISH_ON_RESOURCE_EVENTS = True\n# Where to publish the events\nRESOURCES_ANALYSER_URI = \"http://localhost:8000\"\n# The API key that hydra needs\nRESOURCES_ANALYSER_API_KEY = \"api_key_to_change\"\n```\n\n### 🚀 Run\n\n```bash\n# Install dependencies (see Installation section above)\nuv run adev runserver udata_hydra/app.py\n```\nBy default, the app will listen on `localhost:8000`.\nYou can check the status of the app with `curl http://localhost:8000/api/health`.\n\n### 🛣️ Routes/endpoints\n\nThe API serves the following endpoints:\n\n*Related to checks:*\n- `GET` on `/api/checks/latest?url={url}\u0026resource_id={resource_id}` to get the latest check for a given URL and/or `resource_id`\n- `GET` on `/api/checks/all?url={url}\u0026resource_id={resource_id}` to get all checks for a given URL and/or `resource_id`\n- `GET` on `/api/checks/aggregate?group_by={column}\u0026created_at={date}` to get checks occurrences grouped by a `column` for a specific `date`\n\n*Related to resources:*\n- `GET` on `/api/resources/{resource_id}` to get a resource in the DB \"catalog\" table from its `resource_id`\n- `POST` on `/api/resources` to receive a resource creation event from a source. It will create a new resource in the DB \"catalog\" table and mark it as priority for next crawling\n- `PUT` on `/api/resources/{resource_id}` to update a resource in the DB \"catalog\" table\n- `DELETE` on `/api/resources/{resource_id}` to delete a resource in the DB \"catalog\" table\n\n\u003e :warning: **Warning: the following routes are deprecated and will be removed in the future:**\n\u003e - `POST` on `/api/resource/created` -\u003e use `POST` on `/api/resources/` instead\n\u003e - `POST` on `/api/resource/updated` -\u003e use `PUT` on `/api/resources/` instead\n\u003e - `POST` on `/api/resource/deleted` -\u003e use `DELETE` on `/api/resources/` instead\n\n*Related to resources exceptions:*\n- `GET` on `/api/resources-exceptions` to get the list of all resources exceptions\n- `POST` on `/api/resources-exceptions` to create a new resource exception in the DB\n- `PUT` on `/api/resources-exceptions/{resource_id}` to update a resource exception in the DB\n- `DELETE` on `/api/resources-exceptions/{resource_id}` to delete a resource exception from the DB\n\n*Related to some status and health check:*\n- `GET` on `/api/status/crawler` to get the crawling status\n- `GET` on `/api/status/worker` to get the worker status\n- `GET` on `/api/stats` to get the crawling stats\n- `GET` on `/api/health` to get the API version number and environment\n\nYou may want to use a helper such as [Bruno](https://www.usebruno.com/) to handle API calls, in which case all the endpoints are ready to use [here](https://github.com/datagouv/api-calls).\nMore details about some endpoints are provided below with examples, but not for all of them:\n\n#### Get latest check\n\nWorks with `?url={url}` and `?resource_id={resource_id}`.\n\n```bash\n$ curl -s \"http://localhost:8000/api/checks/latest?url=http://opendata-sig.saintdenis.re/datasets/661e19974bcc48849bbff7c9637c5c28_1.csv\" | json_pp\n{\n   \"status\" : 200,\n   \"catalog_id\" : 64148,\n   \"deleted\" : false,\n   \"error\" : null,\n   \"created_at\" : \"2021-02-06T12:19:08.203055\",\n   \"response_time\" : 0.830198049545288,\n   \"url\" : \"http://opendata-sig.saintdenis.re/datasets/661e19974bcc48849bbff7c9637c5c28_1.csv\",\n   \"domain\" : \"opendata-sig.saintdenis.re\",\n   \"timeout\" : false,\n   \"id\" : 114750,\n   \"dataset_id\" : \"5c34944606e3e73d4a551889\",\n   \"resource_id\" : \"b3678c59-5b35-43ad-9379-fce29e5b56fe\",\n   \"headers\" : {\n      \"content-disposition\" : \"attachment; filename=\\\"xn--Dlimitation_des_cantons-bcc.csv\\\"\",\n      \"server\" : \"openresty\",\n      \"x-amz-meta-cachetime\" : \"191\",\n      \"last-modified\" : \"Wed, 29 Apr 2020 02:19:04 GMT\",\n      \"content-encoding\" : \"gzip\",\n      \"content-type\" : \"text/csv\",\n      \"cache-control\" : \"must-revalidate\",\n      \"etag\" : \"\\\"20415964703d9ccc4815d7126aa3a6d8\\\"\",\n      \"content-length\" : \"207\",\n      \"date\" : \"Sat, 06 Feb 2021 12:19:08 GMT\",\n      \"x-amz-meta-contentlastmodified\" : \"2018-11-19T09:38:28.490Z\",\n      \"connection\" : \"keep-alive\",\n      \"vary\" : \"Accept-Encoding\"\n   }\n}\n```\n\n#### Get all checks for an URL or resource\n\nWorks with `?url={url}` and `?resource_id={resource_id}`.\n\n```bash\n$ curl -s \"http://localhost:8000/api/checks/all?url=http://www.drees.sante.gouv.fr/IMG/xls/er864.xls\" | json_pp\n[\n   {\n      \"domain\" : \"www.drees.sante.gouv.fr\",\n      \"dataset_id\" : \"53d6eadba3a72954d9dd62f5\",\n      \"timeout\" : false,\n      \"deleted\" : false,\n      \"response_time\" : null,\n      \"error\" : \"Cannot connect to host www.drees.sante.gouv.fr:443 ssl:True [SSLCertVerificationError: (1, \\\"[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'www.drees.sante.gouv.fr'. (_ssl.c:1122)\\\")]\",\n      \"catalog_id\" : 232112,\n      \"url\" : \"http://www.drees.sante.gouv.fr/IMG/xls/er864.xls\",\n      \"headers\" : {},\n      \"id\" : 165107,\n      \"created_at\" : \"2021-02-06T14:32:47.675854\",\n      \"resource_id\" : \"93dfd449-9d26-4bb0-a6a9-ee49b1b8a4d7\",\n      \"status\" : null\n   },\n   {\n      \"timeout\" : false,\n      \"deleted\" : false,\n      \"response_time\" : null,\n      \"error\" : \"Cannot connect to host www.drees.sante.gouv.fr:443 ssl:True [SSLCertVerificationError: (1, \\\"[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'www.drees.sante.gouv.fr'. (_ssl.c:1122)\\\")]\",\n      \"domain\" : \"www.drees.sante.gouv.fr\",\n      \"dataset_id\" : \"53d6eadba3a72954d9dd62f5\",\n      \"created_at\" : \"2020-12-24T17:06:58.158125\",\n      \"resource_id\" : \"93dfd449-9d26-4bb0-a6a9-ee49b1b8a4d7\",\n      \"status\" : null,\n      \"catalog_id\" : 232112,\n      \"url\" : \"http://www.drees.sante.gouv.fr/IMG/xls/er864.xls\",\n      \"headers\" : {},\n      \"id\" : 65092\n   }\n]\n```\n\n#### Get checks occurrences grouped by a column for a specific date\n\nWorks with `?group_by={column}` and `?created_at={date}`.\n`date` should be a date in format `YYYY-MM-DD` or the default keyword `today`.\n\n```bash\n$ curl -s \"http://localhost:8000/api/checks/aggregate?group_by=domain\u0026created_at=today\" | json_pp\n[\n  {\n    \"value\": \"www.geo2france.fr\",\n    \"count\": 4\n  },\n  {\n    \"value\": \"static.data.gouv.fr\",\n    \"count\": 4\n  },\n  {\n    \"value\": \"grandestprod.data4citizen.com\",\n    \"count\": 3\n  },\n  {\n    \"value\": \"www.datasud.fr\",\n    \"count\": 2\n  },\n  {\n    \"value\": \"koumoul.com\",\n    \"count\": 2\n  },\n  {\n    \"value\": \"opendata.aude.fr\",\n    \"count\": 2\n  },\n  {\n    \"value\": \"departement-ain.opendata.arcgis.com\",\n    \"count\": 2\n  },\n  {\n    \"value\": \"opendata.agglo-larochelle.fr\",\n    \"count\": 1\n  }\n]\n```\n\n#### Adding a resource exception\n\n```bash\n$ curl   -X POST http://localhost:8000/api/resources-exceptions \\\n         -H 'Authorization: Bearer \u003cmyAPIkey\u003e' \\\n         -d '{\n            \"resource_id\": \"123e4567-e89b-12d3-a456-426614174000\",\n            \"table_indexes\": {\n                  \"siren\": \"index\"\n            },\n            \"comment\": \"This is a comment for the resource exception.\"\n         }'\n```\n\n...or, if you don't want to add table indexes and a comment:\n```bash\n$ curl  -X POST localhost:8000/api/resources-exceptions \\\n        -H 'Authorization: Bearer \u003cmyAPIkey\u003e' \\\n        -d '{\"resource_id\": \"f868cca6-8da1-4369-a78d-47463f19a9a3\"}'\n```\n\n#### Updating a resource exception\n\n```bash\n$ curl   -X PUT http://localhost:8000/api/resources-exceptions/f868cca6-8da1-4369-a78d-47463f19a9a3 \\\n         -H \"Authorization: Bearer \u003cmyAPIkey\u003e\" \\\n         -d '{\n            \"table_indexes\": {\n                  \"siren\": \"index\",\n                  \"code_postal\": \"index\"\n            },\n            \"comment\": \"Updated comment for the resource exception.\"\n         }'\n```\n\n#### Deleting a resource exception\n\n```bash\n$ curl  -X DELETE http://localhost:8000/api/resources-exceptions/f868cca6-8da1-4369-a78d-47463f19a9a3 \\\n        -H \"Authorization: Bearer \u003cmyAPIkey\u003e\"\n```\n\n#### Get crawling status\n\n```bash\n$ curl -s \"http://localhost:8000/api/status/crawler\" | json_pp\n{\n   \"fresh_checks_percentage\" : 0.4,\n   \"pending_checks\" : 142153,\n   \"total\" : 142687,\n   \"fresh_checks\" : 534,\n   \"checks_percentage\" : 0.4,\n   \"resources_statuses_count\": {\n      \"null\": 195339,\n      \"BACKOFF\": 0,\n      \"CRAWLING_URL\": 0,\n      \"TO_ANALYSE_RESOURCE\": 1,\n      \"ANALYSING_RESOURCE\": 0,\n      \"TO_ANALYSE_CSV\": 0,\n      \"ANALYSING_CSV\": 0,\n      \"INSERTING_IN_DB\": 0,\n      \"CONVERTING_TO_PARQUET\": 0\n  }\n}\n```\n\n#### Get worker status\n\n```bash\n$ curl -s \"http://localhost:8000/api/status/worker\" | json_pp\n{\n   \"queued\" : {\n      \"default\" : 0,\n      \"high\" : 825,\n      \"low\" : 655\n   }\n}\n```\n\n#### Get crawling stats\n\n```bash\n$ curl -s \"http://localhost:8000/api/stats\" | json_pp\n{\n   \"status\" : [\n      {\n         \"count\" : 525,\n         \"percentage\" : 98.3,\n         \"label\" : \"ok\"\n      },\n      {\n         \"label\" : \"error\",\n         \"percentage\" : 1.3,\n         \"count\" : 7\n      },\n      {\n         \"label\" : \"timeout\",\n         \"percentage\" : 0.4,\n         \"count\" : 2\n      }\n   ],\n   \"status_codes\" : [\n      {\n         \"code\" : 200,\n         \"count\" : 413,\n         \"percentage\" : 78.7\n      },\n      {\n         \"code\" : 501,\n         \"percentage\" : 12.4,\n         \"count\" : 65\n      },\n      {\n         \"percentage\" : 6.1,\n         \"count\" : 32,\n         \"code\" : 404\n      },\n      {\n         \"code\" : 500,\n         \"percentage\" : 2.7,\n         \"count\" : 14\n      },\n      {\n         \"code\" : 502,\n         \"count\" : 1,\n         \"percentage\" : 0.2\n      }\n   ]\n}\n```\n\n## 🔗 Using Webhook integration\n\n**Set the config values**\n\nCreate a `config.toml` where your service and commands are launched, or specify a path to a TOML file via the `HYDRA_SETTINGS` environment variable. `config.toml` or equivalent will override values from `udata_hydra/config_default.toml`, lookup there for values that can/need to be defined.\n\n```toml\nUDATA_URI = \"https://dev.local:7000/api/2\"\nUDATA_URI_API_KEY = \"example.api.key\"\nSENTRY_DSN = \"https://{my-sentry-dsn}\"\n```\n\nThe webhook integration sends HTTP messages to `udata` when resources are analysed or checked to fill resources extras.\n\nRegarding analysis, there is a phase called \"change detection\". It will try to guess if a resource has been modified based on different criteria:\n- harvest modified date in catalog\n- content-length and last-modified headers\n- checksum comparison over time\n\nThe payload should look something like:\n\n```json\n{\n   \"analysis:content-length\": 91661,\n   \"analysis:mime-type\": \"application/zip\",\n   \"analysis:checksum\": \"bef1de04601dedaf2d127418759b16915ba083be\",\n   \"analysis:last-modified-at\": \"2022-11-27T23:00:54.762000\",\n   \"analysis:last-modified-detection\": \"harvest-resource-metadata\",\n}\n```\n\n## 🛠️ Development\n\n### 🐳 Docker compose\n\nA single `docker-compose.yml` file is provided with profiles to manage different environments:\n- Default services: `database` and `database-csv` (PostgreSQL containers for catalog/metadata and CSV conversion)\n- `test` profile: `test-database` (ephemeral test database)\n- `broker` profile: `broker` (Redis broker)\n\nUsage:\n- Development: `docker compose up -d` (or `docker compose --profile broker up -d` if Redis is needed)\n- Tests: `docker compose --profile test up -d` (broker not needed, queue functionality is mocked)\n- Broker only: `docker compose --profile broker up -d`\n\n### 📝 Logging \u0026 Debugging\n\nThe log level can be adjusted using the environment variable LOG_LEVEL.\nFor example, to set the log level to `DEBUG` when initializing the database, use `LOG_LEVEL=\"DEBUG\" udata-hydra init_db `.\n\n### 📋 Writing a migration\n\n1. Add a file named `migrations/{YYYYMMDD}_{description}.sql` and write the SQL you need to perform migration.\n2. `udata-hydra migrate` will migrate the database as needed.\n\n## 🚀 Deployment\n\n3 services need to be deployed for the full stack to run:\n- worker\n- api / app\n- crawler\n\nRefer to each section to learn how to launch them. The only differences from dev to prod are:\n- use `HYDRA_SETTINGS` env var to point to your custom `config.toml`\n- use `HYDRA_APP_SOCKET_PATH` to configure where aiohttp should listen to a [reverse proxy connection (eg nginx)](https://docs.aiohttp.org/en/stable/deployment.html#nginx-configuration) and use `udata-hydra-app` to launch the app server\n\n## 🤝 Contributing\n\nBefore contributing to the repository and making any PR, it is necessary to initialize the pre-commit hooks:\n```bash\npre-commit install\n```\nOnce this is done, code formatting and linting, as well as import sorting, will be automatically checked before each commit.\n\nIf you cannot use pre-commit, it is necessary to format, lint, and sort imports with [Ruff](https://docs.astral.sh/ruff/) before committing:\n```bash\nuv run ruff check --fix . \u0026\u0026 uv run ruff format .\n```\n\n### 🏷️ Releases and versioning\n\nThe release process uses the [`tag_version.sh`](tag_version.sh) script to create git tags, GitHub releases and update [CHANGELOG.md](CHANGELOG.md) automatically. Package version numbers are automatically derived from git tags using [setuptools_scm](https://github.com/pypa/setuptools_scm), so no manual version updates are needed in `pyproject.toml`.\n\n**Prerequisites**: [GitHub CLI](https://cli.github.com/) must be installed and authenticated, and you must be on the main branch with a clean working directory.\n\n```bash\n# Create a new release\n./tag_version.sh \u003cversion\u003e\n\n# Example\n./tag_version.sh 2.5.0\n\n# Dry run to see what would happen\n./tag_version.sh 2.5.0 --dry-run\n```\n\nThe script automatically:\n- Extracts commits since the last tag and formats them for CHANGELOG.md\n- Identifies breaking changes (commits with `!:` in the subject)\n- Creates a git tag and pushes it to the remote repository\n- Creates a GitHub release with the changelog content\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatagouv%2Fhydra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatagouv%2Fhydra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatagouv%2Fhydra/lists"}