{"id":27962323,"url":"https://github.com/thoughtworks/recce","last_synced_at":"2025-07-24T21:14:58.312Z","repository":{"id":59145483,"uuid":"407469297","full_name":"thoughtworks/recce","owner":"thoughtworks","description":"Server-based database reconciliation tool for developers","archived":false,"fork":false,"pushed_at":"2025-05-01T13:49:50.000Z","size":1964,"stargazers_count":25,"open_issues_count":19,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-05-04T22:15:02.450Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thoughtworks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-17T08:44:54.000Z","updated_at":"2025-05-01T13:49:46.000Z","dependencies_parsed_at":"2023-11-14T11:27:55.381Z","dependency_job_id":"c711a81c-8e0e-4404-a96b-152be4f31d4d","html_url":"https://github.com/thoughtworks/recce","commit_stats":null,"previous_names":["thoughtworks/recce"],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thoughtworks%2Frecce","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thoughtworks%2Frecce/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thoughtworks%2Frecce/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thoughtworks%2Frecce/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thoughtworks","download_url":"https://codeload.github.com/thoughtworks/recce/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252941277,"owners_count":21828842,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-07T19:20:23.854Z","updated_at":"2025-05-07T19:20:24.413Z","avatar_url":"https://github.com/thoughtworks.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Recce Server\n\n[![Build Status](https://github.com/thoughtworks/recce/actions/workflows/build.yml/badge.svg)](https://github.com/thoughtworks/recce/actions/workflows/build.yml)\n[![Coverage](https://img.shields.io/codecov/c/github/thoughtworks/recce.svg)](https://codecov.io/gh/thoughtworks/recce)\n[![License](https://img.shields.io/github/license/thoughtworks/recce.svg)](https://opensource.org/licenses/Apache-2.0)\n\nRecce (_rek-ee_) is a database reconciliation tool **_for developers_**.\n\nIt is intended to make it easier to reconcile arbitrary datasets between **source** and **target** database schemas **on an\nongoing basis** using configured SQL expressions without writing code.\n\nYou can read more about the ideas behind Recce at [DESIGN.md](docs/DESIGN.md) or in the [FAQ](#faq).\n\n## Features\n* 🔥 Trigger reconciliation of source and target database schemas using **simple SQL expressions**\n* 🔒 **Privacy** - no sensitive data stored or output by default. By default, Recce compares **hashes** of data.\n* ✏️ **Flexible configuration** to map groups of data to reconcile to different datasources and databases \n* 🤝 Supports connectivity to **MySQL**, **Postgres**, **MSSQL**, **MariaDB**, **AWS Aurora RDS MySQL**, **Oracle** databases\n* 💪🏾 **Database agnostic** when comparing database types - as long as it is coerced to a comparable JVM type consistently, it will be hashed identically\n* ⏰ **Schedule** reconciliations for low-traffic periods against your data sources\n\n## Table of contents\n\u003c!-- ToC auto-populated via https://github.com/ekalinin/github-markdown-toc --\u003e\n\u003c!--ts--\u003e\n* [Recce Server](README.md#recce-server)\n   * [Features](README.md#features)\n   * [Table of contents](README.md#table-of-contents)\n* [Getting Started](README.md#getting-started)\n* [Configuration](README.md#configuration)\n   * [Configuring Recce itself](README.md#configuring-recce-itself)\n   * [Database configuration conveniences](README.md#database-configuration-conveniences)\n   * [Adding additional configuration files](README.md#adding-additional-configuration-files)\n   * [Configuring authentication](README.md#configuring-authentication)\n   * [Configuring datasources](README.md#configuring-datasources)\n      * [Driver/database specific configuration](README.md#driverdatabase-specific-configuration)\n      * [Customising datasource pooling](README.md#customising-datasource-pooling)\n   * [Configuring datasets](README.md#configuring-datasets)\n   * [Writing dataset queries](README.md#writing-dataset-queries)\n      * [The MigrationKey](README.md#the-migrationkey)\n      * [Ordering of columns](README.md#ordering-of-columns)\n      * [Handling differences](README.md#handling-differences)\n      * [Huge datasets and aggregates](README.md#huge-datasets-and-aggregates)\n* [Development](README.md#development)\n* [FAQ](README.md#faq)\n\u003c!--te--\u003e\n\n# Getting Started\n\nRecce is currently [published and available as a container image via GHCR](https://github.com/thoughtworks/recce/pkgs/container/recce-server).\n\nHowever, it also\n* requires its own Postgres database\n* ...and to be practically useful you will want to configure it with connectivity to some data sources you wish to reconcile.\n\nThe below options require only JDK 11+ and Docker (or equivalent) installed locally.\n\n1. **Run Recce** locally\n    * Either **Build** locally with an [example](./examples) source and target database (More info at [DEVELOPMENT.md](./DEVELOPMENT.md)).\n    ```shell\n    ./batect -f examples/batect-petshop-mariadb.yml run\n    ```\n    * **Or use pre-validated Docker image** locally, using this repository only for setting up a DB for Recce, and an example scenario.\n    ```shell\n    # Run in one shell - starts a DB for Recce, and an example scenario\n    ./batect -f examples/batect-petshop-mariadb.yml run-deps\n  \n    # Run in another shell - runs Recce\n    docker run -p 8080:8080 \\\n      -v $(pwd)/examples/scenario/petshop-mariadb:/examples/scenario/petshop-mariadb \\\n      -e AUTH_USERNAME=admin -e AUTH_PASSWORD=admin \\\n      -e MICRONAUT_CONFIG_FILES=/examples/scenario/petshop-mariadb/application-petshop-mariadb.yml \\\n      -e DATABASE_HOST=host.docker.internal \\\n      -e R2DBC_DATASOURCES_SOURCE_URL=r2dbc:pool:mariadb://host.docker.internal:8000/db \\\n      -e R2DBC_DATASOURCES_TARGET_URL=r2dbc:pool:mariadb://host.docker.internal:8001/db \\\n      ghcr.io/thoughtworks/recce-server:latest\n    ```\n    \n2. **Explore** and trigger runs via Recce's APIs, accessible via interactive UI at http://localhost:8080/rapidoc/. \n    Some non-exhaustive examples are included below, but fuller documentation is available via the UI. \n    Recce uses basic authentication to protect its API endpoints. In the example scenario, both username and password are set to \"admin\".\n \n    * **Synchronously trigger** a run, waiting for it to complete via [UI](http://localhost:8080/rapidoc/#post-/runs) _or_\n      ```shell\n      curl -X POST http://localhost:8080/runs -H 'Content-Type: application/json' -d '{ \"datasetId\": \"categories\" }' -u \"admin:admin\"\n      ``` \n      \u003cdetails\u003e\n        \u003csummary\u003eExpand example results\u003c/summary\u003e\n\n        ```json\n        {\n          \"completedDurationSeconds\": 0.736356642,\n          \"completedTime\": \"2021-12-07T10:37:35.469576795Z\",\n          \"createdTime\": \"2021-12-07T10:37:34.733220153Z\",\n          \"datasetId\": \"categories\",\n          \"id\": 35,\n          \"summary\": {\n            \"bothMatchedCount\": 1,\n            \"bothMismatchedCount\": 0,\n            \"source\": {\n              \"meta\": {\n                \"cols\": [\n                  {\n                    \"javaType\": \"String\",\n                    \"name\": \"MigrationKey\"\n                  },\n                  {\n                    \"javaType\": \"Long\",\n                    \"name\": \"count(distinct category)\"\n                  }\n                ]\n              },\n              \"onlyHereCount\": 0,\n              \"totalCount\": 1\n            },\n            \"target\": {\n              \"meta\": {\n                \"cols\": [\n                  {\n                    \"javaType\": \"String\",\n                    \"name\": \"MigrationKey\"\n                  },\n                  {\n                    \"javaType\": \"Long\",\n                    \"name\": \"count(*)\"\n                  }\n                ]\n              },\n              \"onlyHereCount\": 0,\n              \"totalCount\": 1\n            },\n            \"totalCount\": 1\n          }\n        }\n        ```\n     \u003c/details\u003e\n   \n    * **Retrieve details of an individual run** by ID for a dataset via [UI](http://localhost:8080/rapidoc/#get-/runs/-runId-), _or_\n      ```shell\n      curl 'http://localhost:8080/runs/35' -u \"admin:admin\"\n      ```\n    * **Retrieve details of recent runs** for a dataset via [UI](http://localhost:8080/rapidoc/#get-/runs), _or_\n      ```shell\n      curl 'http://localhost:8080/runs?datasetId=categories' -u \"admin:admin\"\n      ```\n\n# Configuration\n\nRecce is configured by adding **datasources** and **datasets** that you wish to reconcile.\n\nYou can manage configuration as multiple files; or one single file. They will be merged together at runtime.\n\n## Configuring Recce itself\n\nAs a Micronaut application, much of Recce's configuration is open for hacking and can be [expressed in multiple ways](https://docs.micronaut.io/latest/guide/#propertySource).\n\nRecce-specific configuration (as opposed to all possible generic Micronaut configuration) is documented within the [default config](src/main/resources/application.yml).\n\n## Database configuration conveniences\n\nTo make it easier to configure Recce's own DB, some dedicated properties are respected to construct the appropriate URL\nincluding connection pooling.\n\n| Env value         | Default     | Description                                      |\n|-------------------|-------------|--------------------------------------------------|\n| DATABASE_HOST     | `localhost` | Host your Postgres DB is on                      |\n| DATABASE_PORT     | `9000`      | Port your Postgres DB is on                      |\n| DATABASE_NAME     | `db`        | The name of the logical database within Postgres |\n| DATABASE_USERNAME | `user`      | Username to connect with                         |\n| DATABASE_PASSWORD | `password`  | Password to connect with                         |\n\nIn the conventional Micronaut way, additional settings can be configured using\n* `R2DBC_DATASOURCES_DEFAULT_*` environment variables, _or_\n* `r2dbc.datasources.default.*` Java system properties _or_\n* Merging in an additional YAML configuration file (see below)\n\n## Adding additional configuration files\n\nAs a Micronaut application, [configuration can be externalised](https://docs.micronaut.io/latest/guide/#propertySource) in many ways.\n\nHowever, the recommended way to add additional configuration **for your own datasources and datasets** to reconcile is to \nmount a volume with your configuration and set `MICRONAUT_CONFIG_FILES` to point Recce at your additional configuration \nwhich will be merged into the base configuration. This allows you to separate and manage configuration as appropriate\nfor your environment.\n\n```shell\nmkdir -p my-dataset-configs\ntouch my-dataset-configs/config1.yml my-dataset-configs/config2.yml\n\n## Run latest from docker\ndocker run -p 8080:8080 \\\n  -v $(pwd)/my-dataset-configs:/config \\\n  -e MICRONAUT_CONFIG_FILES=/config/config1.yml,/config/config2.yml \\\n  ghcr.io/thoughtworks/recce-server:latest\n```\n\n## Configuring authentication\n\nRecce supports only basic authentication for API endpoints out of the box. The credentials used in basic authentication\nto protect the API endpoints can be configured in the `auth` block of your config file.\n\n```yaml\nauth:\n  username: some-username\n  password: some-password\n```\n\nor read from the environment:\n```bash\nexport AUTH_USERNAME=some-username\nexport AUTH_PASSWORD=some-password\n```\n\n## Configuring datasources\n\nArbitrary #s of data sources can be configured in the `r2dbc.datasources` block of your config file.\n\n```yaml\nr2dbc:\n  datasources:\n    my-source-db: # Name your datasource anything you want, other than \"default\"\n      # R2DBC URL for your database r2dbc:pool:DB_TYPE://DB_HOST:DB_PORT/DB_NAME\n      # DB_TYPE supported = mysql|postgresql|mssql|mariadb|oracle\n      url: r2dbc:pool:mysql://source-db:3306/db\n      username: user\n      password: password\n    my-target-db:\n      url: r2dbc:pool:mysql://target-db:3306/db\n      username: user\n      password: password\n```\n\n### Driver/database specific configuration\n\nFor configuration specific to a given driver/DB you can consult their documentation. Usually additional settings needs to be in the `options:` block; or sometimes inside the connection URL.\n* [MySQL](https://github.com/mirromutth/r2dbc-mysql#configuration-items)\n* [Postgres](https://github.com/pgjdbc/r2dbc-postgresql#getting-started)\n* [MS SQL Server](https://github.com/r2dbc/r2dbc-mssql#getting-started)\n* [MariaDB](https://github.com/mariadb-corporation/mariadb-connector-r2dbc#connection-options) (can also be used for MySQL, as well as AWS RDS Aurora MySQL)\n* [Oracle](https://github.com/oracle/oracle-r2dbc#connection-creation)\n\n### Customising datasource pooling\n\nBy default, Recce is deployed with [r2dbc-pool](https://github.com/r2dbc/r2dbc-pool) to manage connection pooling to data sources. If you remove `:pool` from the URL, this will be disabled.\n\nYou can thus customise the connection pool size, etc\n```yaml\nr2dbc:\n  datasources:\n    my-source-db:\n      url: r2dbc:pool:mysql://source-db:3306/db\n      username: user\n      password: password\n      options: # Connection Pool options\n        initialSize: 1\n        maxSize: 5\n        maxLifeTime: 5\n        # etc, see https://github.com/r2dbc/r2dbc-pool\n```\n\n## Configuring datasets\n\nDatasets are the heart of your configuration. This tells Recce **what** and **how** to reconcile logically equivalent chunks of data.\n\nDatasets are groupings of data which\n* point to source and target **datasources** configured above\n* express queries against source and target that will should produce identical output if the datasets are to be considered **reconciled**\n* can use **normal SQL** to express an equivalent data representation between source and target schemas which accounts for intended differences in the way data has been structured and migrated\n* can be scheduled to run on regular intervals\n\n```yaml\nreconciliation:\n  datasets:\n    my-data-set: # Name your datasets however you would like\n      source:\n        # Reference to a datasource defined in `r2dbc.datasources`\n        datasourceRef: my-source-db\n        # Specify whether to load inline query or read from a query file\n        # If both `query` and `queryFile` are provided, inline query takes precedence\n        # If both query and queryFile are not specified,\n        # load the query statement from ${dataset-name}-${source-or-target}.sql in `queryFileBaseDir`\n        # Optional SQL query to evaluate against the source DB\n        query: \u003e\n          SELECT id AS MigrationKey, * FROM my-table\n        # Optional path to file containing query to evaluate against the source DB\n        queryFile: examples/scenario/\u003c{example-scenario}/queries/\u003c{example-scenario}-my-data-set-source.yml\n      target:\n        # Reference to a datasource defined in `r2dbc.datasources`\n        datasourceRef: my-target-db\n        # Specify whether to load inline query or read from a query file\n        # If both `query` and `queryFile` are provided, inline query takes precedence\n        # If both query and queryFile are not specified,\n        # load the query statement from ${dataset-name}-${source-or-target}.sql in `queryFileBaseDir`\n        # Optional SQL query to evaluate against the target DB\n        query: \u003e\n          SELECT id AS MigrationKey, * FROM my-table\n        # Optional path to file containing query to evaluate against the target DB\n        queryFile: examples/scenario/\u003c{example-scenario}/queries/\u003c{example-scenario}-my-data-set-target.yml\n      # Optional strategy for determining equivalence of column values\n      hashingStrategy: TypeLenient\n      # Optional scheduling of regular or one-of reconciliations\n      schedule:\n        # Must adhere to format https://docs.micronaut.io/latest/api/io/micronaut/scheduling/cron/CronExpression.html\n        # or https://crontab.guru/ (without seconds)\n        cronExpression: 0 0 * * *\n```\nFuller example Recce-specific configuration is [available here](src/main/resources/application.yml).\n\n## Writing dataset queries\n\nThe general philosophy of Recce is that differences between source and target are **best handled by the developer** using regular SQL queries.\n\n### The MigrationKey\n\nRecce needs to know which column represents a unique identifier for the row that should be consistent between `source` and `target` and implies these rows represent the **same entity**.\n\nTo do this, designate a column by naming it as `MigrationKey` (case insensitive)\n```sql\nSELECT natural_id AS MigrationKey, some, other, columns\nFROM my_table\n```\n\nRecce will complain if there is more than one column in your dataset with this name.\n\n### Ordering of columns\n\nCurrently Recce ignores names of columns _other than_ the `MigrationKey` column. That means that the **order of columns is critical and must match** between your two queries.\n\nIf the column in position 3 represents datum `X` in the `source` dataset, then the column in position 3 in the `target` dataset should also represent the same datum.\n\nThe data types of the columns need not match exactly; however if the values produce different hashes, this will lead to a row mismatch. For example an integer of `111` in source will produce a different hash to a string of `\"111\"` in target. If you want these values to be considered identical, you should use SQL to coerce the data types to match and express this intended difference.\n\nIf you would like to see a `nameBased` column matching option, consider adding your thoughts to #55.\n\n### Handling differences\n\nYou should look to handle differences in types and semantics using the SQL expressions and functions available on the relevant database platform.\n\nFor example, if your source database represented an enumeration using integers (`1, 2, 3` etc) whereas your target represented them using `VARCHAR`s, you would use a `CASE` statement to express this expected difference\n```sql\n-- source\nSELECT id AS MigrationKey, CASE WHEN enumerated = 1 THEN 'Completed' ELSE 'PENDING' END\nFROM my_table\n-- target\nSELECT id AS MigrationKey, enumerated_text\nFROM my_table\n```\n\n### Huge datasets and aggregates\n\nWhen reconciling very large data sets, it may not be feasible to do row-by-row, column-by-column comparisons. In such cases it may be sufficient to use aggregate queries to get an idea whether you have lost data. e.g\n\n```sql\n--- check #s of login audit records for a month of data by location and type\nSELECT user_location || user_type AS MigrationKey, count(*) AS Count\nFROM user_login_audit\nWHERE user_login_datetime \u003e= timestamp('2021-11-01 00:00:00') AND\n    user_login_datetime \u003c timestamp('2021-12-01 00:00:00')\nGROUP BY user_location, user_type\n```\n\n# Development\n\nSee [DEVELOPMENT.md](./DEVELOPMENT.md) to get started.\n\n# FAQ\n\nSee [FAQ.md](./FAQ.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthoughtworks%2Frecce","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthoughtworks%2Frecce","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthoughtworks%2Frecce/lists"}