{"id":40393708,"url":"https://github.com/snyk/skemium","last_synced_at":"2026-01-20T13:00:43.316Z","repository":{"id":297250767,"uuid":"931139804","full_name":"snyk/skemium","owner":"snyk","description":"Generate and Compare Debezium CDC (Chance Data Capture) Avro Schema, directly from your Database.","archived":false,"fork":false,"pushed_at":"2026-01-15T16:56:22.000Z","size":759,"stargazers_count":23,"open_issues_count":2,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-01-15T18:56:44.563Z","etag":null,"topics":["comparison","database","debezium","kafka","postgres","schema","schema-registry","tool"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snyk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-02-11T19:27:46.000Z","updated_at":"2026-01-15T16:56:26.000Z","dependencies_parsed_at":"2025-08-28T12:11:44.136Z","dependency_job_id":"671d4f1c-4702-4f17-8afa-c83f20b44c1c","html_url":"https://github.com/snyk/skemium","commit_stats":null,"previous_names":["snyk/skemium"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/snyk/skemium","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snyk%2Fskemium","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snyk%2Fskemium/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snyk%2Fskemium/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snyk%2Fskemium/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snyk","download_url":"https://codeload.github.com/snyk/skemium/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snyk%2Fskemium/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28603402,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-20T12:01:53.233Z","status":"ssl_error","status_checked_at":"2026-01-20T12:01:46.545Z","response_time":117,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["comparison","database","debezium","kafka","postgres","schema","schema-registry","tool"],"created_at":"2026-01-20T13:00:24.535Z","updated_at":"2026-01-20T13:00:43.256Z","avatar_url":"https://github.com/snyk.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CI (GitHub Actions)](https://github.com/snyk/skemium/actions/workflows/ci.yaml/badge.svg)](https://github.com/snyk/skemium/actions/workflows/ci.yaml)\n[![Release Uber-JAR](https://github.com/snyk/skemium/actions/workflows/release-uberjar.yaml/badge.svg)](https://github.com/snyk/skemium/actions/workflows/release-uberjar.yaml)\n[![Release Binaries](https://github.com/snyk/skemium/actions/workflows/release-binaries.yaml/badge.svg)](https://github.com/snyk/skemium/actions/workflows/release-binaries.yaml)\n\n# Skemium\n\nGenerate and Compare [Debezium] Change Data Capture ([CDC]) [Avro] Schema.\n\nLeveraging [Debezium] and [Schema Registry] own codebases, each Table of a Database is mapped to 3 components:\n\n* _Key_ [Avro] schema: describes the `PRIMARY KEY` of the Table - `NULL` if not set\n* _Value_ [Avro] schema: describes each Row of the Table\n* _Envelope_ [Avro] schema: wrapper for the _Value_, used by Debezium to realize [CDC] when Producing to a Topic\n\n[Debezium CDC Source Connector] uses the _Key_ and the _Envelope_ schemas when producing to a Topic:\nthe former is used for the [Message _Key_][Kafka Message Key], the latter for the Message _Payload_.\n\n**Skemium** leverages those schemas to compare between evolutions of the originating Database Schema,\nand identifies compatibility issues executing the comparison logic implemented by [Schema Registry].\n\n**If you make changes to your Database Schema, and want to know if it's going to break your Debezium CDC production,\n`skemium` is the tool for you.**\n\n# Background\n\nIn our experience, the way [Debezium] works can catch users off guard in 2 major ways:\n\n1. Making changes to the _source_ Database Schema in ways that break [Schema Compatibility]\n2. Non-zero amount of time between making changes to the _source_ Database Schema, and that change being _captured_\n   by Debezium and published to [Schema Registry]\n\n**Avoiding the first is made much harder by the second!**\n\n## Delayed schema publishing\n\nThere is sometimes confusion between “making a DB Schema” vs “making a [Schema Registry] Schema” change:\n\n* the former happen when developers apply changes to their RDBMS: usually, before their application code start relying on the new schema\n* the latter happens when data is actually updated into one of the changed tables:\n  1. Debezium detects it (reading the [RDBMS WAL] and the `DESCRIBE TABLE` command)\n  2. Debezium's `Producer` attempts to create a new Schema version for the associated [schema subject]\n     1. _either_ fails if the change violates the configured [Schema Compatibility]\n     2. _or_ succeeds in publishing a new version for the [schema subject] creation was successful\n  3. Debezium's `Producer` resumes producing to the related Kafka Topic\n\nWhen 2.1. above happens, Debezium stops producing and in turns stops consuming the [RDBMS WAL]:\n\n1. Traffic from the RDBMS to Kafka halts (bad!)\n2. RDBMS storage fills up, as the WAL is not getting flushed (worse!)\n\n![Debezium \"delayed schema publishing\"](./images/debezium_delayed_schema_publishing.jpg)\n\n## The role of `skemium`\n\nSkemium's primary objective is to target this issue and empower developers to instrument their CI process\nto detect a _breakage_ of a [CDC] [schema subject] **as early as possible**.\n\nIdeally, when a PR is submitted and before any RDBMS schema has been changed in production, it should be possible to:\n\n* spin up a local instance of the RDBMS\n* apply the latest _desired_ DB Schema\n* execute `skemium generate` to obtain the corresponding Avro Schemas\n  (i.e. what would eventually land in [Schema Registry])\n* execute `skemium compare` to compare an existing copy of the Avro Schemas,\n  with the new one, applying the desired [Schema Compatibility]\n\n![Example CI that would detect \"schema breakage\" sooner](./images/skemium_schema_breakage_CI_detection.jpg)\n\n## What if a **_Breaking Schema Change_** is necessary?\n\nSometimes is going to be inevitable: you _need_ to make a change in your Database Schema,\nand a new version of the [schema subject] must be released - a version that breaks [Schema Compatibility].\n\nThis is beyond the scope of Skemium (for now?), but in those situations\nwhat you _can_ do is something along the lines of:\n\n* Make a coordinated plan with Consumer Services of the [CDC] Topic\n* Temporarily disable [Schema Compatibility] in [Schema Registry]\n* Let [Debezium] publish a new [schema subject] version\n* Restore [Schema Compatibility]\n\nThe details will depend on your specific circumstances, and your mileage may vary `¯\\_(ツ)_/¯`.\n\n## Generating the _correct_ Avro Schema\n\nSkemium **does not** implement its own schema extraction or serialization logic: it relies instead on the source code\nof [Debezium] and [Schema Registry]. Specifically, the following 2 packages do the bulk of the work:\n\n```xml\n\u003c!-- https://mvnrepository.com/artifact/io.debezium/debezium-core --\u003e\n\u003cdependency\u003e\n  \u003cgroupId\u003eio.debezium\u003c/groupId\u003e\n  \u003cartifactId\u003edebezium-core\u003c/artifactId\u003e\n  \u003cversion\u003e${ver.debezium}\u003c/version\u003e\n\u003c/dependency\u003e\n\u003c!-- https://mvnrepository.com/artifact/io.confluent/kafka-connect-avro-converter --\u003e\n\u003cdependency\u003e\n  \u003cgroupId\u003eio.confluent\u003c/groupId\u003e\n  \u003cartifactId\u003ekafka-connect-avro-converter\u003c/artifactId\u003e\n  \u003cversion\u003e${ver.kafka-connect-avro-converter}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nTo dig deeper, please look at the [`pom.xml`](./pom.xml).\n\n### Key classes \"borrowed\"\n\nSkemium design brings together [Debezium][Debezium source code] and\n[Schema Registry][Schema Registry source code] codebases, using the Apache [Avro] codebase as _lingua franca_:\n\n* `TableSchemaFetcher` extracts a `List\u003cio.debezium.relational.TableSchema\u003e`, using the Debezium's\n  RDBMS-specific connector source code to connect and query the DB schema\n* A `TableAvroSchemas` is created for each `io.debezium.relational.TableSchema`, invoking the provided methods\n  that extract _Key_, _Value_ and _Envelope_ as `org.apache.kafka.connect.data.Schema`\n* Each `org.apache.kafka.connect.data.Schema` is converted to `io.confluent.kafka.schemaregistry.avro.AvroSchema` via\n  the provided constructor\n* `io.confluent.kafka.schemaregistry.CompatibilityChecker` is used to compare current and next\n  versions of `io.confluent.kafka.schemaregistry.avro.AvroSchema`,\n  applying the desired `io.confluent.kafka.schemaregistry.CompatibilityLevel`\n\nThis gives us confidence that the generation of the Avro Schema, as well as the compatibility check, are the exact\nsame that Debezium will apply in production.\n\n# Usage\n\n## Binaries\n\nSkemium can be [compiled from source](#development), but for convenience we\n[release binaries](https://github.com/snyk/skemium/releases):\n\n* `skemium-${VER}-jar-with-dependencies`:\n  an [_uber jar_](https://support.sonatype.com/hc/en-us/articles/28958118202131-What-is-an-uber-jar),\n  easy to use in an environment where a JRE is present\n* `skemium-${VER}-${OS}-${ARCH}`: native binary, generated via [GraalVM] (see [below](#package-native-binary))\n\nAll binaries are generated when a new _tag_ is pushed to the `main` branch.\n\n## `generate` command\n\nThe `generate` command connects to a Database, reads its _Database Schema_ and coverts it to a [CDC] _Avro Schema_,\nusing [Debezium Avro Serialization].\n\nThe output is saved in a user given _output directory_. The directory will contain:\n\n* For each Table, a set of files following the naming structure `DB_NAME.DB_SCHEMA.DB_TABLE.EXTENSION`\n  * _Table Key_ schema file (`EXTENSION = .key.avsc`)\n  * _Table Value_ schema file (`EXTENSION = .val.avsc`)\n  * _Table Envelope_ schema file (`EXTENSION = .env.avsc`)\n  * The checksum of all 3 schema files above (`EXTENSION = .sha256`)\n* A metadata file named `.skemium.meta.json` ([schema](#generated-avro-schemas-metadata-file))\n\nFor example, if the database `example` contains 2 tables `user` and `address` in the database schema `public`, the output\ndirectory will look like:\n\n```shell\n$ tree example_schma_dir/\n\nexample_schma_dir/\n├── .skemium.meta.json\n├── example.public.address.env.avsc\n├── example.public.address.key.avsc\n├── example.public.address.sha256\n├── example.public.address.val.avsc\n├── example.public.user.env.avsc\n├── example.public.user.key.avsc\n├── example.public.user.sha256\n└── example.public.user.val.avsc\n```\n\n### Help\n\n\u003cdetails\u003e\n\u003csummary\u003eRun `skemium help generate` for usage instructions\u003c/summary\u003e\n\n```shell\n$ skemium help generate\n\nGenerates Avro Schema from Tables in a Database\n\nskemium generate [-v] -d=\u003cdbName\u003e -h=\u003chostname\u003e [--kind=\u003ckind\u003e] -p=\u003cport\u003e --password=\u003cpassword\u003e -u=\u003cusername\u003e [-s=\u003cdbSchemas\u003e[,\n                 \u003cdbSchemas\u003e...]]... [-t=\u003cdbTables\u003e[,\u003cdbTables\u003e...]]... [-x=\u003cdbExcludedColumns\u003e[,\u003cdbExcludedColumns\u003e...]]... [DIRECTORY_PATH]\n\nDescription:\n\nConnects to Database, finds schemas and tables,\nconverts table schemas to Avro Schemas, stores them in a directory.\n\nParameters:\n      [DIRECTORY_PATH]        Output directory\n                                Default: skemium-20250610-161400\n\nOptions:\n  -d, --database=\u003cdbName\u003e     Database name (env: DB_NAME)\n  -h, --hostname=\u003chostname\u003e   Database hostname (env: DB_HOSTNAME)\n      --kind=\u003ckind\u003e           Database kind (env: DB_KIND - optional)\n                                Values: POSTGRES\n                                Default: POSTGRES\n  -p, --port=\u003cport\u003e           Database port (env: DB_PORT)\n      --password=\u003cpassword\u003e   Database password (env: DB_PASSWORD)\n  -s, --schema=\u003cdbSchemas\u003e[,\u003cdbSchemas\u003e...]\n                              Database schema(s); all if omitted (env: DB_SCHEMA - optional)\n  -t, --table=\u003cdbTables\u003e[,\u003cdbTables\u003e...]\n                              Database table(s); all if omitted (fmt: DB_SCHEMA.DB_TABLE|DB_TABLE - env: DB_TABLE - optional)\n  -u, --username=\u003cusername\u003e   Database username (env: DB_USERNAME)\n  -v, --verbose               Logging Verbosity - use multiple -v to increase (default: ERROR)\n  -x, --exclude-column=\u003cdbExcludedColumns\u003e[,\u003cdbExcludedColumns\u003e...]\n                              Database table column(s) to exclude (fmt: DB_SCHEMA.DB_TABLE.DB_COLUMN - env: DB_EXCLUDED_COLUMN - optional)\n```\n\u003c/details\u003e\n\n### Filtering parts of the Database Schema\n\n`generate` offers options to filter parts of the Database Schema:\n\n* `-s | --schema`: only include selected schema(s) in the output\n  * Format: `-s DB_SCHEMA`\n  * Cardinality: `1..*`, either repeat option OR use comma (`,`) separator\n* `-t | --table`: only include selected table(s) in the output\n  * Format: `-t DB_TABLE` or `-t DB_SCHEMA.DB_TABLE`\n  * Cardinality: `1..*`, either repeat option OR use comma (`,`) separator\n  * Specificity:\n    * `DB_TABLE`: includes all tables of that name, across all selected schemas\n    * `DB_SCHEMA.DB_TABLE` includes only a table of that name, in the specific schema\n* `-x | --exclude-column`: exclude selected column(s) from the output\n  * Format: `-x DB_SCHEMA.DB_TABLE.DB_COLUMN`\n  * Cardinality: `1..*`, either repeat option OR use comma (`,`) separator\n\nWhere:\n\n* `DB_SCHEMA` is the name of a schema defined in the database (e.g. `public`)\n* `DB_TABLE` is the name of a table in the database\n* `DB_COLUMN` is the name of a column in the database\n\n## `compare` command\n\nThe `compare` command takes 2 directories (created via `generate`) containing the [CDC] _Avro Schema_ of a Database,\nand compares them applying the given [Schema Compatibility] type.\nThe directories are identified as _`CURRENT`_ and _`NEXT`_:\n\n* `CURRENT`: [CDC] Avro Schema of a Database, generated at time `T`\n* `NEXT`: [CDC] Avro Schema of the Database, generated at time `T+1`\n\n`compare` executes a table-by-table [Schema Compatibility] check, and reports on the result.\nExit Code will be `0` in case of success, `1` otherwise.\n\n### Schema changes and CI mode\n\nThe `compare` command reports (via `WARN` logging) if discrepancies are detected between `CURRENT` and `NEXT`:\n\n1. **Table additions/removals**: When tables are added or removed between the two schema directories\n2. **Schema modifications**: When existing table schemas are modified (even if the changes are compatible)\n\nThe flag `--ci-mode` can be used to _force_ a failure in case of **any** schema changes:\n\n* **Table discrepancies**: Additions or removals of tables\n* **Schema changes**: Modifications to existing table schemas (key, value, or envelope schemas)\n\n**This enhanced CI mode is ideally used in CI/CD automations** to ensure that:\n\n* Engineers are forced to acknowledge schema changes by updating their schema copies\n* All schema modifications are tracked and reviewed, even if they are compatible\n* Schema drift is prevented by requiring explicit acknowledgment of changes\n\n#### CI Mode Behavior\n\n| Scenario                    | Normal Mode                | CI Mode    |\n| --------------------------- | -------------------------- | ---------- |\n| No changes                  | ✅ Success                 | ✅ Success |\n| Compatible schema changes   | ✅ Success (with warnings) | ❌ Failure |\n| Incompatible schema changes | ❌ Failure                 | ❌ Failure |\n| Table additions/removals    | ✅ Success (with warnings) | ❌ Failure |\n\n### JSON output\n\nIf necessary, the output of `compare` can be stored in a output JSON file, using the `--output` option ([schema](#avro-schemas-comparison-result)).\n\n### Help\n\n\u003cdetails\u003e\n\u003csummary\u003eRun `skemium help compare` for usage instructions\u003c/summary\u003e\n\n```shell\n$ skemium help compare\n\nCompares Avro Schemas generated from Tables in a Database\n\nskemium compare [-iv] [-c=\u003ccompatibilityLevel\u003e] [-o=\u003coutput\u003e] CURR_SCHEMAS_DIR NEXT_SCHEMAS_DIR\n\nDescription:\n\nGiven 2 directories (CURRENT / NEXT) containing Avro Schemas of Database Tables,\ncompares them according to Compatibility Level.\n\nParameters:\n      CURR_SCHEMAS_DIR    Directory with the CURRENT Database Table schemas\n      NEXT_SCHEMAS_DIR    Directory with the NEXT Database Table schemas\n\nOptions:\n  -c, --compatibility=\u003ccompatibilityLevel\u003e\n                          Compatibility Level (env: COMPATIBILITY - optional)\n                          See: https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html\n                            Values: NONE, BACKWARD, BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE\n                            Default: BACKWARD\n  -i, --ci, --ci-mode     CI mode - Fail when schema changes are detected (table additions/removals or schema modifications) (env: CI_MODE - optional)\n                            Default: false\n  -o, --output=\u003coutput\u003e   Output file (JSON); overridden if exists (env: OUTPUT_FILE - optional)\n  -v, --verbose           Logging Verbosity - use multiple -v to increase (default: ERROR)\n```\n\u003c/details\u003e\n\n## `compare-files` command\n\nThe `compare-files` command compares two individual Avro Schema files (`.avsc`) directly, without requiring them to be part of a database-generated schema directory. This is useful for comparing bespoke or custom Avro schemas.\n\nThe command takes 2 Avro schema files and compares them applying the given [Schema Compatibility] type:\n\n* `CURR_SCHEMA_FILE`: The current version of the Avro schema file\n* `NEXT_SCHEMA_FILE`: The next version of the Avro schema file\n\n`compare-files` executes a [Schema Compatibility] check and reports on the result.\nExit Code will be `0` in case of success, `1` otherwise.\n\n### Including external type definitions\n\nWhen your schema references types defined in separate files, the Avro parser needs those type definitions to be available. Use the `--include-schema` / `-s` option to include additional schema files for type resolution.\n\nFor example, if your main schema references an `Issue` type defined in another file:\n\n```json\n{\n  \"name\": \"issues\",\n  \"type\": {\n    \"type\": \"array\",\n    \"items\": \"io.snyk.events.issue.Issue\"\n  }\n}\n```\n\nYou can include the `Issue` type definition:\n\n```shell\nskemium compare-files \\\n  --include-schema issue-type.avsc \\\n  base/main-schema.avsc \\\n  current/main-schema.avsc\n```\n\nMultiple type definitions can be included by repeating the option:\n\n```shell\nskemium compare-files \\\n  -s types/issue.avsc \\\n  -s types/project.avsc \\\n  base/event-schema.avsc \\\n  current/event-schema.avsc\n```\n\n### CI mode for file comparison\n\nSimilar to the `compare` command, `compare-files` supports a `--ci-mode` flag that will force a failure when any schema changes are detected, even if they are compatible. This ensures that all schema modifications are explicitly acknowledged.\n\n### JSON output\n\nThe output of `compare-files` can be stored in a JSON file using the `--output` option ([schema](#schema-file-comparison-result)).\n\n### Help\n\n\u003cdetails\u003e\n\u003csummary\u003eRun `skemium help compare-files` for usage instructions\u003c/summary\u003e\n\n```shell\n$ skemium help compare-files\n\nCompares two Avro Schema (.avsc) files\n\nskemium compare-files [-iv] [-c=\u003ccompatibilityLevel\u003e] [-o=\u003coutput\u003e]\n                      [-s=\u003cincludeSchemas\u003e]... CURR_SCHEMA_FILE NEXT_SCHEMA_FILE\n\nDescription:\n\nGiven 2 Avro Schema files (.avsc), compares them according to the specified Compatibility Level.\nThis command is designed for comparing bespoke/custom Avro schemas that are not generated from database tables.\n\nParameters:\n      CURR_SCHEMA_FILE    Path to the CURRENT Avro schema file (.avsc)\n      NEXT_SCHEMA_FILE    Path to the NEXT Avro schema file (.avsc)\n\nOptions:\n  -c, --compatibility=\u003ccompatibilityLevel\u003e\n                          Compatibility Level (env: COMPATIBILITY - optional)\n                          See: https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html\n                            Values: NONE, BACKWARD, BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE\n                            Default: BACKWARD\n  -i, --ci, --ci-mode     CI mode - Fail when schema changes are detected (env: CI_MODE - optional)\n                            Default: false\n  -o, --output=\u003coutput\u003e   Output file (JSON); overridden if exists (env: OUTPUT_FILE - optional)\n  -s, --include-schema=\u003cincludeSchemas\u003e\n                          Additional schema files to include for type resolution.\n                          These schemas are parsed before the main schema files,\n                          populating the parser's type registry to enable resolution\n                          of referenced types. Can be specified multiple times.\n  -v, --verbose           Logging Verbosity - use multiple -v to increase (default: ERROR)\n```\n\u003c/details\u003e\n\n## Logging verbosity\n\nThe option `-v | --verbose` (availabel for all commands) controls the logging verbosity.\nBy default, the logging level is `ERROR`.\nBut it can be increased by passing one or more `-v` options, to a maximum level of `TRACE`. The mapping is:\n\n```\n\u003cnone\u003e    -\u003e ERROR\n-v        -\u003e WARN\n-vv       -\u003e INFO\n-vvv      -\u003e DEBUG\n-vvvv     -\u003e TRACE\n-vvvvv... -\u003e TRACE\n```\n\n## Skemium's schemas\n\nIn a _kinda_ meta twist, Skemium command outputs _also_ have their own schemas.\n\n### Generated Avro Schemas metadata file\n\nThe output of the command `generate` includes the _metadata_ file `.skemium.meta.json`.\nThis summarises the output of the command, information about the moment in time, the local repository, and more.\n\nThe schema for this file is at [\u003cprj_root\u003e/schemas/skemium.generate.meta.avsc](./schemas/skemium.generate.meta.avsc).\n\n### Avro Schemas comparison result\n\nThe `compare` command can optionally save the result to an `--output` file.\nThis summarises what was compared, what [Schema Compatibility] was applied,\nand what issues (if any) were identified by the comparison.\n\nThe schema for this file is at [\u003cprj_root\u003e/schemas/skemium.compare.result.avsc](./schemas/skemium.compare.result.avsc).\n\n### Schema file comparison result\n\nThe `compare-files` command can optionally save the result to an `--output` file.\nThis summarises the two schema files that were compared, what [Schema Compatibility] was applied,\nand what incompatibilities (if any) were identified.\n\nThe schema for this file is at [\u003cprj_root\u003e/schemas/skemium.compare-files.result.avsc](./schemas/skemium.compare-files.result.avsc).\n\n# Interested in contributing?\n\nHere are some major features that we haven't had time to tackle yet:\n\n* [ ] Support for additional Databases (MySQL, MariaDB, MongoDB, Oracle, SQL Server, ...): currently only PostgreSQL is supported\n* [ ] Support connecting to GCP CloudSQL databases via\n  [dedicated `SocketFactory`](https://github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory)\n* [ ] Support custom key definition for a table, similar to what\n  [`message.key.columns`](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-message-key-columns)\n  allows when configuring Debezium. This is useful when an arbitrary key is desired or the table is missing a primary key.\n* [ ] Support for _generating_ and _comparing_ [JSON Schema]\n* [ ] Support for _generating_ and _comparing_ [Protobuf] schemas\n\nOf course, small contributions and bugfixes are also _very_ welcome.\n\n# Development\n\n## Requirements\n\n* Maven 3.9+\n* JDK 21+\n\nWe recommend using [asdf] to setup your local development, as it makes it very easy to get set up:\n\n```shell\nasdf install\n```\n\n## Build \u0026 Test\n\n```shell\n$ mvn clean package\n```\n\n### Package Uber-JAR\n\n```shell\n$ mvn clean package assembly:single -DskipTests\n```\n\n### Package Native Binary\n\n\u003e [!NOTE]\n\u003e For this option, you need to use [GraalVM]. If you are using [asdf], edit the `.tools-versions` file and uncomment\n\u003e the line to switch on `oracle-graalvm`; then, `asdf install`.\n\n```shell\n$ mvn clean package native:compile-no-fork -DskipTests\n```\n\n#### GraalVM `native-image` and choosing a machine type\n\n[`native-image`][native-image] is what [GraalVM] uses to compiles Java code into native executables.\nThe default options used by `native-image` are OK, but one determines the exact set of CPU features the compilation\nhappens against: `-march`\n(documented in the [`native-image` build options](https://www.graalvm.org/latest/reference-manual/native-image/overview/Options/#build-options)).\n\n\u003cdetails\u003e\n\u003csummary\u003eOutput of `-march=list` on `x86_64`\u003c/summary\u003e\n\n```shell\nOn AMD64, the following machine types are available:\n\n'compatibility'\n  CPU features: all of 'x86-64'\n'haswell'\n  CPU features: all of 'x86-64' + SSE3 + SSSE3 + SSE4_1 + SSE4_2 + POPCNT + LZCNT + AVX + AVX2 + AES + CLMUL + BMI1 + BMI2 + FMA\n'native'\n  CPU features: CX8 + CMOV + FXSR + HT + MMX + AMD_3DNOW_PREFETCH + SSE + SSE2 + SSE3 + SSSE3 + SSE4A + SSE4_1 + SSE4_2 + POPCNT + LZCNT + TSC + TSCINV_BIT + AVX + AVX2 + AES + ERMS + CLMUL + BMI1 + BMI2 + ADX + SHA + FMA + VZEROUPPER + FLUSH + FLUSHOPT + HV + RDTSCP + RDPID + FSRM + F16C + CET_SS\n'skylake'\n  CPU features: all of 'haswell' + AMD_3DNOW_PREFETCH + ADX + FLUSHOPT\n'skylake-avx512'\n  CPU features: all of 'skylake' + AVX512F + AVX512DQ + AVX512CD + AVX512BW + AVX512VL + CLWB\n'x86-64'\n  CPU features: CX8 + CMOV + FXSR + MMX + SSE + SSE2\n'x86-64-v1'\n  CPU features: all of 'x86-64'\n'x86-64-v2'\n  CPU features: all of 'x86-64-v1' + SSE3 + SSSE3 + SSE4_1 + SSE4_2 + POPCNT\n'x86-64-v3'\n  CPU features: all of 'x86-64-v2' + LZCNT + AVX + AVX2 + BMI1 + BMI2 + FMA\n'x86-64-v4'\n  CPU features: all of 'x86-64-v3' + AVX512F + AVX512DQ + AVX512CD + AVX512BW + AVX512VL\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eOutput of `-march=list` on `aarch64_64`\u003c/summary\u003e\n\n```shell\nOn AArch64, the following machine types are available:\n\n'armv8-a'\n  CPU features: FP + ASIMD\n'armv8.1-a'\n  CPU features: all of 'armv8-a' + CRC32 + LSE\n'compatibility'\n  CPU features: all of 'armv8-a'\n'native'\n  CPU features: FP + ASIMD + EVTSTRM + AES + PMULL + SHA1 + SHA2 + CRC32 + LSE + DCPOP + SHA3 + SHA512 + SVE + PACA + SVEBITPERM + SVE2\n\n\nThe option also supports one or more feature modifiers via the form '-march=arch{+[no]feature}*'. Example: 'armv8.1-a+lse' enables Large System Extension instructions.\nThe following feature modifiers are available: 'aes', 'lse', 'fp', 'simd'.\n```\n\u003c/details\u003e\n\nAfter a bit of experimentation, we determined that `-march=compatibility` was the best choice.\n\n## Taskfile (optional)\n\nIf you have taken advantage of the [asdf] setup (i.e. `asdf install`), you have already installed https://taskfile.dev/.\nMost frequently used _tasks_ are already configured in the [Taskfile](./Taskfile.yml). Give it a go!\n\n## Cutting a new release\n\nThe CI is setup to create a new release when a new tag in the format `vX.Y.Z` is pushed to the repository.\nWe use [gh-release] to handle most of the process (see [.github/workflows/release-*.yaml](./.github/workflows) for details),\nand the only manual step is to:\n\n1. ensure all PRs we aim to merge and release are merged\n2. figure out what the next version should be, by following [Semantic Versioning] principles: e.g. `$NEXT_VERSION`\n3. tag `main` and update `pom.xml` with the new version: `task tag-version -- $NEXT_VERSION`\n4. push the new tag to the `main` branch of the repo: `git push origin main --follow-tags --tags`\n5. Go to https://github.com/snyk/skemium/actions and confirm the release process has started\n\n# Credits\n\nAs any open source tool, this builds on the shoulders of the great work of others (see the [pom.xml](./pom.xml)).\n\nBut I want to especially thank 2 projects for the _core_ of the functionality:\n  * [Debezium], providing logic to extract database table schemas\n  * Confluent [Schema Registry], providing logic to convert to/from Avro Schemas\n\n# License\n\n[Apache 2.0](./LICENSE)\n\n---\n\n**Made with 💜 by Snyk**\n\n[Avro]: https://avro.apache.org/\n[CDC]: https://en.wikipedia.org/wiki/Change_data_capture\n[CI]: https://www.atlassian.com/continuous-delivery/continuous-integration\n[Debezium Avro Serialization]: https://debezium.io/documentation/reference/stable/configuration/avro.html\n[Debezium CDC Source Connector]: https://debezium.io/documentation/reference/stable/connectors/index.html\n[Debezium source code]: https://github.com/debezium/debezium\n[Debezium]: https://debezium.io/\n[GraalVM]: https://www.graalvm.org/\n[JSON Schema]: https://json-schema.org/\n[Kafka Message Key]: https://www.confluent.io/learn/kafka-message-key/\n[Protobuf]: https://protobuf.dev/\n[RDBMS WAL]: https://debezium.io/documentation/reference/stable/connectors/postgresql.html#how-the-postgresql-connector-works\n[Schema Compatibility]: https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html#compatibility-types\n[Schema Registry source code]: https://github.com/confluentinc/schema-registry\n[Schema Registry]: https://docs.confluent.io/platform/6.2/schema-registry/index.html\n[Semantic Versioning]: https://semver.org/\n[asdf]: https://asdf-vm.com/\n[gh-release]: https://github.com/softprops/action-gh-release\n[native-image]: https://www.graalvm.org/latest/reference-manual/native-image/\n[schema subject]: https://developer.confluent.io/courses/schema-registry/schema-subjects/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnyk%2Fskemium","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnyk%2Fskemium","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnyk%2Fskemium/lists"}