{"id":13880645,"url":"https://github.com/BemiHQ/bemi","last_synced_at":"2025-07-16T17:30:50.596Z","repository":{"id":220979832,"uuid":"751391341","full_name":"BemiHQ/bemi","owner":"BemiHQ","description":"Automatic data change tracking for PostgreSQL","archived":false,"fork":false,"pushed_at":"2025-07-02T20:14:58.000Z","size":3833,"stargazers_count":356,"open_issues_count":0,"forks_count":9,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-07-02T21:25:58.594Z","etag":null,"topics":["audit-log","audit-trail","change-data-capture","change-tracking","data-versioning","postgresql"],"latest_commit_sha":null,"homepage":"https://bemi.io","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BemiHQ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-02-01T14:20:27.000Z","updated_at":"2025-07-02T20:15:02.000Z","dependencies_parsed_at":"2024-07-12T16:32:09.438Z","dependency_job_id":"ab58e3a2-7663-4875-a9e9-14a724d314c5","html_url":"https://github.com/BemiHQ/bemi","commit_stats":{"total_commits":102,"total_committers":5,"mean_commits":20.4,"dds":"0.23529411764705888","last_synced_commit":"eaaeba06f84d9b96cbb8830893bd17a59736fd42"},"previous_names":["bemihq/bemi"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/BemiHQ/bemi","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BemiHQ%2Fbemi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BemiHQ%2Fbemi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BemiHQ%2Fbemi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BemiHQ%2Fbemi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BemiHQ","download_url":"https://codeload.github.com/BemiHQ/bemi/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BemiHQ%2Fbemi/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265527543,"owners_count":23782480,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["audit-log","audit-trail","change-data-capture","change-tracking","data-versioning","postgresql"],"created_at":"2024-08-06T08:03:20.844Z","updated_at":"2025-07-16T17:30:50.017Z","avatar_url":"https://github.com/BemiHQ.png","language":"TypeScript","readme":"\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://bemi.io\"\u003e\n    \u003cimg width=\"1201\" alt=\"bemi-banner\" src=\"https://docs.bemi.io/img/bemi-banner.png\"\u003e\n  \u003c/a\u003e\n\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://bemi.io\"\u003eWebsite\u003c/a\u003e\n    ·\n    \u003ca href=\"https://docs.bemi.io\"\u003eDocs\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/BemiHQ/bemi/issues/new\"\u003eReport Bug\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/BemiHQ/bemi/issues/new\"\u003eRequest Feature\u003c/a\u003e\n    ·\n    \u003ca href=\"https://discord.gg/mXeZ6w2tGf\"\u003eDiscord\u003c/a\u003e\n    ·\n    \u003ca href=\"https://twitter.com/BemiHQ\"\u003eTwitter\u003c/a\u003e\n    ·\n    \u003ca href=\"https://www.linkedin.com/company/bemihq/about\"\u003eLinkedIn\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n# Bemi\n\nBemi automatically tracks database changes ensuring 100% reliability and a comprehensive understanding of every change. It does it by connecting PostgreSQL's [Write-Ahead Log](https://www.postgresql.org/docs/current/wal-intro.html) (WAL) and implementing [Change Data Capture](https://en.wikipedia.org/wiki/Change_data_capture) (CDC) data pattern. Designed with simplicity and non-invasiveness in mind, Bemi operates in the background and doesn't require any alterations to your existing database tables.\n\n## Contents\n\n- [Highlights](#highlights)\n- [Use cases](#use-cases)\n- [System dependencies](#system-dependencies)\n- [Quickstart for local development](#quickstart-for-local-development)\n  - [Running with Docker](#running-with-docker)\n  - [Running with Devbox](#running-with-devbox)\n  - [Running natively](#running-natively)\n- [Contextualizing data changes](#contextualizing-data-changes)\n- [Architecture](#architecture)\n- [Testing](#testing)\n- [License](#license)\n\n## Highlights\n\n- Automatic and 100% reliable database change tracking\n- High performance without affecting runtime execution\n- Easy-to-use without changing table structures\n- Time travel querying and ability to easily filter changes\n- Optional application-specific context by using [ORM packages](https://docs.bemi.io/#supported-orms)\n\n## Use cases\n\nThere's a wide range of use cases that Bemi is built for! The tech was initially built as a compliance engineering system for fintech that supported $15B worth of assets under management, but has since been extracted into a general-purpose utility. Some use cases include:\n\n- **Audit Trails:** Use logs for compliance purposes or surface them to customer support and external customers.\n- **Streaming:** Real-time data stream to an application.\n- **Time Travel:** Retrieve historical data without implementing event sourcing.\n- **Troubleshooting:** Identify the root cause of application issues.\n- **Change Reversion:** Revert changes made by a user or rollback all data changes within an API request.\n- **Distributed Tracing:** Track changes across distributed systems.\n- **Testing:** Rollback or roll-forward to different application test states.\n- **Analyzing Trends:** Gain insights into historical trends and changes for informed decision-making.\n\n## System dependencies\n\n- Node.js\n- Java\n- [NATS server](https://github.com/nats-io/nats-server)\n\nAnd of course, you need a PostgreSQL database that you want to connect to to track data changes. Make sure your database has `SHOW wal_level;` returning `logical`. Otherwise, you need to run the following SQL command and restart your PostgreSQL server:\n\n```sql\nALTER SYSTEM SET wal_level = logical;\n```\n\nTo track both the \"before\" and \"after\" states on data changes, please run the following SQL command:\n\n```sql\nALTER TABLE [tracked_table_name] REPLICA IDENTITY FULL;\n```\n\n## Quickstart for local development\n\n### Running with Docker\n\nRun a Docker container that connects to your local PostgreSQL database:\n\n```sh\ndocker run \\\n  -e DB_HOST=host.docker.internal \\\n  -e DB_PORT=5432 \\\n  -e DB_NAME=[YOUR_DATABASE] \\\n  -e DB_USER=postgres \\\n  -e DB_PASSWORD=postgres \\\n  public.ecr.aws/bemi/dev:latest\n```\n\nReplace `DB_NAME` with your local database name. Note that `DB_HOST` pointing to `host.docker.internal` allows accessing `127.0.0.1` on your host machine if you run PostgreSQL outside Docker.\n\nNow try making some database changes like:\n\n```sql\nUPDATE _bemi_migrations SET executed_at = NOW() WHERE id = 1;\n```\n\nThis will add a new record in the `changes` table within the same database after a few seconds:\n\n```sql\nSELECT \"primary_key\", \"table\", \"operation\", \"before\", \"after\", \"context\", \"committed_at\" FROM changes;\n```\n\n### Running with Devbox\n\nYou can install all system dependencies manually or use [Devbox](https://github.com/jetpack-io/devbox) that relies on\n[Nix Packages](https://github.com/NixOS/nixpkgs) for providing isolated shells without containerization.\nRun a single command that will install Node.js with pnpm and NATS server:\n\n```sh\nmake worker-install\n```\n\nSet environment variables specifying connection settings for a PostgreSQL database you want to track and run a worker:\n\n```sh\nexport DB_HOST=127.0.0.1 DB_PORT=5432 DB_NAME=postgres DB_USER=postgres DB_PASSWORD=postgres\nmake worker-up\n```\n\nNow try making some database changes like:\n\n```sql\nUPDATE _bemi_migrations SET executed_at = NOW() WHERE id = 1;\n```\n\nThis will add a new record in the `changes` table within the same database after a few seconds.\n\n### Running natively\n\nAfter installing all system dependencies, install all project dependencies with Node.js:\n\n```sh\nmake worker-setup \u0026\u0026 cd worker \u0026\u0026 npm install\n```\n\nSet environment variables specifying connection settings for a PostgreSQL database you want to track run a worker as a single process with directly installed Node.js:\n\n\n```sh\nexport DB_HOST=127.0.0.1 DB_PORT=5432 DB_NAME=postgres DB_USER=postgres DB_PASSWORD=postgres\nnpm concurrently -- \"npm:up:*\"\n```\n\nNow try making some database changes like:\n\n```sql\nUPDATE _bemi_migrations SET executed_at = NOW() WHERE id = 1;\n```\n\nThis will add a new record in the `changes` table within the same database after a few seconds.\n\n## Contextualizing data changes\n\nOptionally, to automatically enhance these low-level database changes with application-specific context (e.g., user ID, API endpoint, etc.), check out our compatible [ORM packages](https://docs.bemi.io/#supported-orms).\n\n## Architecture\n\n![Bemi Worker Architecture](docs/static/img/worker.png)\n\nBemi consists of three main parts:\n\n1. [Debezium](https://github.com/debezium/debezium), a very flexible tool for implementing Change Data Capture that is written in Java. It is used by many companies that need to implement ETL such as [Airbyte](https://github.com/airbytehq/airbyte) and [Materialize](https://github.com/MaterializeInc/materialize). We rely on it to be able to connect to PostgreSQL replication log, perform logical decoding, and send raw data to a data sink.\n2. [NATS JetStream](https://github.com/nats-io/nats-server), a cloud-native messaging system written in Go. Debezium is historically designed to send data to Kafka, but it can be also re-configured to send data to NATS JetStream. It is much more lightweight and easy to manage while being very performant and having over 45 clients for different programming languages.\n3. Bemi Worker, a process responsible for stitching data change with app context sent via our open-source [ORM packages](https://docs.bemi.io/#supported-orms) and storing data changes. It is written in TypeScript and uses the `core` that we rely on for our [Bemi](https://bemi.io/) cloud platform.\n\nThe described architecture and the `worker` code in this repository are a simplified version that can be easily run without much overhead.\nIf you want to self-host it in a production environment, see our [self-hosting docs](https://docs.bemi.io/self-hosting).\nAlternatively, [sign up](https://dashboard.bemi.io/log-in?ref=gh) with your GitHub login to use Bemi Cloud.\n\n## License\n\nDistributed under the terms of the [SSPL-1.0 License](/LICENSE). If you need to modify and distribute the code, please release it to contribute back to the open-source community.\n","funding_links":[],"categories":["TypeScript","Uncategorized","Utilities"],"sub_categories":["Uncategorized"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBemiHQ%2Fbemi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FBemiHQ%2Fbemi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBemiHQ%2Fbemi/lists"}