{"id":37779687,"url":"https://github.com/ihippik/wal-listener","last_synced_at":"2026-01-16T15:03:59.087Z","repository":{"id":53866859,"uuid":"232385658","full_name":"ihippik/wal-listener","owner":"ihippik","description":"PostgreSQL WAL listener","archived":false,"fork":false,"pushed_at":"2025-08-20T07:29:54.000Z","size":258,"stargazers_count":448,"open_issues_count":8,"forks_count":52,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-11-22T13:00:03.780Z","etag":null,"topics":["golang","logical-replication","microservices-architecture","postgresql"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ihippik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-01-07T18:06:32.000Z","updated_at":"2025-11-21T22:05:28.000Z","dependencies_parsed_at":"2024-02-22T19:51:39.934Z","dependency_job_id":"a306fb52-280e-42fa-8394-2962078fcf90","html_url":"https://github.com/ihippik/wal-listener","commit_stats":{"total_commits":99,"total_committers":8,"mean_commits":12.375,"dds":"0.21212121212121215","last_synced_commit":"af4a3de71c08a9165350038765ae2778a95709d5"},"previous_names":[],"tags_count":38,"template":false,"template_full_name":null,"purl":"pkg:github/ihippik/wal-listener","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ihippik%2Fwal-listener","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ihippik%2Fwal-listener/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ihippik%2Fwal-listener/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ihippik%2Fwal-listener/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ihippik","download_url":"https://codeload.github.com/ihippik/wal-listener/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ihippik%2Fwal-listener/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28479406,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","logical-replication","microservices-architecture","postgresql"],"created_at":"2026-01-16T15:03:58.984Z","updated_at":"2026-01-16T15:03:59.066Z","avatar_url":"https://github.com/ihippik.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WAL-Listener\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n![GitHub go.mod Go version](https://img.shields.io/github/go-mod/go-version/ihippik/wal-listener)\n![GitHub tag (latest SemVer)](https://img.shields.io/github/v/tag/ihippik/wal-listener)\n[![Publish Docker image](https://github.com/ihippik/wal-listener/actions/workflows/github-actions.yml/badge.svg)](https://github.com/ihippik/wal-listener/actions/workflows/github-actions.yml)\n\n![WAL-Listener](wal-listener.png)\n\nA service that helps implement the **Event-Driven architecture**.\n\nTo maintain the consistency of data in the system, we will use **transactional messaging** -\npublishing events in a single transaction with a domain model change.\n\nThe service allows you to subscribe to changes in the PostgreSQL database using its logical decoding capability\nand publish them to a message broker.\n\n## Logic of work\nTo receive events about data changes in our PostgreSQL DB\n  we use the standard logic decoding module (**pgoutput**) This module converts\nchanges read from the WAL into a logical replication protocol.\n  And we already consume all this information on our side.\nThen we filter out only the events we need and publish them in the queue\n\n### Event publishing\n\nAs the message broker will be used is of your choice:\n- NATS JetStream [`type=nats`];\n- Apache Kafka [`type=kafka`];\n- RabbitMQ [`type=rabbitmq`].\n- Google Pub/Sub [`type=google_pubsub`].\n\nThe service publishes the following structure.\nThe name of the topic for subscription to receive messages is formed from the prefix of the topic,\nthe name of the database, and the name of the table `prefix + schema_table`.\n\n\u003e If you are using Kafka, you may want to select a partition using a message key. \n\u003e You can do this in the producer configuration by specifying the **messageKeyFrom** variable, \n\u003e which will indicate from which table field to take the key. \n\u003e If there is no such field, the table name will be used.\n\n\n```go\n{\n\tID        uuid.UUID       # unique ID\n\tSchema    string\n\tTable     string\n\tAction    string\n\tData      map[string]any\n\tDataOld   map[string]any  # old data (see DB-settings note #1)\n\tEventTime time.Time       # commit time\n}\n```\n\nMessages are published to the broker at least once!\n\n### Filter configuration example\n\n```yaml\nlistener:\n  filter:\n    tables:\n      users:\n        - insert\n        - update\n\n```\nThis filter means that we only process events occurring with the `users` table,\nand in particular `insert` and `update` data.\n\n**Possible filters**:\n- insert\n- update\n- delete\n- truncate\n\n### Topic mapping\nBy default, the output NATS topic name consists of prefix, DB schema, and DB table name,\nbut if you want to send all updates in one topic, you should be configured the topic map:\n```yaml\ntopicsMap:\n  main_users: \"notifier\"\n  main_customers: \"notifier\"\n```\n\n## DB setting\nYou must make the following settings in the db configuration (postgresql.conf)\n* wal_level \u003e= “logical”\n* max_replication_slots \u003e= 1\n\nThe publication \u0026 slot created automatically when the service starts (for all tables and all actions).\nYou can delete the default publication and create your own (name: _wal-listener_) with the necessary filtering conditions, and then the filtering will occur at the database level and not at the application level.\n\nhttps://www.postgresql.org/docs/current/sql-createpublication.html\n\nIf you change the publication, remember to change the slot name or delete the current one.\n\nNotes:\n\n1. To receive `DataOld` field you need to change REPLICA IDENTITY to FULL as described here:\n   [#SQL-ALTERTABLE-REPLICA-IDENTITY](https://www.postgresql.org/docs/current/sql-altertable.html#SQL-ALTERTABLE-REPLICA-IDENTITY)\n\n## Service configuration\n\u003e All config entities are written in camelCase, except for the database entities themselves\n```yaml\nlistener:\n  slotName: myslot_1\n  refreshConnection: 30s\n  heartbeatInterval: 10s\n  filter:\n    tables:\n      seasons:\n        - insert\n        - update\n  topicsMap:\n    schema_table_name: \"notifier\"\nlogger:\n  level: info\n  fmt: json\ndatabase:\n  host: localhost\n  port: 5432\n  name: my_db\n  user: postgres\n  password: postgres\n  debug: false\npublisher:\n   type: nats\n   address: localhost:4222\n   topic: \"wal_listener\"\n   topicPrefix: \"\"\nmonitoring:\n  sentryDSN: \"dsn string\"\n  promAddr: \":2112\"\n```\nWe are using [Viper;](https://github.com/spf13/viper) it means you can override each value via env variables with `WAL_` prefix.\n\n_for instance: `WAL_DATABASE_PORT=5433`_\n\n## Monitoring\n\n### Sentry\nIf you specify an DSN-string for the [Sentry](https://sentry.io/) project, the next level errors will be posted there via a hook:\n* Panic\n* Fatal\n* Error\n\n### Prometheus\nYou can take metrics by specifying an endpoint for Prometheus in the configuration.\n#### Available metrics\n\n| name                        | description                          | fields             |\n|-----------------------------|--------------------------------------|--------------------|\n| published_events_total      | the total number of published events | `subject`, `table` |\n| filter_skipped_events_total | the total number of skipped events   | `table`            |\n\n### Kubernetes\nApplication initializes a web server (*if a port is specified in the configuration*) with two endpoints \nfor readiness `/ready`  and liveness `/healthz` probes.\n\n## Docker\n\nYou can start the container from the project folder (configuration file is required).\n\nSee `./config_example.yml` for an example configuration.\nBe sure to copy the file to the docker image in the `Dockerfile` prior to running [after the build setp](https://github.com/ihippik/wal-listener/blob/master/Dockerfile#L31)\nex:\n```docker\nCOPY /config.yml .\n```\n\nСontainer preparation is carried out with the help of a multi-stage build, which creates after itself auxiliary images of a large size.\nPlease don't forget to delete them:\n```shell\ndocker image prune --filter label=stage=builder\n```\n\n#### Docker Hub\nhttps://hub.docker.com/r/ihippik/wal-listener\n#### Example\n```shell\ndocker run -v $(pwd)/config.yml:/app/config.yml ihippik/wal-listener:tag\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fihippik%2Fwal-listener","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fihippik%2Fwal-listener","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fihippik%2Fwal-listener/lists"}