{"id":13809198,"url":"https://github.com/cpursley/walex","last_synced_at":"2025-04-04T07:08:32.508Z","repository":{"id":40753532,"uuid":"435525398","full_name":"cpursley/walex","owner":"cpursley","description":"Postgres change events (CDC) in Elixir","archived":false,"fork":false,"pushed_at":"2024-04-10T17:13:16.000Z","size":237,"stargazers_count":250,"open_issues_count":7,"forks_count":12,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-04-13T22:00:30.071Z","etag":null,"topics":["cdc","change-data-capture","debezium","elixir","postgres","postgresql","replication"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cpursley.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-06T14:20:51.000Z","updated_at":"2024-05-01T17:21:05.954Z","dependencies_parsed_at":"2023-02-15T17:30:49.297Z","dependency_job_id":"7eae311d-1f3e-4afc-9667-e8d6bc95f141","html_url":"https://github.com/cpursley/walex","commit_stats":{"total_commits":61,"total_committers":5,"mean_commits":12.2,"dds":"0.39344262295081966","last_synced_commit":"823f053bc7638ae193494f04643302acccd7e444"},"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpursley%2Fwalex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpursley%2Fwalex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpursley%2Fwalex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpursley%2Fwalex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cpursley","download_url":"https://codeload.github.com/cpursley/walex/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247135144,"owners_count":20889421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdc","change-data-capture","debezium","elixir","postgres","postgresql","replication"],"created_at":"2024-08-04T01:02:07.260Z","updated_at":"2025-04-04T07:08:32.486Z","avatar_url":"https://github.com/cpursley.png","language":"Elixir","funding_links":[],"categories":["ORM and Datamapping","\u003ca name=\"Elixir\"\u003e\u003c/a\u003eElixir"],"sub_categories":[],"readme":"# WalEx\n\nSimple and reliable Postgres [Change Data Capture\n(CDC)](https://en.wikipedia.org/wiki/Change_data_capture) in Elixir.\n\n![Walex mascot](mascot.jpeg)\n\nWalEx allows you to listen to change events on your Postgres tables then perform callback-like actions with the data via the [Event DSL](#event-module). For example:\n\n- Stream database changes to an external service\n- Send a user a welcome email after they create a new account\n- Augment an existing Postgres-backed application with business logic\n- Send events to third party services (analytics, CRM, webhooks, etc))\n- Update index / invalidate cache whenever a record is changed\n\nYou can learn more about CDC and what you can do with it here: [Why capture changes?](https://bbhoss.io/posts/announcing-cainophile/#why-capture-changes)\n\n## Credit\n\nThis library borrows liberally from\n[realtime](https://github.com/supabase/realtime) from Supabase, which in turn\ndraws heavily on [cainophile](https://github.com/cainophile/cainophile).\n\n## Installation\n\nIf [available in Hex](https://hex.pm/docs/publish), the package can be installed\nby adding `walex` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [\n    {:walex, \"~\u003e 4.6.0\"}\n  ]\nend\n```\n\n## PostgreSQL Configuration\n\n### Logical Replication\n\nWalEx only supports PostgreSQL. To get started, you first need to configure\nPostgreSQL for [logical replication](https://www.crunchydata.com/blog/data-to-go-postgres-logical-replication):\n\n```sql\nALTER SYSTEM SET wal_level = 'logical';\n```\n\nDocker Compose:\n\n```bash\ncommand: [ \"postgres\", \"-c\", \"wal_level=logical\" ]\n```\n\n### Publication\n\nWhen you change the `wal_level` variable, you'll need to restart your\nPostgreSQL server. Once you've restarted, go ahead and [create a\npublication](https://www.postgresql.org/docs/current/sql-createpublication.html)\nfor the tables you want to receive changes for:\n\nAll tables:\n\n```sql\nCREATE PUBLICATION events FOR ALL TABLES;\n```\n\nOr just specific tables:\n\n```sql\nCREATE PUBLICATION events FOR TABLE user, todo;\n```\n\nFilter based on [row conditions](https://www.postgresql.fastware.com/blog/introducing-publication-row-filters) (Postgres v15+ only):\n\n```sql\nCREATE PUBLICATION user_event FOR TABLE user WHERE (active IS TRUE);\n```\n\n### Replica Identity\n\nWalEx supports all of the settings for [REPLICA\nIDENTITY](https://www.postgresql.org/docs/current/sql-altertable.html#SQL-CREATETABLE-REPLICA-IDENTITY).\nUse `FULL` if you can use it, as it will make tracking differences [easier](https://xata.io/blog/replica-identity-full-performance) as\nthe old data will be sent alongside the new data. You'll need to set this for\neach table.\n\nSpecific tables:\n\n```sql\nALTER TABLE user REPLICA IDENTITY FULL;\nALTER TABLE todo REPLICA IDENTITY FULL;\n```\n\nAlso, be mindful of [replication gotchas](https://pgdash.io/blog/postgres-replication-gotchas.html).\n\n### AWS RDS\n\nAmazon (AWS) RDS Postgres allows you to configure logical replication.\n\n- \u003chttps://dev.to/vumdao/how-to-change-rds-postgresql-configurations-2kmk\u003e\n\nWhen creating a new Postgres database on RDS, you'll need to set a Parameter\nGroup with the following settings:\n\n```text\nrds.logical_replication = 1\nmax_replication_slots = 5\nmax_slot_wal_keep_size = 2048\n```\n\n## Elixir Configuration\n\n### Config\n\n```elixir\n# config.exs\n\nconfig :my_app, WalEx,\n  hostname: \"localhost\",\n  username: \"postgres\",\n  password: \"postgres\",\n  port: \"5432\",\n  database: \"postgres\",\n  publication: \"events\",\n  subscriptions: [\"user\", \"todo\"],\n  # WalEx assumes your module names match this pattern: MyApp.Events.User, MyApp.Events.ToDo, etc\n  # but you can also specify custom modules like so:\n  # modules: [MyApp.CustomModule.User, MyApp.OtherCustomModule.ToDo],\n  name: MyApp\n```\n\nIt is also possible to just define the URL configuration for the database\n\n```elixir\n# config.exs\n\nconfig :my_app, WalEx,\n  url: \"postgres://username:password@hostname:port/database\"\n  publication: \"events\",\n  subscriptions: [\"user\", \"todo\"],\n  name: MyApp\n```\n\nYou can also dynamically update the config at runtime:\n\n```elixir\nWalEx.Configs.add_config(MyApp, :subscriptions, [\"new_subscriptions_1\", \"new_subscriptions_2\"])\nWalEx.Configs.remove_config(MyApp, :subscriptions, \"subscriptions\")\nWalEx.Configs.replace_config(MyApp, :password, \"new_password\")\n```\n\n### Application Supervisor\n\n```elixir\ndefmodule MyApp.Application do\n  use Application\n\n  def start(_type, _args) do\n    children = [\n      {WalEx.Supervisor, Application.get_env(:my_app, WalEx)}\n    ]\n\n    opts = [strategy: :one_for_one, name: MyApp.Supervisor]\n    Supervisor.start_link(children, opts)\n  end\nend\n```\n\n## Usage\n\n### Event\n\nReturned change data is a List of [%Event{}](lib/walex/event/event.ex) structs with changes. UPDATE event example\nwhere _name_ field was changed):\n\n```elixir\n[\n  %Walex.Event{\n    name: :user,\n    type: :update,\n    source: %WalEx.Event.Source{\n      name: \"WalEx\",\n      version: \"3.8.0\",\n      db: \"todos\",\n      schema: \"public\",\n      table: \"user\",\n      columns: %{\n        id: \"integer\",\n        name: \"varchar\",\n        created_at: \"timestamptz\"\n      }\n    },\n    new_record: %{\n      id: 1234,\n      name: \"Chase Pursley\",\n      created_at: #DateTime\u003c2023-08-18 14:09:05.988369-04:00 -04 Etc/UTC-4\u003e\n    },\n    # we don't show old_record for update to reduce payload size\n    # however, you can see any old values that changed under \"changes\"\n    old_record: nil,\n    changes: %{\n      name: %{\n        new_value: \"Chase Pursley\",\n        old_value: \"Chase\"\n      }\n    },\n    timestamp: ~U[2023-12-18 15:50:08.329504Z]\n  }\n]\n```\n\n### Event Module\n\nIf your app is named _MyApp_ and you have a subscription called _:user_ (which represents a database table), WalEx assumes you have a module called `MyApp.Events.User` that uses WalEx Event. But you can also define any custom module, just be sure to add it to the _modules_ config.\n\nNote that the result of `events` is a list. This is because WalEx returns a _List_ of  _transactions_ for a particular table when there's a change event. Often times this will just contain one result, but it could be many (for example, if you use database triggers to update a column after an insert).\n\n```elixir\ndefmodule MyApp.Events.User do\n  use WalEx.Event, name: MyApp\n\n  # any subscribed event\n  on_event(:all, fn events -\u003e\n    IO.inspect(events: events)\n  end)\n\n  # any user event\n  on_event(:user, fn users -\u003e\n    IO.inspect(on_event: users)\n    # do something with users data\n  end)\n\n  # any user insert event\n  on_insert(:user, fn users -\u003e\n    IO.inspect(on_insert: users)\n  end)\n\n  on_update(:user, fn users -\u003e\n    IO.inspect(on_update: users)\n  end)\n\n  on_delete(:user, fn users -\u003e\n    IO.inspect(on_delete: users)\n  end)\nend\n```\n\n##### Filters\n\nA common scenario is where you want to _\"unsubscribe\"_ from specific records (for example, temporarily for a migration or data fix). One way to accomplish this is to have a column with a value like `event_subscribe: false`. Then you can ignore specific events by specifying their key and value to *unwatched_records*.\n\nAnother scenario is you might not care when just certain fields change. For example, maybe a database trigger sets updated_at _after_ a record is updated. Or a count changes, or several do that you don't need to react to. In this case, you can ignore the event change by adding them to *unwatched_fields*.\n\nAdditional filter helpers available in the\n[WalEx.TransactionFilter](lib/walex/transaction_filter.ex) module.\n\n```elixir\ndefmodule MyApp.Events.User do\n  use WalEx.Event, name: MyApp\n\n  @filters %{\n    unwatched_records: %{event_subscribe: false},\n    unwatched_fields: ~w(event_id updated_at todos_count)a\n  }\n\n  on_insert(:user, @filters, fn users -\u003e\n    IO.inspect(on_insert: users)\n    # resulting users data is filtered\n  end)\nend\n```\n\n##### Functions\n\nYou can also provide a list of functions (as atoms) to be applied to each Event (after optional filters are applied). Each function is run as an async Task on each event. The functions must be defined in the current module and take a single _event_ argument. Use with caution!\n\n```elixir\ndefmodule MyApp.Events.User do\n  use WalEx.Event, name: MyApp\n\n  @filters %{unwatched_records: %{event_subscribe: false}}\n  @functions ~w(send_welcome_email add_to_crm clear_cache)a\n\n  on_insert(:user, @filters, @functions, fn users -\u003e\n    IO.inspect(on_insert: users)\n    # resulting users data is first filtered then functions are applied\n  end)\n\n  def send_welcome_email(user) do\n    # logic for sending welcome email to new user\n  end\n\n  def add_to_crm(user) do\n  # logic for adding user to crm system\n  end\n\n  def clear_cache(user) do\n  # logic for clearing user cache\n  end\nend\n```\n\n### Advanced usages\n\n##### Durable slot\n\nBy default WalEx will create a temporary replication slot in Postgres.\n\nThis means that if the connection between WalEx and Postgres gets interrupted (crash / disconnection / etc.),\nthe replication slot will get dropped by Postgres.\nThis makes using WalEx safer as there is not risk of filling up the disk of the Postgres writer instance in\ncase of downtime.\n\nThe downside being that this event-loss is more than likely.\nIf this is a no-go, WalEx also supports durable replication.\n\n```elixir\n# config.exs\n\nconfig :my_app, WalEx,\n  # ...\n  durable_slot: true,\n  slot_name: my_app_replication_slot\n```\n\nOnly a single process can be connected to a durable slot at once,\nin case the slot is already used `WalEx.Supervisor` will fail to start with a `RuntimeError`.\n\nBe warned that there are many additional potential gotchas (a detailed guide is planned).\n\n##### Event middleware / Back-pressure\n\nWalEx receives events from Postgres in `WalEx.Replication.Server` and then `cast` those to `WalEx.Replication.Publisher`.\nIt's then `WalEx.Replication.Publisher` that is responsible to join these events together and process them.\n\nIn the event where you'd expect Postgres to overwhelm WalEx and potentially cause OOMs,\nWalEx provides a config option that should help you implement back-pressure.\n\nAs it's a quite advanced use, with many strong requirements,\nit's recommended instead to increase the amount of RAM of your instance.\n\nNever the less, if it's not an option or would like to control the consumption rate of events,\nWalEx provide the following configuration option:\n\n```\nconfig :my_app, WalEx,\n  # ...\n  message_middleware: fn message, app_name -\u003e ... end\n```\n\n`message_middleware` allows you to define the way `WalEx.Replication.Server` and `WalEx.Replication.Publisher` communicate.\n\nIf for instance you'd like to store these events to disk before processing them you would need to:\n\n- provide a `message_middleware` callback. It should serialize messages and store them to disk\n- add a supervised strictly-ordered disk consumer. On each event it would call one of:\n  - `WalEx.Replication.Publisher.process_message_async(message, app_name)`\n  - `WalEx.Replication.Publisher.process_message_sync(message, app_name)`\n\nAny back-pressure implementation needs to guarantee:\n\n- exact message ordering\n- exactly-once-delivery\n- that each running walex has an isolated back-pressure system (for instance one queue per instance)\n\n## Test\n\nYou'll need a local Postgres instance running\n\n```bash\nMIX_ENV=test mix walex.setup\nMIX_ENV=test mix test\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpursley%2Fwalex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcpursley%2Fwalex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpursley%2Fwalex/lists"}