{"id":14968782,"url":"https://github.com/supabase/walrus","last_synced_at":"2025-08-12T16:40:35.840Z","repository":{"id":37098385,"uuid":"394438963","full_name":"supabase/walrus","owner":"supabase","description":"Applying RLS to PostgreSQL WAL","archived":false,"fork":false,"pushed_at":"2024-08-27T16:14:46.000Z","size":356,"stargazers_count":129,"open_issues_count":7,"forks_count":9,"subscribers_count":29,"default_branch":"master","last_synced_at":"2025-07-31T17:48:08.399Z","etag":null,"topics":["cdc","json","postgres","replication"],"latest_commit_sha":null,"homepage":"","language":"PLpgSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/supabase.png","metadata":{"funding":{"github":["supabase"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null},"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-09T21:07:11.000Z","updated_at":"2025-07-24T14:21:22.000Z","dependencies_parsed_at":"2024-04-12T07:59:38.943Z","dependency_job_id":"c3df1ecf-3cdc-4f96-9ba6-8ef2d9f6b7aa","html_url":"https://github.com/supabase/walrus","commit_stats":{"total_commits":169,"total_committers":8,"mean_commits":21.125,"dds":"0.10650887573964496","last_synced_commit":"ef516d68133c4e716bea5c82c8376880ba382c64"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/supabase/walrus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supabase%2Fwalrus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supabase%2Fwalrus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supabase%2Fwalrus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supabase%2Fwalrus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/supabase","download_url":"https://codeload.github.com/supabase/walrus/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supabase%2Fwalrus/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268828571,"owners_count":24313772,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-05T02:00:12.334Z","response_time":2576,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdc","json","postgres","replication"],"created_at":"2024-09-24T13:40:32.942Z","updated_at":"2025-08-12T16:40:35.798Z","avatar_url":"https://github.com/supabase.png","language":"PLpgSQL","funding_links":["https://github.com/sponsors/supabase"],"categories":[],"sub_categories":[],"readme":"# `walrus`\n\u003cp\u003e\n\n\u003ca href=\"\"\u003e\u003cimg src=\"https://img.shields.io/badge/postgresql-12+-blue.svg\" alt=\"PostgreSQL version\" height=\"18\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/supabase/wal_rls/blob/master/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/pypi/l/markdown-subtemplate.svg\" alt=\"License\" height=\"18\"\u003e\u003c/a\u003e\n\n\n\u003c/p\u003e\n\n---\n\n**Source Code**: \u003ca href=\"https://github.com/supabase/walrus\" target=\"_blank\"\u003ehttps://github.com/supabase/walrus\u003c/a\u003e\n\n---\n\nWrite Ahead Log Realtime Unified Security (WALRUS) is a utility for managing realtime subscriptions to tables and applying row level security rules to those subscriptions.\n\nThe subscription stream is based on logical replication slots.\n\n## Summary\n### Managing Subscriptions\n\nUser subscriptions are managed through a table\n\n```sql\ncreate table realtime.subscription (\n    id bigint generated always as identity primary key,\n    subscription_id uuid not null,\n    entity regclass not null,\n    filters realtime.user_defined_filter[] not null default '{}',\n    claims jsonb not null,\n    claims_role regrole not null generated always as (realtime.to_regrole(claims -\u003e\u003e 'role')) stored,\n    created_at timestamp not null default timezone('utc', now()),\n\n    unique (subscription_id, entity, filters)\n);\n```\nwhere `realtime.user_defined_filter` is\n```sql\ncreate type realtime.user_defined_filter as (\n    column_name text,\n    op realtime.equality_op,\n    value text\n);\n```\nand `realtime.equality_op`s are a subset of [postgrest ops](https://postgrest.org/en/v4.1/api.html#horizontal-filtering-rows). Specifically:\n```sql\ncreate type realtime.equality_op as enum(\n    'eq', 'neq', 'lt', 'lte', 'gt', 'gte', 'in'\n);\n```\n\nFor example, to subscribe to a table named `public.notes` where the `id` is `6` as the `authenticated` role:\n```sql\ninsert into realtime.subscription(subscription_id, entity, filters, claims)\nvalues ('832bd278-dac7-4bef-96be-e21c8a0023c4', 'public.notes', array[('id', 'eq', '6')], '{\"role\", \"authenticated\"}');\n```\n\n\n### Reading WAL\n\nThis package exposes 1 public SQL function `realtime.apply_rls(jsonb)`. It processes the output of a `wal2json` decoded logical replication slot and returns:\n\n- `wal`: (jsonb) The WAL record as JSONB in the form\n- `is_rls_enabled`: (bool) If the entity (table) the WAL record represents has row level security enabled\n- `subscription_ids`: (uuid[]) An array subscription ids that should be notified about the WAL record\n- `errors`: (text[]) An array of errors\n\nThe jsonb WAL record is in the following format for inserts.\n```json\n{\n    \"type\": \"INSERT\",\n    \"schema\": \"public\",\n    \"table\": \"todos\",\n    \"columns\": [\n        {\n            \"name\": \"id\",\n            \"type\": \"int8\",\n        },\n        {\n            \"name\": \"details\",\n            \"type\": \"text\",\n        },\n        {\n            \"name\": \"user_id\",\n            \"type\": \"int8\",\n        }\n    ],\n    \"commit_timestamp\": \"2021-09-29T17:35:38Z\",\n    \"record\": {\n        \"id\": 1,\n        \"user_id\": 1,\n        \"details\": \"mow the lawn\"\n    }\n}\n```\n\nupdates:\n```json\n{\n    \"type\": \"UPDATE\",\n    \"schema\": \"public\",\n    \"table\": \"todos\",\n    \"columns\": [\n        {\n            \"name\": \"id\",\n            \"type\": \"int8\",\n        },\n        {\n            \"name\": \"details\",\n            \"type\": \"text\",\n        },\n        {\n            \"name\": \"user_id\",\n            \"type\": \"int8\",\n        }\n    ],\n    \"commit_timestamp\": \"2021-09-29T17:35:38Z\",\n    \"record\": {\n        \"id\": 2,\n        \"user_id\": 1,\n        \"details\": \"mow the lawn\"\n    },\n    \"old_record\": {\n        \"id\": 1,\n    }\n}\n```\n\n\ndeletes:\n```json\n{\n    \"type\": \"DELETE\",\n    \"schema\": \"public\",\n    \"table\": \"todos\",\n    \"columns\": [\n        {\n            \"name\": \"id\",\n            \"type\": \"int8\",\n        },\n        {\n            \"name\": \"details\",\n            \"type\": \"text\",\n        },\n        {\n            \"name\": \"user_id\",\n            \"type\": \"int8\",\n        }\n    ],\n    \"old_record\": {\n        \"id\": 1\n    }\n}\n```\n\nImportant Notes:\n\n- Row level security is not applied to delete statements\n- The key/value pairs displayed in the `old_record` field include the table's identity columns for the record being updated/deleted. To display all values in `old_record` set the replica identity for the table to full\n- When a delete occurs, the contents of `old_record` will be broadcast to all subscribers to that table so ensure that each table's replica identity only contains information that is safe to expose publicly\n\n## Error States\n\n### Error 400: Bad Request, no primary key\nIf a WAL record for a table that does not have a primary key is passed through `realtime.apply_rls`, an error is returned\n\nEx:\n```sql\n(\n    {\n        \"type\": ...,\n        \"schema\": ...,\n        \"table\": ...\n    },                               -- wal\n    true,                            -- is_rls_enabled\n    [...],                           -- subscription_ids,\n    array['Error 400: Bad Request, no primary key'] -- errors\n)::realtime.wal_rls;\n```\n\n### Error 401: Unauthorized\nIf a WAL record is passed through `realtime.apply_rls` and the subscription's `clams_role` does not have permission to `select` the primary key columns in that table, an `Unauthorized` error is returned with no WAL data.\n\nEx:\n```sql\n(\n    {\n        \"type\": ...,\n        \"schema\": ...,\n        \"table\": ...\n    },                               -- wal\n    true,                            -- is_rls_enabled\n    [...],                           -- subscription_ids,\n    array['Error 401: Unauthorized'] -- errors\n)::realtime.wal_rls;\n```\n\n### Error 413: Payload Too Large\nWhen the size of the wal2json record exceeds `max_record_bytes` the `record` and `old_record` objects are filtered to include only fields with a value size \u003c= 64 bytes. The `errors` output array is set to contain the string `\"Error 413: Payload Too Large\"`.\n\nEx:\n```sql\n(\n    {..., \"record\": {\"id\": 1}, \"old_record\": {\"id\": 1}}, -- wal\n    true,                                  -- is_rls_enabled\n    [...],                                 -- subscription_ids,\n    array['Error 413: Payload Too Large']  -- errors\n)::realtime.wal_rls;\n```\n\n## How it Works\n\nEach WAL record is passed into `realtime.apply_rls(jsonb)` which:\n\n- impersonates each subscribed user by setting the appropriate role and `request.jwt.claims` that RLS policies depend on\n- queries for the row using its primary key values\n- applies the subscription's filters to check if the WAL record is filtered out\n- filters out all columns that are not visible to the user's role\n\n## Usage\n\nGiven a `wal2json` replication slot with the name `realtime`\n```sql\nselect * from pg_create_logical_replication_slot('realtime', 'wal2json')\n```\n\nA complete list of config options can be found [here](https://github.com/eulerto/wal2json):\n\nThe stream can be polled with\n\n```sql\nselect\n    xyz.wal,\n    xyz.is_rls_enabled,\n    xyz.subscription_ids,\n    xyz.errors\nfrom\n    pg_logical_slot_get_changes(\n        'realtime', null, null,\n        'include-pk', '1',\n        'include-transaction', 'false',\n        'include-timestamp', 'true',\n        'include-type-oids', 'true',\n        'write-in-chunks', 'true',\n        'format-version', '2',\n        'actions', 'insert,update,delete',\n        'filter-tables', 'realtime.*'\n    ),\n    lateral (\n        select\n            x.wal,\n            x.is_rls_enabled,\n            x.subscription_ids,\n            x.errors\n        from\n            realtime.apply_rls(data::jsonb) x(wal, is_rls_enabled, subcription_ids, errors)\n    ) xyz\nwhere\n    xyz.subscription_ids[1] is not null\n```\n\nOr, if the stream should be filtered according to a publication:\n\n```sql\nwith pub as (\n    select\n        concat_ws(\n            ',',\n            case when bool_or(pubinsert) then 'insert' else null end,\n            case when bool_or(pubupdate) then 'update' else null end,\n            case when bool_or(pubdelete) then 'delete' else null end\n        ) as w2j_actions,\n        coalesce(\n            string_agg(\n                realtime.quote_wal2json(format('%I.%I', schemaname, tablename)::regclass),\n                ','\n            ) filter (where ppt.tablename is not null and ppt.tablename not like '% %'),\n            ''\n        ) w2j_add_tables\n    from\n        pg_publication pp\n        left join pg_publication_tables ppt\n            on pp.pubname = ppt.pubname\n    where\n        pp.pubname = 'supabase_realtime'\n    group by\n        pp.pubname\n    limit 1\n),\nw2j as (\n    select\n        x.*, pub.w2j_add_tables\n    from\n         pub,\n         pg_logical_slot_get_changes(\n            'realtime', null, null,\n            'include-pk', '1',\n            'include-transaction', 'false',\n            'include-type-oids', 'true',\n            'include-timestamp', 'true',\n            'write-in-chunks', 'true',\n            'format-version', '2',\n            'actions', pub.w2j_actions,\n            'add-tables', pub.w2j_add_tables\n        ) x\n)\nselect\n    xyz.wal,\n    xyz.is_rls_enabled,\n    xyz.subscription_ids,\n    xyz.errors\nfrom\n    w2j,\n    realtime.apply_rls(\n        wal := w2j.data::jsonb,\n        max_record_bytes := 1048576\n    ) xyz(wal, is_rls_enabled, subscription_ids, errors)\nwhere\n    w2j.w2j_add_tables \u003c\u003e ''\n    and xyz.subscription_ids[1] is not null\n```\n\n## Configuration\n\n### max_record_bytes\n\n`max_record_bytes` (default 1 MiB): Controls the maximum size of a WAL record that will be emitted with complete `record` and `old_record` data. When the size of the wal2json record exceeds `max_record_bytes` the `record` and `old_record` objects are filtered to include only fields with a value size \u003c= 64 bytes. The `errors` output array is set to contain the string `\"Error 413: Payload Too Large\"`.\n\nEx:\n```sql\nrealtime.apply_rls(wal := w2j.data::jsonb, max_record_bytes := 1024*1024) x(wal, is_rls_enabled, subscription_ids, errors)\n```\n\n\n## Installation\n\nThe project is SQL only and can be installed by executing the contents of `sql/walrus--0.1.sql` in a database instance.\n\n## Tests\n\nRequires\n\n- Postgres 13+\n- wal2json \u003e= 53b548a29ebd6119323b6eb2f6013d7c5fe807ec\n\nOn a Mac:\n\nInstall postgres\n```sh\nbrew install postgres\n```\n\nInstall wal2json\n```sh\ngit clone https://github.com/eulerto/wal2json.git\ncd wal2json\ngit reset --hard 53b548a\nmake\nmake install\n```\n\nRun the tests, from the repo root.\n```sh\n./bin/installcheck\n```\n\n## RFC Process\n\nTo open an request for comment (RFC), open a [github issue against this repo and select the RFC template](https://github.com/supabase/walrus/issues/new/choose).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsupabase%2Fwalrus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsupabase%2Fwalrus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsupabase%2Fwalrus/lists"}