{"id":13807178,"url":"https://github.com/xstevens/pg_kafka","last_synced_at":"2025-05-14T00:30:59.734Z","repository":{"id":20437210,"uuid":"23713984","full_name":"xstevens/pg_kafka","owner":"xstevens","description":"INACTIVE: A PostgreSQL extension to produce messages to Apache Kafka.","archived":true,"fork":false,"pushed_at":"2015-04-03T00:35:32.000Z","size":171,"stargazers_count":112,"open_issues_count":3,"forks_count":15,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-11-18T23:52:10.308Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xstevens.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-09-05T18:33:03.000Z","updated_at":"2024-05-04T10:28:33.000Z","dependencies_parsed_at":"2022-07-26T09:17:48.675Z","dependency_job_id":null,"html_url":"https://github.com/xstevens/pg_kafka","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xstevens%2Fpg_kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xstevens%2Fpg_kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xstevens%2Fpg_kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xstevens%2Fpg_kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xstevens","download_url":"https://codeload.github.com/xstevens/pg_kafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254046238,"owners_count":22005559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T01:01:22.124Z","updated_at":"2025-05-14T00:30:54.713Z","avatar_url":"https://github.com/xstevens.png","language":"C","funding_links":[],"categories":["Data Ingestion"],"sub_categories":[],"readme":"# pg_kafka\n\nVersion: 0.0.1\n\n**pg_kafka** is a PostgreSQL extension to produce messages to Apache Kafka. When combined with PostgreSQL it \ncreates a convenient way to get operations and row data without the limits of using LISTEN/NOTIFY.\n\n**pg_kafka** is released under the MIT license (See LICENSE file).\n\nShout out to the [pg_amqp extension](https://github.com/omniti-labs/pg_amqp) authors. I used their project as \na guide to teach myself how to write a PostgreSQL extension.\n\n### Version Compatability\nThis code is built with the following assumptions.  You may get mixed results if you deviate from these versions.\n\n* [PostgreSQL](http://www.postgresql.org) 9.2+\n* [Kafka](http://kafka.apache.org) 0.8.1+\n* [librdkafka](https://github.com/edenhill/librdkafka) 0.8.x\n\n### Requirements\n* PostgreSQL\n* librdkafka\n* libsnappy\n* zlib\n\n### Building\n\nTo build you will need to install PostgreSQL (for pg_config) and PostgreSQL server development packages. On Debian \nbased distributions you can usually do something like this:\n\n    apt-get install -y postgresql postgresql-server-dev-9.2\n    \nYou will also need to make sure the librdkafka library and it's header files have been installed. See their Github \npage for further details.\n\nIf you have all of the prerequisites installed you should be able to just:\n\n    make \u0026\u0026 make install\n\nOnce the extension has been installed you just need to enable it in postgresql.conf:\n\n    shared_preload_libraries = 'pg_kafka.so'\n\nAnd restart PostgreSQL.\n\n### Usage\n    -- run once\n    create extension kafka;\n    -- insert broker information\n    insert into kafka.broker values ('localhost', 9092);\n    -- produce a message\n    select kafka.produce('test_topic', 'my message');\n\nFor something a bit more useful, consider setting this up on a trigger and producing a message for every INSERT, UPDATE, \nand DELETE that happens on a table.\n\nIf the kafka schema wasn't auto-created by installing the extension take a look at sql/kafka.sql.\n            \n### Support\n\nFile bug reports, feature requests and questions using\n[GitHub Issues](https://github.com/xstevens/pg_kafka/issues)\n\n### Notes\n\nBefore implementing this project I had looked into LISTEN/NOTIFY operations in PostgreSQL. NOTIFY is unfortunately limited \nto 8000 bytes for the total payload size. I also found several mentions in the PostgreSQL mailing lists that NOTIFY was \nnever intended to send row data; rather it was intended to get change notifications on keys to clean up external caching, etc. \n\nSo my next approach was trying to read PostgreSQL WAL data by creating a process that acted as a replication \nclient. This approach would have a number of advantages compared to being an extension in database (namely you wouldn't \npotentially affect any transactions and you can run outside of the database). I successfully started receiving WAL data, \nbut I could not find any material on how to actually decode that data. Fortunately as of Postgres 9.4 they have added Logical Decoding. I have started working on a logical decoding output plugin called [decoderbufs](https://github.com/xstevens/decoderbufs).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxstevens%2Fpg_kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxstevens%2Fpg_kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxstevens%2Fpg_kafka/lists"}