{"id":18644209,"url":"https://github.com/pavdmyt/kafka-pg-upload","last_synced_at":"2026-05-14T23:37:28.170Z","repository":{"id":66157262,"uuid":"292092199","full_name":"pavdmyt/kafka-pg-upload","owner":"pavdmyt","description":"Kafka consumer + PosgreSQL producer","archived":false,"fork":false,"pushed_at":"2020-10-04T15:04:41.000Z","size":222,"stargazers_count":2,"open_issues_count":9,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-12T07:11:26.547Z","etag":null,"topics":["async","kafka-consumer","postgresql","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pavdmyt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-01T19:43:58.000Z","updated_at":"2021-08-11T07:16:44.000Z","dependencies_parsed_at":"2023-02-21T08:31:22.649Z","dependency_job_id":null,"html_url":"https://github.com/pavdmyt/kafka-pg-upload","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pavdmyt%2Fkafka-pg-upload","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pavdmyt%2Fkafka-pg-upload/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pavdmyt%2Fkafka-pg-upload/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pavdmyt%2Fkafka-pg-upload/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pavdmyt","download_url":"https://codeload.github.com/pavdmyt/kafka-pg-upload/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239449369,"owners_count":19640530,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","kafka-consumer","postgresql","python"],"created_at":"2024-11-07T06:10:38.429Z","updated_at":"2026-05-14T23:37:23.139Z","avatar_url":"https://github.com/pavdmyt.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"kafka-pg-upload\n===============\n\n[![Build Status](https://travis-ci.org/pavdmyt/kafka-pg-upload.svg?branch=master)](https://travis-ci.org/pavdmyt/kafka-pg-upload)\n[![codecov](https://codecov.io/gh/pavdmyt/kafka-pg-upload/branch/master/graph/badge.svg)](https://codecov.io/gh/pavdmyt/kafka-pg-upload)\n[![Codacy Badge](https://app.codacy.com/project/badge/Grade/10ac01fad4944e3985569b150e21431c)](https://www.codacy.com/manual/pavdmyt/kafka-pg-upload?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=pavdmyt/kafka-pg-upload\u0026amp;utm_campaign=Badge_Grade)\n[![Updates](https://pyup.io/repos/github/pavdmyt/kafka-pg-upload/shield.svg)](https://pyup.io/repos/github/pavdmyt/kafka-pg-upload/)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\nIs a microservice to consume messages from [Kafka](https://kafka.apache.org/)\ntopic and write them into [PostgreSQL](https://www.postgresql.org/).\n\n\n# Installation\n\nRun the following commands:\n\n```\n$ make build\n$ pip install dist/kafka_pg_upload-*-py3-none-any.whl\n```\n\n\n# Configuration\n\nConfiguration is provided via environment variables. It contains good defaults\nto start experimenting with. It is possible to override defaults either directly\n(e.g. `export KAPG_KAFKA_TOPIC=foo`) or via a file:\n\n```\n$ cat \u003e kapg.cfg \u003c\u003cEOF\n\u003e export KAPG_GROUP_ID=123\n\u003e export KAPG_AUTOOFFSETRESET=smallest\n\u003eEOF\n$\n$ source kapg.cfg\n```\n\n| Environment variable           | Parameter                | Description                                        | Default                                   |\n|--------------------------------|--------------------------|----------------------------------------------------|-------------------------------------------|\n| `KAPG_BROKER_LIST`             | `kafka_broker_list`      | Initial list of brokers as host:port (comma delimited) | `\"localhost:9092,\"`                   |\n| `KAPG_KAFKA_TOPIC`             | `kafka_topic`            | Kafka topic name to send monitoring data           | `\"pagemonitor_metrics\"`                   |\n| `KAPG_GROUP_ID`                | `consumer_group.id`      | Client group id string. All clients sharing the same group.id belong to the same group | `42`  |\n| `KAPG_AUTOOFFSETRESET`         | `consumer_auto.offset.reset` | Action to take when there is no initial offset in offset store or the desired offset is out of range | `\"earliest\"` |\n| `KAPG_CONSUMER_SLEEP`          | `consumer_sleep_interval` | How long consumer waits when there is no messages to consume (seconds) | `2`                  |\n| `KAPG_KAFKA_ENABLE_CERT_AUTH`  | `kafka_enable_cert_auth`  | Enable SSL mode for communication with Kafka      | `False`                                   |\n| `KAPG_KAFKA_SSL_CA`            | `kafka_ssl_ca`           | Path to Kafka CA file                              | `\"/etc/kapg/ssl/kafka/ca.pem\"`            |\n| `KAPG_KAFKA_SSL_CERT`          | `kafka_ssl_cert`         | Path to Kafka certificate file                     | `\"/etc/kapg/ssl/kafka/service.cert\"`      |\n| `KAPG_KAFKA_SSL_KEY`           | `kafka_ssl_key`          | Path to Kafka key file                             | `\"/etc/kapg/ssl/kafka/service.key\"`       |\n| `KAPG_PG_HOST`                 | `pg_host`                | PostgreSQL host                                    | `\"localhost\"`                             |\n| `KAPG_PG_PORT`                 | `pg_port`                | PostgreSQL port                                    | `5432`                                    |\n| `KAPG_PG_USER`                 | `pg_user`                | PostgreSQL user                                    | `\"postgres\"`                              |\n| `KAPG_PG_PWD`                  | `pg_password`            | PostgreSQL password                                | `\"changeme\"`                              |\n| `KAPG_PG_DB_NAME`              | `pg_db_name`             | PostgreSQL database name to write into             | `\"metrics\"`                               |\n| `KAPG_PG_TABLE_NAME`           | `pg_table_name`          | PostgreSQL table name to write into                | `\"pagemonitor\"`                           |\n| `KAPG_PG_CONN_TIMEOUT`         | `pg_conn_timeout`        | PostgreSQL connection timeout (seconds)            | `10.0`                                    |\n| `KAPG_PG_COMMAND_TIMEOUT`      | `pg_command_timeout`     | PostgreSQL default timeout for operations on existing connection (seconds) | `10.0`            |\n| `KAPG_PG_ENABLE_SSL`           | `pg_enable_ssl`          | Enable SSL for communication with PostgreSQL       | `False`                                   |\n| `KAPG_PG_SSL_CA`               | `pg_ssl_ca`              | Path to PostgreSQL CA file                         | `\"/etc/kapg/ssl/postgres/ca.pem\"`         |\n\n\n# Usage\n\nAfter installation `kafka-pg-upload` executable should be available in your `$PATH`.\nProvide a desired config and start the service:\n\n```\n$ kafka-pg-upload\n{\"bin\": \"/Users/me/.pyenv/versions/playground-py38/bin/kafka-pg-upload\", \"version\": \"0.1.0\", \"config\": {\"kafka_broker_list\": \"localhost:9092,\", \"kafka_topic\": \"pagemonitor_metrics\", \"consumer_group.id\": \"42\", \"consumer_auto.offset.reset\": \"earliest\", \"consumer_sleep_interval\": 2.0, \"kafka_enable_cert_auth\": false, \"kafka_ssl_ca\": \"PosixPath('/etc/kapg/ssl/kafka/ca.pem')\", \"kafka_ssl_cert\": \"PosixPath('/etc/kapg/ssl/kafka/service.cert')\", \"kafka_ssl_key\": \"PosixPath('/etc/kapg/ssl/kafka/service.key')\", \"pg_host\": \"localhost\", \"pg_port\": 5432, \"pg_user\": \"postgres\", \"pg_password\": \"changeme\", \"pg_db_name\": \"metrics\", \"pg_table_name\": \"pagemonitor\", \"pg_conn_timeout\": 10.0, \"pg_command_timeout\": 10.0, \"pg_enable_ssl\": false, \"pg_ssl_ca\": \"PosixPath('/etc/kapg/ssl/postgres/ca.pem')\"}}\n```\n\n\n# Development\n\nClone the repository:\n\n```\n$ git clone git@github.com:pavdmyt/kafka-pg-upload.git\n```\n\nInstall dev dependencies:\n\n```\n$ poetry install\n\n# if you don't have poetry installed:\npip install -r requirements.txt\n```\n\nLint, format, isort code:\n\n```\n$ make lint\n$ make fmt\n$ make isort\n```\n\nRun tests:\n\n```\n$ make test\n```\n\nBuild:\n\n```\n$ make build\n```\n\n\n# Contributing\n\n1. Fork it!\n2. Create your feature branch: `git checkout -b my-new-feature`\n3. Commit your changes: `git commit -m 'Add some feature'`\n4. Push to the branch: `git push origin my-new-feature`\n5. Submit a pull request\n6. Make sure tests are passing\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpavdmyt%2Fkafka-pg-upload","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpavdmyt%2Fkafka-pg-upload","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpavdmyt%2Fkafka-pg-upload/lists"}