{"id":20986581,"url":"https://github.com/city-mobil/kafka-proxy","last_synced_at":"2025-05-14T17:33:01.423Z","repository":{"id":46071318,"uuid":"364319164","full_name":"city-mobil/kafka-proxy","owner":"city-mobil","description":"Rust Kafka HTTP proxy","archived":false,"fork":false,"pushed_at":"2021-11-16T16:04:44.000Z","size":141,"stargazers_count":27,"open_issues_count":2,"forks_count":4,"subscribers_count":7,"default_branch":"master","last_synced_at":"2023-03-10T15:32:59.071Z","etag":null,"topics":["kafka","kafka-producer","kafka-proxy","rust","rust-lang"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/city-mobil.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-04T16:28:20.000Z","updated_at":"2023-03-06T11:16:31.000Z","dependencies_parsed_at":"2022-08-20T15:01:21.364Z","dependency_job_id":null,"html_url":"https://github.com/city-mobil/kafka-proxy","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/city-mobil%2Fkafka-proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/city-mobil%2Fkafka-proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/city-mobil%2Fkafka-proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/city-mobil%2Fkafka-proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/city-mobil","download_url":"https://codeload.github.com/city-mobil/kafka-proxy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225304004,"owners_count":17453037,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-producer","kafka-proxy","rust","rust-lang"],"created_at":"2024-11-19T06:14:16.009Z","updated_at":"2024-11-19T06:14:16.475Z","avatar_url":"https://github.com/city-mobil.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Proxy Rust Forever (kprf)\n\nHTTP Kafka producer-proxy based on librdkafka \u0026 warp\n\n### NOTE: This is non-stable alpha version so API and configuration can be changed.\n\n## Inspirations\n\nIn CityMobil we heavily rely on data-processing. It is widely used for pricing, statistics collection etc.\nMost of this data is processed from php-fpm. \n\nFor producing data to kafka internal librdkafka thread-pool is explicitly created. It is quite heavy operation\nfor every php-fpm worker.\n\nOne of the easiest solutions is to use some external http-proxy. This is where 'kafka-proxy' (or kprf) comes to play.\n\n## Setup\n\n`rustc 1.51.0` is required to build kafka-proxy.\n\n- To build debug version of kafka-proxy:\n```shell\nmake # builds rust-debug version\n```\n\nBuilt debug binary is stored at `target/debug/kprf`\n\n- To build release version of kafka-proxy:\n```shell\nmake build_release\n```\n\nBuilt release binary is stored at `target/release/kprf`\n\n- To run kafka-proxy(release version is recommended to use in production environment):\n```shell\ntarget/release/kprf --config=config_example.yaml\n```\n\n## QuickStart\n\nFollowing examples work if kafka-proxy is set up and running.\n\n```\n# Try asynchronous producing\ncurl 'http://127.0.0.1:4242/push' -H 'Content-Type: application/json' -d '{\"records\": [{\"topic\": \"SOME_TOPIC\", \"data\": \"{\"\\a\"\\: \"\\b\"\\}\"}],\n\"wait_for_send\": false}'\n\n# Possible success response:\n{\"status\": \"ok\", \"errors\": []}\n\n# Possible erroring response:\n{\"status\": \"error\", \"errors\": [{\"status\": \"error\", \"error\": \"some_message\"}, {\"status\": \"ok\", \"error\":\"\"}]}\n```\n\nJSON Fields:\n- `records` – describes records for further producing.\n- `records[i].topic` – describes some kafka topic for producing.\n- `records[i].data` –  describes some string data. Can be JSON, XML or anything.\n- `wait_for_send` – describes if http-producer client has to wait for delivery result or not.\nIf false, message is produced asynchronously. Otherwise, synchronously. Default value is `false`\n\n## Configuration\n\nAt this moment, this options from [librdkafka](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) are supported:\n\n- `kafka.brokers` – alias for `bootstrap.servers` from librdkafka. Default value is empty array.\n- `kafka.user` – alias for `sasl.username` from librdkafka. Default value is empty string.\n- `kafka.password` – alias for `sasl.password` from librdkafka. Default value is empty string.\n- `kafka.message_max_bytes` – alias for `message.max.bytes` from librdkafka. Default value is `1 MiB`.\n- `kafka.queue_buffering_max_messages` – alias for `queue.buffering.max.messages` from librdkafka. Default value is `100000`.\n- `kafka.queue_buffering_max_ms` – alias for `queue.buffering.max.ms` from librdkafka. Default value is `10`.\n- `kafka.queue_buffering_max_kbytes` – alias for `queue.buffering.max.kbytes` from librdkafka. Default value is `1048576`.\n- `kafka.retries` – alias for `retries` from librdkafka. Default value is `3`.\n- `kafka.message_timeout_ms` – alias for `message.timeout.ms` from librdkafka. Default value is `2000`.\n- `kafka.request_timeout_ms` – alias for `request.timeout.ms` from librdkafka. Default value is `30000`.\n- `kafka.request_required_acks` – alias for `request.required.acks` from librdkafka. Default value is `-1`.\n- `http.port` – port for HTTP server for producing messages. Default value is `4242`\n- `http.metrics_port` – port for HTTP server for metrics. Default value is `8088`\n- `output_file` – output file for logging. Default value is `/dev/stdout`\n- `ratelimit.enabled` – enable or disable rate limits. Default value is `false`\n- `ratelimit.rules` – rules for rate limits. Default value is `[]`\n\n### Example configuration\n```yaml\nkafka:\n  brokers:\n    - '127.0.0.1:9092'\n  request_required_acks: 1\n  queue_buffering_max_ms: 20\n  queue_buffering_max_kbytes: 2048 # 2 MiB\n\nhttp:\n  port: 4242\n  metrics_port: 8088\n\noutput_file: \"/dev/stdout\"\n\nratelimit: # ratelimit settings.\n  enabled: true # enables or disables ratelimiting.\n  rules: # defines list of rules for concrete topics.\n    - topic_name: \"some_topic_name\" # concrete topic name.\n      max_requests_per_minute: 42 # maximum requests per minute allowed for concrete topic.\n```\n\n## Benchmarks\n\nWe managed to get such results with `wrk` on v0.0.1-alpha version with `4 KiB` message size.\n\nExternal host configuration:\n```text\nCPU: 2-core Intel Core Processor (Broadwell) 2095 MHz.\nRAM: 4 GiB DDR4\nVirtualization: VT-x\nHypervisor vendor: KVM.\n```\n\nKafka-proxy configuration:\n\n```yaml\nkafka:\n  brokers:\n    - \"some_kafka_broker_1.some_external_server\"\n    - \"some_kafka_broker_2.some_external_server\"\n    - \"some_kafka_broker_3.some_external_server\"\n    - \"some_kafka_broker_4.some_external_server\"\n    - \"some_kafka_broker_5.some_external_server\"\n  request_required_acks: 1\n  queue_buffering_max_ms: 10\n  queue_buffering_max_kbytes: 2048 # 2 MiB\n\nhttp:\n  port: 4242\n```\n\n- Asynchronous producing\n```text\nwrk -s wrk.lua -t256 -c256 -d120s 'http://some_external_host:4242/push'\nRunning 2m test @ http://some_external_host:4242/push\n  256 threads and 256 connections\n\n  Thread Stats   Avg      Stdev     Max   +/- Stdev\n    Latency    20.72ms   11.53ms 145.52ms   72.16%\n    Req/Sec    49.57     21.97     1.02k    87.15%\n  864241 requests in 1.14m, 108.79MB read\nRequests/sec:  12672.49\nTransfer/sec:      1.60MB\n```\n\n- Synchronous producing\n```text\n  256 threads and 256 connections\n  Thread Stats   Avg      Stdev     Max   +/- Stdev\n    Latency    31.41ms   88.92ms   1.01s    96.82%\n    Req/Sec    59.35     13.91   111.00     74.87%\n  1765717 requests in 2.00m, 222.28MB read\nRequests/sec:  14701.92\nTransfer/sec:      1.85MB\n```\n\n## Metrics\n\nAll metrics are available in OpenMetrics(prometheus) format on `:8088` port(non-configurable at v0.0.1-alpha).\n\nAt this moment(v0.1.0) these metrics are available:\n\n- `http_requests_duration` – Histogram of HTTP requests. Label `push/sync` is set when message produced asynchronously, \notherwise, `push/async` is set. `code` label is set according to HTTP response status.\n- `kafka_internal_queue_size` – Gauge of internal kafka-queue size, per topic.\n- `kafka_message_send_duration` – Histogram of kafka message duration before delivery result callback is received, per topic.\n- `kafka_sent_messages` – Counter of total kafka messages sent, per topic.\n- `kafka_errors_count` – Counter of total kafka errors, per topic.\n- `ratelimit_messages_count` – Counter of total ratelimited messages, per topic.\n\nKafka librdkafka metrics:\n- `kafka_producer_reply_queue_size` – Operations (callbacks, events, etc.) waiting in queue.\n- `kafka_producer_current_messages_in_queue` – Current number of messages in producer queues.\n- `kafka_producer_current_messages_in_queue_bytes` – Current total size of messages in producer queues.\n- `kafka_producer_total_requests_count` – Total number of requests sent to brokers\n- `kafka_producer_total_bytes_sent` – Total number of bytes transmitted to brokers\n- `kafka_producer_total_responses_received` – Total number of responses received from brokers\n- `kafka_producer_total_bytes_received` – Total number of bytes received from brokers\n- `kafka_producer_total_messages_sent` – Total number of messages transmitted (produced) to brokers\n- `kafka_producer_total_messages_bytes_sent` – Total number of bytes transmitted (produced) to brokers\n- `kafka_producer_metadata_cache_topics_count` – Number of topics in the metadata cache\n- `kafka_producer_broker_state` – Broker state (INIT, DOWN, CONNECT, AUTH, APIVERSION_QUERY, AUTH_HANDSHAKE, UP, UPDATE).\n- `kafka_producer_broker_state_age` – The time since the last broker state change, in microseconds\n- `kafka_producer_broker_outbuf_count` – Number of requests awaiting transmission to the broker\n- `kafka_producer_broker_outbuf_msg_count` – Number of messages awaiting transmission to the broker\n- `kafka_producer_broker_waitresp_count` – Number of requests in-flight to the broker that are awaiting response\n- `kafka_producer_broker_waitresp_msg_count` – Number of messages awaiting transmission to the broker\n- `kafka_producer_broker_requests_sent` – Total number of requests sent to the broker\n- `kafka_producer_broker_requests_sent_bytes` – Total number of bytes sent to the broker\n- `kafka_producer_broker_transmission_errors` – Total number of transmission errors\n- `kafka_producer_broker_request_retries` – Total number of request retries\n- `kafka_producer_request_timeouts` – Total number of requests that timed out\n- `kafka_producer_broker_responses_count` – Total number of responses received from the broker\n- `kafka_producer_broker_bytes_received` – Total number of bytes received from the broker\n- `kafka_producer_broker_errors_count` – Total number of received errors\n- `kafka_producer_topic_metadata_age` – The age of the client's metadata for this topic, in milliseconds\n- `kafka_producer_topic_batchsize_avg` – Rolling window statistics for batch sizes, in bytes\n- `kafka_producer_topic_batchcount_avg` – Rolling window statistics for batch message counts\n- `kprf_app_metadata` – KPRF application metadata (commit_hash, version)\n\n## Further improvements\n\n1. `Write-Ahead-Log`. Write-Ahead-Log can be a good improvement if pattern of usage is asynchronous producing. Client does not know\nif his records were successfully sent to kafka. Adding `WAL` can improve durability.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcity-mobil%2Fkafka-proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcity-mobil%2Fkafka-proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcity-mobil%2Fkafka-proxy/lists"}