{"id":20271686,"url":"https://github.com/ossmafia/fastapi-kafka","last_synced_at":"2025-04-11T04:32:42.732Z","repository":{"id":252726768,"uuid":"840579225","full_name":"OSSMafia/fastapi-kafka","owner":"OSSMafia","description":"A Kafka wrapper class around FastAPI to create consumers as endpoints.","archived":false,"fork":false,"pushed_at":"2024-08-14T20:36:39.000Z","size":28,"stargazers_count":3,"open_issues_count":4,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-04-06T23:18:37.478Z","etag":null,"topics":["api-rest","fastapi","kafka","kafka-consumer","python","upstash-kafka"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OSSMafia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-10T04:20:29.000Z","updated_at":"2025-02-25T07:31:24.000Z","dependencies_parsed_at":"2024-08-14T22:17:15.556Z","dependency_job_id":null,"html_url":"https://github.com/OSSMafia/fastapi-kafka","commit_stats":null,"previous_names":["ossmafia/fastkafka"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSSMafia%2Ffastapi-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSSMafia%2Ffastapi-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSSMafia%2Ffastapi-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSSMafia%2Ffastapi-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OSSMafia","download_url":"https://codeload.github.com/OSSMafia/fastapi-kafka/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248345202,"owners_count":21088231,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-rest","fastapi","kafka","kafka-consumer","python","upstash-kafka"],"created_at":"2024-11-14T12:39:05.649Z","updated_at":"2025-04-11T04:32:42.667Z","avatar_url":"https://github.com/OSSMafia.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FastAPI Kafka\n\nThis package is a wrapper around the well-known [FastAPI](https://pypi.org/project/fastapi/) framework for introducing Kafka consumers as routes. No core FastAPI functionality has been altered and the documentation can be found [here](https://fastapi.tiangolo.com/).\n\nInspiration for this package comes from working on a NestJS API using [KafkaJS](https://docs.nestjs.com/microservices/kafka) where kafka consumer topics are defined as controllers. It's a very convenient pattern to work with.\n\nNew project so definitely more work to be done, contributions welcome.\n\n\n## Installation\n\nInstall FastAPI Kafka with pip\n\n```bash\n  pip install fastapi_kafka\n```\n    \n## Usage/Examples\n\nUnder the hood FastAPI Kafka uses [AIOKafka](https://github.com/aio-libs/aiokafka) to create a consumer. You define a consumer with the following, it matches the exact class inputs that [AIOKafkaConsumer](https://aiokafka.readthedocs.io/en/stable/api.html#aiokafka.AIOKafkaConsumer) uses:\n\n```python\nclass KafkaConsumerConfig:\n    loop: Optional[asyncio.AbstractEventLoop] = None\n    bootstrap_servers: str = \"localhost\"\n    client_id: Optional[str] = None\n    group_id: Optional[str] = None\n    group_instance_id: Optional[str] = None\n    key_deserializer: Optional[Any] = None\n    value_deserializer: Optional[Any] = None\n    fetch_max_wait_ms: int = 500\n    fetch_max_bytes: int = 52428800  # 50 MB\n    fetch_min_bytes: int = 1\n    max_partition_fetch_bytes: int = 1 * 1024 * 1024  # 1 MB\n    request_timeout_ms: int = 40 * 1000  # 40 seconds\n    retry_backoff_ms: int = 100\n    auto_offset_reset: str = \"latest\"\n    enable_auto_commit: bool = True\n    auto_commit_interval_ms: int = 5000\n    check_crcs: bool = True\n    metadata_max_age_ms: int = 5 * 60 * 1000  # 5 minutes\n    partition_assignment_strategy: tuple[Any, ...] = (RoundRobinPartitionAssignor,)\n    max_poll_interval_ms: int = 300000  # 5 minutes\n    rebalance_timeout_ms: Optional[int] = None\n    session_timeout_ms: int = 10000  # 10 seconds\n    heartbeat_interval_ms: int = 3000  # 3 seconds\n    consumer_timeout_ms: int = 200  # 200 ms\n    max_poll_records: Optional[int] = None\n    ssl_context: Optional[ssl.SSLContext] = None\n    security_protocol: str = \"PLAINTEXT\"\n    api_version: str = \"auto\"\n    exclude_internal_topics: bool = True\n    connections_max_idle_ms: int = 540000  # 9 minutes\n    isolation_level: str = \"read_uncommitted\"\n    sasl_mechanism: str = \"PLAIN\"\n    sasl_plain_password: Optional[str] = None\n    sasl_plain_username: Optional[str] = None\n    sasl_kerberos_service_name: str = \"kafka\"\n    sasl_kerberos_domain_name: Optional[str] = None\n    sasl_oauth_token_provider: Optional[Any] = None\n```\n\n---\n### Simple Example\n\nHere's a full example running a local kafka server:\n```python\nfrom typing import Any\n\nfrom pydantic import BaseModel\n\nfrom fastapi_kafka import FastAPIKafka\nfrom fastapi_kafka.consumer import KafkaConsumerConfig\n\n\nclass SomeMessage(BaseModel):\n    test: str\n\n\napp = FastAPIKafka(\n    kafka_consumer_config=KafkaConsumerConfig(\n        bootstrap_servers=\"localhost:9092\"\n    )\n)\n\n\n@app.get(\"/\")\ndef read_root():\n    return {\"Hello\": \"World\"}\n\n\n@app.consumer(\"test-topic\")\nasync def read_item(item: Any):\n    return item\n\n\n@app.consumer(\"test-topic-model\")\nasync def read_item_model(item: SomeMessage):\n    return item\n```\n\nThe FastAPIKafka can accept all standard constructor inputs as well.\n\nTo define a consumer topic you use `@app.consumer(\"\u003ctopic name\u003e\")`, you can define a kafka message as `Any` and it will return a UTF-8 decoded message when reading the `item`. \n\nIf you use JSON messages you can define a pydantic model to read it much the same as a regular http route, the message will be transformed into an instance of that model.\n\n\n### More Advanced Example (Using Upstash Kafka)\n```python\nfrom typing import Any\nfrom uuid import uuid4\n\nfrom pydantic import BaseModel\n\nfrom fastapi_kafka import FastAPIKafka, create_ssl_context\nfrom fastapi_kafka.consumer import KafkaConsumerConfig\n\n\nclass SomeMessage(BaseModel):\n    test: str\n\n\nconsumer_config = KafkaConsumerConfig(\n    bootstrap_servers='example-us1-kafka.upstash.io:9092',\n    sasl_mechanism='SCRAM-SHA-256',\n    security_protocol='SASL_SSL',\n    ssl_context=create_ssl_context(),\n    sasl_plain_username='xxx',\n    sasl_plain_password='xxx',\n    client_id=f'consumer-{uuid4()}',\n    group_id='consumer-group-api',\n    auto_offset_reset='earliest',\n)\n\napp = FastAPIKafka(kafka_consumer_config=consumer_config)\n\n\n@app.get(\"/\")\ndef read_root():\n    return {\"Hello\": \"World\"}\n\n\n@app.consumer(\"test-topic\")\nasync def read_item(item: Any):\n    return item\n\n\n@app.consumer(\"test-topic-model\")\nasync def read_item_model(item: SomeMessage):\n    return item\n\n```\n\nFor `security_protocol` using `SASL_SSL` you must pass in a `create_ssl_context()`.\n\nFor even more advanced configuration such as defining your own certificates, more information can be found here: [AIOKafka SSL](https://aiokafka.readthedocs.io/en/stable/examples/ssl_consume_produce.html).\n\n### FastAPI Lifespan\n\nOne material change that was made to FastAPI was how [Lifespan](https://fastapi.tiangolo.com/advanced/events/?h=lifespan) works. FastAPI Kafka initializes the Lifespan automatically to create the defined Kafka consumer, if you need additional start-up/shutdown behavior you can use the `startup_functions` or `shutdown_functions` when initializing the `FastAPIKafka` instance.\n\nThe `startup_functions` or `shutdown_functions` fields accept a list of synchronous or asynchronous functions to be executed before/after the yield in lifespan.\n\n```python\n\nasync def my_startup_actions():\n    await do_something()\n    do_something_sync()\n\nasync def my_shutdown_actions():\n    await do_something_on_shutdown()\n\napp = FastAPIKafka(\n    kafka_consumer_config=consumer_config,\n    startup_functions=[my_startup_actions],\n    shutdown_functions=[my_shutdown_actions]\n)\n```\n## Acknowledgements\n\n Building upon the awesome [FastAPI](https://pypi.org/project/fastapi/) framework.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fossmafia%2Ffastapi-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fossmafia%2Ffastapi-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fossmafia%2Ffastapi-kafka/lists"}