https://github.com/lostclus/django-kafka-streamer

Stream data from Django application to Apache Kafka
https://github.com/lostclus/django-kafka-streamer

data-synchronization django kafka message-bus message-queue microservices straming

Last synced: 7 months ago
JSON representation

Stream data from Django application to Apache Kafka

• Host: GitHub
• URL: https://github.com/lostclus/django-kafka-streamer
• Owner: lostclus
• License: bsd-2-clause
• Created: 2023-01-13T17:47:52.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2025-01-29T09:25:40.000Z (10 months ago)
• Last Synced: 2025-04-04T18:48:22.507Z (8 months ago)
• Topics: data-synchronization, django, kafka, message-bus, message-queue, microservices, straming
• Language: Python
• Homepage:
• Size: 75.2 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.rst
  • License: LICENSE

Awesome Lists containing this project

README

          
django-kafka-streamer

=====================

.. image:: https://github.com/lostclus/django-kafka-streamer/actions/workflows/tests.yml/badge.svg

    :target: https://github.com/lostclus/django-kafka-streamer/actions

.. image:: https://readthedocs.org/projects/django-kafka-streamer/badge/?version=latest

    :target: http://django-kafka-streamer.readthedocs.io/

    :alt: Documentation

.. image:: https://img.shields.io/pypi/v/django-kafka-streamer.svg

    :target: https://pypi.org/project/django-kafka-streamer/

    :alt: Current version on PyPi

.. image:: https://img.shields.io/pypi/pyversions/django-kafka-streamer

    :alt: PyPI - Python Version

.. image:: https://img.shields.io/pypi/djversions/django-kafka-streamer

    :alt: PyPI - Django Version

.. image:: https://img.shields.io/badge/Published%20on-Django%20Packages-0c3c26

    :target: https://djangopackages.org/packages/p/django-kafka-streamer/

    :alt: Published on Django Packages

django-kafka-streamer is a Django application and library for streaming data to

Apache Kafka.

Features:

* Setup signal handlers to ORM models to transparently send create/update/delete

  events to Kafka

* Handle database object relations

* Celery task to stream large amount of data in background

Links:

* GitHub: https://github.com/lostclus/django-kafka-streamer/

* PyPI: https://pypi.org/project/django-kafka-streamer/

* Documentation: http://django-kafka-streamer.readthedocs.io/

* Consumer library: https://github.com/lostclus/aiosafeconsumer

* Example application: https://github.com/lostclus/WeatherApp

Usage:

`yourapp/models.py`::

    from django.db import models

    class MyModel(models.Model):

        field1 = models.IntegerField()

        field2 = models.CharField(max_length=10)

`yourapp/stramers.py`::

    from kafkastreamer import Streamer, register

    from .models import MyModel

    @register(MyModel)

    class MyModelStreamer(Streamer):

        topic = "model-a"

`yourproject/settings.py`::

    INSTALLED_APPS = [

        ...

        "kafkastreamer",

    ]

    KAFKA_STREAMER = {

        "BOOTSTRAP_SERVERS": ["localhost:9092"],

    },

Any changes in ``MyModel`` data will be automatically streamed to Kafka. To

force stream all data in all registered models type::

    python manage.py kafkastreamer_refresh

The data streamed to the ``model-a`` Kafka topic has following structure::

    {

        "_time": "2023-01-01T00:00:00Z",

        "_type": "create",

        "id": 1,

        "field1": 1,

        "field2": "abc"

    }