https://github.com/karolinepauls/pytest-kafka

ZK and Kafka fixtures for Pytest. Mirror of https://gitlab.com/karolinepauls/pytest-kafka
https://github.com/karolinepauls/pytest-kafka

fixture kafka pytest zookeeper

Last synced: 5 months ago
JSON representation

ZK and Kafka fixtures for Pytest. Mirror of https://gitlab.com/karolinepauls/pytest-kafka

• Host: GitHub
• URL: https://github.com/karolinepauls/pytest-kafka
• Owner: karolinepauls
• License: mit
• Created: 2019-03-28T19:09:57.000Z (about 7 years ago)
• Default Branch: master
• Last Pushed: 2024-08-14T15:12:07.000Z (almost 2 years ago)
• Last Synced: 2025-04-04T18:04:14.494Z (about 1 year ago)
• Topics: fixture, kafka, pytest, zookeeper
• Language: Python
• Homepage: https://pytest-kafka.readthedocs.io/en/latest/
• Size: 52.7 KB
• Stars: 1
• Watchers: 0
• Forks: 2
• Open Issues: 1
• Metadata Files:
  • Readme: README.rst

Awesome Lists containing this project

README

          
pytest-kafka |pypi| |pyversions| |ci| |docs|

============================================

.. |pypi| image:: https://img.shields.io/pypi/v/pytest-kafka.svg

    :target: https://pypi.org/project/pytest-kafka/

.. |pyversions| image:: https://img.shields.io/pypi/pyversions/pytest-kafka.svg

    :target: https://pypi.org/project/pytest-kafka/

.. |ci| image:: https://gitlab.com/karolinepauls/pytest-kafka/badges/master/pipeline.svg

    :target: https://gitlab.com/karolinepauls/pytest-kafka/pipelines

.. |docs| image:: https://readthedocs.org/projects/pytest-kafka/badge/?version=latest

    :target: https://pytest-kafka.readthedocs.io/en/latest/

Pytest fixture factories for Zookeeper, Kafka server and Kafka consumer.

`Read the API docs `__.

.. warning::

    ``kafka-python`` doesn't work with Python 3.12. To solve this problem, pytest-kafka, from 0.8.x, no

    longer depends on ``kafka-python``. You can depend on extras: ``pytest-kafka[kafka-python]`` or

    ``pytest-kafka[kafka-python-ng]`` for Python>3.12.

.. code:: python

    from pathlib import Path

    from pytest_kafka import (

        make_zookeeper_process, make_kafka_server, make_kafka_consumer,

        terminate,

    )

    ROOT = Path(__file__).parent

    KAFKA_SCRIPTS = ROOT / 'kafka/bin/'

    KAFKA_BIN = str(KAFKA_SCRIPTS / 'kafka-server-start.sh')

    ZOOKEEPER_BIN = str(KAFKA_SCRIPTS / 'zookeeper-server-start.sh')

    # You can pass a custom teardown function (or parametrise ours). Just don't call it `teardown`

    # or Pytest will interpret it as a module-scoped teardown function.

    teardown_fn = partial(terminate, signal_fn=Popen.kill)

    zookeeper_proc = make_zookeeper_process(ZOOKEEPER_BIN, teardown_fn=teardown_fn)

    kafka_server = make_kafka_server(KAFKA_BIN, 'zookeeper_proc', teardown_fn=teardown_fn)

    kafka_consumer = make_kafka_consumer(

        'kafka_server', seek_to_beginning=True, kafka_topics=['topic'])

This creates 3 fixtures:

#. ``zookeeper_proc`` - Zookeeper process

#. ``kafka_server`` - Kafka process

#. ``kafka_consumer`` - usable ``kafka.KafkaConsumer`` instance

``ZOOKEEPER_BIN`` and ``KAFKA_BIN`` are paths to launch scripts in your Kafka distribution. Check

this project's setup.py to see a way of installing Kafka for development.

It is advised to pass ``seek_to_beginning=True`` because otherwise some messages may not be captured

by the consumer. This requires knowing the topics upfront because without topics there's no

partitions to seek.

Kafka server is known to take a couple of seconds to terminate gracefully. You probably don't

need that, so you can pass ``partial(terminate, signal_fn=Popen.kill)`` to make it killed with

SIGKILL and waited for afterwards.

It's possible to create multiple Kafka fixtures forming a cluster by passing the same Zookeeper

fixture to them. For an example, check the `tests

`__.

Session-scoped fixtures are also available. Consult the `test suite

`__.

System requirements

-------------------

- Python 3.6+

- a JVM that can run Kafka and Zookeeper

- Kafka - https://kafka.apache.org/downloads

Development

-----------

.. code:: sh

   pip install -e .[dev]

   ./pytest_kafka/install.py  # will install kafka to ./kafka

Acknowledgements

----------------

The library has been open-sourced from a codebase belonging to

`Infectious Media `__.