{"id":15672772,"url":"https://github.com/elijahr/ringbuf","last_synced_at":"2025-05-06T22:14:21.055Z","repository":{"id":57461984,"uuid":"246482745","full_name":"elijahr/ringbuf","owner":"elijahr","description":"A lock-free ring buffer for Python and Cython","archived":false,"fork":false,"pushed_at":"2022-10-01T00:38:36.000Z","size":108,"stargazers_count":18,"open_issues_count":0,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-06T22:14:10.824Z","etag":null,"topics":["audio","cython","embedded","python","ringbuffer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elijahr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-11T05:28:19.000Z","updated_at":"2025-04-21T23:50:32.000Z","dependencies_parsed_at":"2023-01-19T04:01:58.332Z","dependency_job_id":null,"html_url":"https://github.com/elijahr/ringbuf","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elijahr%2Fringbuf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elijahr%2Fringbuf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elijahr%2Fringbuf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elijahr%2Fringbuf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elijahr","download_url":"https://codeload.github.com/elijahr/ringbuf/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252776600,"owners_count":21802469,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["audio","cython","embedded","python","ringbuffer"],"created_at":"2024-10-03T15:31:31.535Z","updated_at":"2025-05-06T22:14:20.755Z","avatar_url":"https://github.com/elijahr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ringbuf\n\nA lock-free, single-producer, single-consumer, ring buffer for Python and Cython.\n\n[![Test](https://github.com/elijahr/ringbuf/actions/workflows/test.yml/badge.svg)](https://github.com/elijahr/ringbuf/actions/workflows/test.yml) [![PyPI version](https://badge.fury.io/py/ringbuf.svg)](https://badge.fury.io/py/ringbuf) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![linting: pylint](https://img.shields.io/badge/linting-pylint-yellowgreen)](https://github.com/PyCQA/pylint)\n\n## Installation\n\nOS X: `brew install boost`\n\nUbuntu: `apt-get install libboost-all-dev`\n\nWindows: Install the latest version of [`Boost`](https://www.boost.org/) then set the `BOOST_ROOT` environment variable to point to its folder.\n\nThen:\n\n```shell\npip install ringbuf\n```\n\n## Motivation\n\nWhen working with realtime DSP in Python, we might be wrapping some external C/C++ library (for instance, PortAudio) which runs some user-provided callback function in realtime. The callback function shouldn't allocate/deallocate memory, shouldn't contain any critical sections (mutexes), and so forth, to prevent priority inversion. If the callback were to contain Python objects, we'd likely be allocating and deallocating, and at the very least, acquiring and releasing the GIL. So, the callback cannot interact with Python objects if we expect realtime performance. As such, there's a need for buffering data in a non-locking way between a C/C++ callback and Python.\n\nEnter ringbuf, Cython wrappers for [`boost::lockfree::spsc_queue`](https://www.boost.org/doc/libs/1_72_0/doc/html/boost/lockfree/spsc_queue.html). Our Python code can read from and write to a `ringbuf.RingBuffer` object, and our C++ code can read from and write to that buffer's underlying `spsc_queue`, no GIL required.\n\n## Usage\n\nAny Python object which supports the [buffer protocol](https://docs.python.org/3/c-api/buffer.html) can be stored in `ringbuf.RingBuffer`. This includes, but is not limited to: `bytes`, `bytearray`, `array.array`, and `numpy.ndarray`.\n\n### NumPy\n\n```python\nimport numpy as np\nfrom ringbuf import RingBuffer\n\nbuffer = RingBuffer(format='f', capacity=100)\n\ndata = np.linspace(-1, 1, num=100, dtype='f')\n\nbuffer.push(data)\n\npopped = buffer.pop(100)\n\nassert np.array_equal(data, popped)\n```\n\n### bytes\n\n```python\nfrom ringbuf import RingBuffer\n\nbuffer = RingBuffer(format='B', capacity=11)\n\nbuffer.push(b'hello world')\n\npopped = buffer.pop(11)\n\nassert bytes(popped) == b'hello world'\n```\n\n### Interfacing with C/C++\n\nmymodule.pxd:\n\n```cython\n# distutils: language = c++\ncdef void callback(void* q)\n```\n\nmymodule.pyx:\n\n```cython\n# distutils: language = c++\nfrom array import array\n\nfrom ringbuf.boost cimport spsc_queue, void_ptr_to_spsc_queue_char_ptr\nfrom ringbuf.ringbufcy cimport RingBuffer\n\nfrom some_c_library cimport some_c_function\n\n\ncdef void callback(void* q):\n    cdef:\n        # Cast the void* back to an spsc_queue.\n        # The underlying queue always holds chars.\n        spsc_queue[char] *queue = void_ptr_to_spsc_queue_char_ptr(q)\n        double[5] to_push = [1.0, 2.0, 3.0, 4.0, 5.0]\n\n    # Since the queue holds chars, you'll have to cast and adjust size accordingly.\n    queue.push(\u003cchar*\u003eto_push, sizeof(double) * 5)\n\n\ndef do_stuff():\n    cdef:\n        RingBuffer buffer = RingBuffer(format='d', capacity=100)\n        void* queue = buffer.queue_void_ptr()\n\n    # Pass our callback and a void pointer to the buffer's queue to some third party library.\n    # Presumably, the C library schedules the callback and passes it the queue's void pointer.\n    some_c_function(callback, queue)\n\n    sleep(1)\n\n    assert array.array('d', buffer.pop(5)) == array.array('d', range(1, 6))\n```\n\n### Handling overflow \u0026 underflow\n\nWhen `RingBuffer.push()` overflows, it returns the data that couldn't be pushed (or None, if all was pushed):\n\n```python\nfrom ringbuf import RingBuffer\n\nbuffer = RingBuffer(format='B', capacity=10)\noverflowed = buffer.push(b'spam eggs ham')\nassert overflowed == b'ham'\n```\n\nWhen `RingBuffer.pop()` underflows, it returns whatever data could be popped:\n\n```python\nfrom ringbuf import RingBuffer\n\nbuffer = RingBuffer(format='B', capacity=13)\nbuffer.push(b'spam eggs ham')\npopped = buffer.pop(buffer.capacity * 100)\nassert bytes(popped) == b'spam eggs ham'\n```\n\nFor additional usage see the [tests](https://github.com/elijahr/ringbuf/blob/master/test.py).\n\n## Supported platforms\n\nGitHub Actions tests the following matrix:\n\n- Linux:\n  - CPython 3.7\n  - CPython 3.8\n  - CPython 3.9\n  - CPython 3.10\n- macOS:\n  - CPython 3.10\n- Windows:\n  - CPython 3.10\n\nAny platform with a C++11 compiler and boost installed should work.\n\n## Contributing\n\nPull requests are welcome, please file any issues you encounter.\n\nThe code is linted with [lintball](https://github.com/elijahr/lintball). There is a pre-commit hook to lint, configured by running:\n\n```shell\nnpm install -g lintball\ngit config --local core.hooksPath .githooks\n```\n\n## Changelog\n\n### v2.6.0 2022-09-27\n\n- Move CI to GitHub Actions.\n- Lint codebase with lintball\n- Improve project structure\n\n### v2.5.0 2020-04-17\n\n- Added experimental support for Windows.\n\n### v2.4.0 2020-03-23\n\n- Added `RingBuffer.reset()` method to clear the buffer.\n\n### v2.3.0 2020-03-22\n\n- Added `concatenate` function for joining multiple arbitrary Python objects that support the buffer protocol.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felijahr%2Fringbuf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felijahr%2Fringbuf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felijahr%2Fringbuf/lists"}