{"id":19758832,"url":"https://github.com/daq-tools/lorrystream","last_synced_at":"2025-04-30T12:32:17.054Z","repository":{"id":176545656,"uuid":"624238771","full_name":"daq-tools/lorrystream","owner":"daq-tools","description":"A lightweight and polyglot stream-processing library, to be used as a data backplane-, message relay-, or pipeline-subsystem.","archived":false,"fork":false,"pushed_at":"2025-01-20T00:18:25.000Z","size":266,"stargazers_count":4,"open_issues_count":9,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-06T01:51:12.850Z","etag":null,"topics":["amqp","broker","cratedb","data-stream","data-stream-processing","data-stream-processing-framework","data-streaming","kotori-daq","message-broker","message-bus","message-queue","mosquitto","mqtt","pandas","sqlalchemy","stream","streaming","streamz","zeromq","zmq"],"latest_commit_sha":null,"homepage":"https://lorrystream.readthedocs.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/daq-tools.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-06T03:19:11.000Z","updated_at":"2025-03-05T22:17:52.000Z","dependencies_parsed_at":"2023-10-01T22:33:12.690Z","dependency_job_id":"b35d6722-f8be-42ad-953f-f15c7c5e87e6","html_url":"https://github.com/daq-tools/lorrystream","commit_stats":null,"previous_names":["daq-tools/lorrystream"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daq-tools%2Florrystream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daq-tools%2Florrystream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daq-tools%2Florrystream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daq-tools%2Florrystream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/daq-tools","download_url":"https://codeload.github.com/daq-tools/lorrystream/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251701783,"owners_count":21629896,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amqp","broker","cratedb","data-stream","data-stream-processing","data-stream-processing-framework","data-streaming","kotori-daq","message-broker","message-bus","message-queue","mosquitto","mqtt","pandas","sqlalchemy","stream","streaming","streamz","zeromq","zmq"],"created_at":"2024-11-12T03:26:10.334Z","updated_at":"2025-04-30T12:32:16.632Z","avatar_url":"https://github.com/daq-tools.png","language":"Python","readme":"# LorryStream\n\n## About\n\n**LorryStream** is a lightweight and polyglot stream-processing library,\nto be used as a data backplane-, message relay-, or pipeline-subsystem,\nin the spirit of [socat] and [GStreamer]. It is based on [Streamz],\n[Dask], and other Python libraries.\n\nYou can use **LorryStream** to store data received from the network into\ndatabases, or to relay it back to the network, for example into\ndifferent bus systems. It can be used both as a standalone program, and\nas a library.\n\nIt is conceived to generalize and improve the corresponding subsystems\nof programs and frameworks like [Kotori], [Wetterdienst], [Luftdatenpumpe],\n[amqp-forward], [ttnlogger], [Kahn], or [mqttwarn].\n\n\n## Synopsis\n\nThe canonical command is `lorry relay \u003csource\u003e \u003csink\u003e`. Please note\n`%23` is `#`.\n\n``` sh\nlorry relay \\\n    \"mqtt://localhost/testdrive/%23\" \\\n    \"crate://localhost/?table=testdrive\"\n```\n\nIf you prefer a GStreamer-like pipeline definition syntax.\n\n``` sh\nlorry launch \"mqttsrc location=mqtt://localhost/testdrive/%23 ! sqlsink location=crate://localhost/?table=testdrive\"\n```\n\n## Quickstart\n\nIf you are in a hurry, and want to run LorryStream without any\ninstallation, just use the OCI image on Podman or Docker.\n\n``` sh\ndocker run --rm --network=host ghcr.io/daq-tools/lorrystream \\\n    lorry relay \\\n    \"mqtt://localhost/testdrive/%23\" \\\n    \"crate://localhost/?table=testdrive\"\n```\n\n## Setup\n\nInstall `lorrystream` from PyPI.\n\n``` sh\npip install lorrystream\n```\n\n## Usage\n\nThis section outlines some example invocations of LorryStream, both on\nthe command line, and per library use. Other than the resources\navailable from the web, testing data can be acquired from the\nrepository's [testdata] folder.\n\n### Prerequisites\n\nFor properly running some of the example invocations outlined below, you\nwill need a few servers. The easiest way to spin up those instances is\nto use Podman or Docker.\n\n``` sh\ndocker run --name=mosquitto --rm -it --publish=1883:1883 \\\n    eclipse-mosquitto:2.0.15 mosquitto -c /mosquitto-no-auth.conf\n```\n\n-- \u003chttps://github.com/docker-library/docs/blob/master/eclipse-mosquitto/README.md\u003e\n\n``` sh\ndocker run --name=cratedb --rm -it --publish=4200:4200 --publish=5432:5432 \\\n    crate:5.2 -Cdiscovery.type=single-node\n```\n\n-- \u003chttps://github.com/docker-library/docs/blob/master/crate/README.md\u003e\n\n### Command line use\n\n#### Help\n\n``` sh\nlorry --help\nlorry info\nlorry relay --help\n```\n\n#### Bus to storage\n\n``` sh\n# Relay messages from MQTT to CrateDB.\nlorry relay \\\n    \"mqtt://localhost/testdrive/%23\" \\\n    \"crate://localhost/?table=testdrive\"\n```\n\n#### Bus to bus\n\n``` sh\n# Relay messages from AMQP to MQTT.\nlorry relay \\\n    \"amqp://localhost/testdrive/demo\" \\\n    \"mqtt://localhost/testdrive/demo\"\n```\n\n### Library use\n\n``` python\n\u003e\u003e\u003e from lorrystream import parse_launch\n\u003e\u003e\u003e parse_launch(\"mqttsrc location=mqtt://localhost/testdrive/%23 ! sqlsink location=crate://localhost/?table=testdrive\")\n```\n\n#### OCI\n\nOCI images are available on the GitHub Container Registry (GHCR). We are\npublishing image variants for general availability- and\nnightly-releases, and pull requests.\n\nIn order to always run the latest `nightly` development version, and to\nuse a shortcut for that, this section outlines how to use an alias for\n`lorry`, and a variable for storing the data source and sink URIs. It\nmay be useful to save a few keystrokes on subsequent invocations.\n\n``` sh\ndocker pull ghcr.io/daq-tools/lorrystream:nightly\nalias lorry=\"docker run --rm --interactive ghcr.io/daq-tools/lorrystream:nightly lorry\"\nSOURCE=mqtt://localhost/testdrive/%23\nSINK=crate://crate@localhost:4200/?table=testdrive\n\nlorry relay \"${SOURCE}\" \"${SINK}\"\n```\n\n\n## Story\n\n### Details\n\n- Data sources are message bus systems like AMQP, Kafka, MQTT, ZeroMQ,\n  and network listener endpoints for TCP, UDP, HTTP, and WebSocket.\n- Data sinks are RDBMS databases supported by SQLAlchemy, or other\n  message brokers.\n\n### Motivation\n\n- Implement a reusable solution, simple to install and operate, that\n  doesn't depend on vendor-provided infrastructure, and can easily be\n  embedded into existing frameworks and software stacks, or integrated\n  otherwise by running it as a separate service.\n- Help the community and industry to modernize their aging DAQ backend\n  systems designed within the previous decades.\n- Use as pipeline elements, protocol translator, bridge elements.\n\n### Background\n\n\u003e Flow-Based Programming ([FBP]) is a programming paradigm that uses a\n\u003e \"data processing factory\" metaphor for designing and building applications.\n\u003e It is a special case of [dataflow] programming characterized by asynchronous,\n\u003e concurrent processes \"under the covers\".\n\u003e\n\u003e FBP has been found to support improved development time and\n\u003e maintainability, reusability, rapid prototyping, simulation, improved\n\u003e performance, and good communication among developers, maintenance\n\u003e staff, users, systems people, and management - not to mention that FBP\n\u003e naturally takes advantage of multiple cores, without the programmer\n\u003e having to struggle with the intricacies of multitasking.\n\u003e\n\u003e -- [Flow-based Programming]\n\n### Caveat\n\nPlease note that LorryStream is alpha-quality software, and a work in\nprogress. Contributions of all kinds are very welcome, in order to make\nit more solid.\n\nBreaking changes should be expected until a 1.0 release, so version\npinning is recommended, especially when you use it as a library.\n\n\n## Project information\n\n\n### Contributions\n\nThe LorryStream library is an open source project, and is [managed on\nGitHub]. Every kind of contribution, feedback, or patch, is much welcome.\n[Create an issue] or submit a patch if you think we should include a new\nfeature, or to report or fix a bug.\n\n### Development\n\nIn order to setup a development environment on your workstation, please\nhead over to the [development sandbox]\ndocumentation. When you see the software tests succeed, you should be\nready to start hacking.\n\n### License\n\nThe project is licensed under the terms of the LGPL license, see\n[LICENSE].\n\n### Prior art\n\nWe are maintaining a [list of other projects] with\nthe same or similar goals like LorryStream.\n\n### Kudos\n\n- [J. Paul Rodker Morrison] for discovering/inventing the Flow-Based\n  Programming ([FBP]) paradigm in the late '60s.\n\n- [Matthew Rocklin], [Christopher J. 'CJ' Wright], and [Chinmay Chandak]\n  for conceiving [Streamz].\n\n\n[amqp-forward]: https://github.com/daq-tools/amqp-forward\n[Chinmay Chandak]: https://github.com/chinmaychandak\n[Christopher J. 'CJ' Wright]: https://github.com/CJ-Wright\n[Create an issue]: https://github.com/daq-tools/lorrystream/issues\n[Dask]: https://github.com/dask/dask\n[Dataflow]: https://en.wikipedia.org/wiki/Dataflow\n[development sandbox]: https://github.com/daq-tools/lorrystream/blob/main/doc/development.rst\n[FBP]: https://en.wikipedia.org/wiki/Flow-based_programming\n[Flow-based Programming]: https://jpaulm.github.io/fbp/\n[fsspec]: https://pypi.org/project/fsspec/\n[GStreamer]: https://en.wikipedia.org/wiki/GStreamer\n[J. Paul Rodker Morrison]: https://jpaulm.github.io/\n[Kahn]: https://github.com/maritime-labs/kahn\n[Kotori]: https://github.com/daq-tools/kotori\n[LICENSE]: https://github.com/daq-tools/lorrystream/blob/main/LICENSE\n[list of other projects]: https://github.com/daq-tools/lorrystream/blob/main/doc/prior-art.rst\n[Luftdatenpumpe]: https://github.com/earthobservations/luftdatenpumpe\n[managed on GitHub]: https://github.com/daq-tools/lorrystream\n[Matthew Rocklin]: https://github.com/mrocklin\n[mqttwarn]: https://github.com/jpmens/mqttwarn\n[pandas]: https://pandas.pydata.org/\n[socat]: http://www.dest-unreach.org/socat/\n[SQLAlchemy]: https://pypi.org/project/SQLAlchemy/\n[Streamz]: https://github.com/python-streamz/streamz\n[testdata]: https://github.com/daq-tools/lorrystream/tree/main/tests/testdata\n[ttnlogger]: https://github.com/daq-tools/ttnlogger\n[Wetterdienst]: https://github.com/earthobservations/wetterdienst/\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaq-tools%2Florrystream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdaq-tools%2Florrystream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaq-tools%2Florrystream/lists"}