{"id":17910806,"url":"https://github.com/dgzlopes/python-liftbridge","last_synced_at":"2025-03-23T22:33:10.956Z","repository":{"id":57457195,"uuid":"203393654","full_name":"dgzlopes/python-liftbridge","owner":"dgzlopes","description":"WIP: Python client for Liftbridge.","archived":false,"fork":false,"pushed_at":"2020-07-05T12:27:06.000Z","size":67,"stargazers_count":10,"open_issues_count":3,"forks_count":5,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-10-11T22:23:37.162Z","etag":null,"topics":["client","cloud-native","grpc","liftbridge","messaging","nats","pubsub","python","python3","streaming"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/python-liftbridge/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dgzlopes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-20T14:30:43.000Z","updated_at":"2021-05-25T18:41:10.000Z","dependencies_parsed_at":"2022-09-05T18:11:54.958Z","dependency_job_id":null,"html_url":"https://github.com/dgzlopes/python-liftbridge","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dgzlopes%2Fpython-liftbridge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dgzlopes%2Fpython-liftbridge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dgzlopes%2Fpython-liftbridge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dgzlopes%2Fpython-liftbridge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dgzlopes","download_url":"https://codeload.github.com/dgzlopes/python-liftbridge/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221919602,"owners_count":16901811,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["client","cloud-native","grpc","liftbridge","messaging","nats","pubsub","python","python3","streaming"],"created_at":"2024-10-28T19:34:40.627Z","updated_at":"2024-10-28T19:34:41.298Z","avatar_url":"https://github.com/dgzlopes.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# python-liftbridge\n[![PyPI](https://img.shields.io/pypi/v/python-liftbridge.svg)](https://pypi.org/project/python-liftbridge/)\n[![GitHub](https://img.shields.io/github/license/dgzlopes/python-liftbridge)](https://github.com/dgzlopes/python-liftbridge/blob/master/LICENSE.md)\n\n**This project is under development.**\n\nPython client for [Liftbridge](https://github.com/liftbridge-io/liftbridge), a system that provides lightweight, fault-tolerant message streams for [NATS](https://nats.io).\n\nLiftbridge provides the following high-level features:\n\n- Log-based API for NATS\n- Replicated for fault-tolerance\n- Horizontally scalable\n- Wildcard subscription support\n- At-least-once delivery support and message replay\n- Message key-value support\n- Log compaction by key\n\n## Installation\n\n```\n$ pip install python-liftbridge\n```\n\n## Basic Usage\n\n```python\nfrom python_liftbridge import Lift, Message, Stream, ErrStreamExists\n\n# Create a Liftbridge client.\nclient = Lift(ip_address='localhost:9292', timeout=5)\n\n# Create a Liftbridge stream with name \"foo-stream\"\ntry:\n    # create a stream with 5 partitions. default value is 1\n    client.create_stream(Stream(subject='foo', name='foo-stream', partitions=5))\nexcept ErrStreamExists:\n    print('This stream already exists!')\n\n# Publish a message to the stream with the name \"foo-stream\".\nclient.publish(Message(value='hello', stream='foo-stream'))\n\n# Subscribe to the stream starting from the beginning.\nfor message in client.subscribe(\n    Stream(\n        subject='foo',\n        name='foo-stream',\n        subscribe_to_partition=0  # subscribe to specific partition of stream. default value is 0\n    ).start_at_earliest_received(),\n):\n    print(\"Received: '{}'\".format(message.value))\n\n```\n\n### Create Stream\n\n[Streams](https://github.com/liftbridge-io/liftbridge/blob/master/documentation/concepts.md#stream) are a durable message log attached to a NATS subject. They record messages published to the subject for consumption.\n\nStreams have a few key properties: a subject, which is the corresponding NATS subject, a name, which is a human-readable identifier for the stream, and a replication factor, which is the number of nodes the stream should be replicated to for redundancy.  Optionally, there is a group which is the name of a load-balance group for the stream to join. Also you can specify partitions count. When there are multiple streams in the same group, messages will be balanced among them.\n\n```python\n\"\"\"\n    Create a stream attached to the NATS subject \"foo.*\" that is replicated to\n    all the brokers in the cluster. ErrStreamExists is returned if a stream with\n    the given name already exists for the subject.\n\"\"\"\nclient.create_stream(Stream(subject='foo.*', name='my-stream', max_replication=True, partitions=3))\n```\n\n### Subscription Start/Replay Options\n\n[Subscriptions](https://github.com/liftbridge-io/liftbridge/blob/master/documentation/concepts.md#subscription) are how Liftbridge streams are consumed. Clients can choose where to start consuming messages from in a stream. This is controlled using options passed to Subscribe.\n\n```python\n# Subscribe starting with new messages only.\nclient.subscribe(\n    Stream(subject='foo', name='foo-stream')\n)\n# Subscribe starting with the most recently published value.\nclient.subscribe(\n    Stream(subject='foo', name='foo-stream').start_at_earliest_received()\n)\n# Subscribe starting with the oldest published value.\nclient.subscribe(\n    Stream(subject='foo', name='foo-stream').start_at_latest_received()\n)\n# Subscribe starting at a specific offset.\nclient.subscribe(\n    Stream(subject='foo', name='foo-stream').start_at_offset(4)\n)\n# Subscribe starting at a specific time.\nclient.subscribe(\n    Stream(subject='foo', name='foo-stream').start_at_time(datetime.now())\n)\n# Subscribe starting at a specific amount of time in the past.\nclient.subscribe(\n    Stream(subject='foo', name='foo-stream').start_at_time_delta(timedelta(days=1))\n)\n# Subscribe specific partition\nclient.subscribe(\n    Stream(subject='foo', name='foo-stream', subscribe_to_partition=2).start_at_latest_received()\n)\n```\n\n### Publishing\n\nA publish API is provided to make it easy to write messages to streams. This includes a number of options for decorating messages with metadata like a message key.\n\nKeys are used by Liftbridge's log compaction. When enabled, Liftbridge streams will retain only the last message for a given key.\n\n```python\n# Publish a message with a key\nclient.publish(Message(stream='foo-stream', value='Hello', key='key'))\n```\n\nAlso, it is possible to publish a message to the NATS subject (and, in turn, any streams that match the subject).\n\n```python\n# Publish a message to the NATS subject\nclient.publish_to_subject(Message(subject='foo', value='Hello foo'))\n```\n\nAnd you can publish message to specific stream partition.\n\n```python\n# Publish a message to the NATS subject\nclient.publish_to_subject(Message(subject='foo', value='Hello foo', partition=3))\n```\n#### Publishing Directly with NATS\n\nSince Liftbridge is an extension of [NATS](https://github.com/nats-io/gnatsd), a [NATS client](https://github.com/nats-io/nats.py) can also be used to publish messages. This means existing NATS publishers do not need any changes for messages to be consumed in Liftbridge.\n\n## How to contribute\n1. Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug.\n2. Fork [the repository](https://github.com/dgzlopes/python-liftbridge) on GitHub to start making your changes to the master branch (or branch off of it).\n3. Write a test which shows that the bug was fixed or that the feature works as expected.\n4. Send a [pull request](https://help.github.com/en/articles/creating-a-pull-request-from-a-fork) and bug [me](https://github.com/dgzlopes) until it gets merged and published.\n\nSome things on the backlog:\n\n- [ ] Add documentation (Sphynx)\n- [ ] Add CI (CircleCI or TravisCI)\n- [ ] Add tests\n- [ ] Add code coverage\n- [x] Add TLS support for gRPC\n- [ ] Add message headers support\n- [ ] Add message ACK support (scaffolding is already done)\n- [x] Add method to close connection\n- [ ] Add async client\n- [ ] Add gRPC connection pool\n- [x] Add logging (and remove all the random prints)\n- [ ] Add proper docstrings\n- [x] Add version file\n- [ ] Add Contributing.md and explanation of the workflow (pyenv,tox,make,pre-commit...)\n- [ ] Improve fetch metadata\n- [ ] Improve error handling\n- [x] Add to the makefile run-liftbridge using Docker [container](https://github.com/dgzlopes/liftbridge-docker)\n- [ ] Better instrumentation/observability (OpenCensus support?)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdgzlopes%2Fpython-liftbridge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdgzlopes%2Fpython-liftbridge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdgzlopes%2Fpython-liftbridge/lists"}