Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/rupurt/branch
A streaming toolkit with bottomless storage
https://github.com/rupurt/branch
azure-blob-storage gcs kafka kappa kappa-architecture minio object-storage python r2 s3 streaming
Last synced: 21 days ago
JSON representation
A streaming toolkit with bottomless storage
- Host: GitHub
- URL: https://github.com/rupurt/branch
- Owner: rupurt
- License: mit
- Created: 2024-01-06T09:56:30.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2024-01-18T00:13:16.000Z (10 months ago)
- Last Synced: 2024-10-10T14:39:11.695Z (about 1 month ago)
- Topics: azure-blob-storage, gcs, kafka, kappa, kappa-architecture, minio, object-storage, python, r2, s3, streaming
- Language: Python
- Homepage:
- Size: 29.3 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Branch
A streaming toolkit with bottomless storage
- [x] Embedded
- [ ] Single node
- [ ] Clustered
- [x] HTTP API
- [ ] Web UI
- [ ] Kafka gateway
- [ ] S3
- [x] GCS
- [ ] Azure blob storage
- [ ] MinIO
- [ ] R2
- [x] Local
- [x] Python bindings
- [ ] Elixir bindings
- [ ] Zig bindings
- [ ] C bindings
## Usage
### CLI
```shell
> branch --help
Usage: branch [OPTIONS] COMMAND [ARGS]...
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --install-completion Install completion for the current shell. │
│ --show-completion Show completion for the current shell, to copy it or customize the installation. │
│ --help Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ init │
│ produce │
│ consume │
│ topics │
│ server │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```
Create your configuration file by following the prompts
```shell
> branch init
...
```
Create a topic
```shell
> branch topics create \
--topic=my-topic.v1 \
--partitions=3
```
Produce a new record every second
```shell
> watch -n 1 \
'branch produce --topic=my-topic.v1 --partition=0 --message=\'{"hello": "cli!", "now": "$(date --iso-8601=ns)"}\''
...
```
Consume records produced to `my-topic.v1`
```shell
> branch consume --topic=my-topic.v1
...
```
### Python
```python
# my-branch.py
import branch
from datetime import datetime
msg = '{"hello": "python!", "now": datetime.now()}'
producer_record = branch.ProducerRecord(topic="my-topic.v1", value=msg)
branch.produce(producer_record)
branch.consume("my-topic.v1")
```
```shell
> python my-branch.py
...
```
## Single node
```shell
> branch server start --port=9000 --ui-port=8999
...
```
## Clustered
```shell
> branch server start --host=0.0.0.0 --port=9001 --name=my-branch-1
...
> branch server start --host=0.0.0.0 --port=9002 --name=my-branch-2
...
```
## HTTP
```shell
> curl -X POST -d '{...}' http://localhost:9000/api/v1/produce
...
```
```shell
> curl http://localhost:9000/api/v1/consume
...
```
## Web UI
```shell
> open http://localhost:8999
```
## Development
This repository manages the dev environment as a [Nix flake](https://nixos.wiki/wiki/Flakes)
and requires [Nix to be installed](https://github.com/DeterminateSystems/nix-installer)
```shell
nix develop -c $SHELL
```
```shell
make all
```
```shell
make deps.install
make deps.install/dev
```
```shell
make deps.freeze
make deps.freeze/dev
```
```shell
make deps/reqs
make deps/outdated
make deps/check
```
```shell
make run
make run/server
```
```shell
make format
make format/check
```
```shell
make typecheck
```
```shell
make lint
```
```shell
make test
make test/watch
```
```shell
make coverage
```
```shell
make build
make build/pyinstaller
make build/nuitka
```
```shell
make exec
```
```shell
make clean
```
```shell
make compose.build
make compose.up
```
```shell
make docker.build/deps
make docker.build/run
```
## License
`branch` is released under the [MIT license](./LICENSE)