Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/typestreamio/typestream
Open Source streaming platform. Write and run typed data pipelines with a minimal, familiar syntax.
https://github.com/typestreamio/typestream
kafka kafka-streams pipelines
Last synced: 4 months ago
JSON representation
Open Source streaming platform. Write and run typed data pipelines with a minimal, familiar syntax.
- Host: GitHub
- URL: https://github.com/typestreamio/typestream
- Owner: typestreamio
- License: apache-2.0
- Created: 2023-08-31T07:39:26.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2024-02-15T09:53:06.000Z (5 months ago)
- Last Synced: 2024-02-15T10:32:27.544Z (5 months ago)
- Topics: kafka, kafka-streams, pipelines
- Language: Kotlin
- Homepage: https://typestream.io
- Size: 7.9 MB
- Stars: 56
- Watchers: 1
- Forks: 2
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE.md
- Code of conduct: CODE_OF_CONDUCT.md
Lists
- awesome-kafka - TypeStream
README
TypeStream
Why TypeStream?
·
Getting started
·
How to contribute
·
Code of conduct
·
License
TypeStream is an abstraction layer on top of Kafka that allows you to write
and run typed data pipelines with a minimal, familiar syntax.
## Why TypeStream?
Building streaming data pipelines on top of Kafka comes with some fixed costs.
You have to write an app, test it, then deploy and manage it in production. Even
for the simplest pipelines, this can be a lot of work.
With TypeStream you can write powerful, typed data pipelines the way you'd write
a simple UNIX pipeline in your terminal. For example, imagine you'd like to
filter a "books" topic. With TypeStream, it's a one liner:
```sh
$ typestream
> grep /dev/kafka/local/topics/books the > /dev/kafka/local/topics/books_with_the
```
TypeStream will take care of type-checking your pipeline and then run it for
you. Here's how it looks like in action:
Another common use case that requires a lot of boilerplate is to enrich a topic
with data from an external source. For example, you might have a topic with an
ip address field and you may want to enrich it with country information. With
TypeStream, you can do it (again!) in a single line:
```sh
$ typestream
> cat /dev/kafka/local/topics/page_views | enrich { view -> http "https://api.country.is/#{$view.ip_address}" | cut .country } > /dev/kafka/local/topics/page_views_with_country
```
Here's how enriching looks like in action:
As you can see from the previous command, in the spirit of UNIX, we used cut to
extract the country field from the response. Here's the kick, you can use `cut`
(and many other Unix commands) on streams as well:
```sh
$ typestream
> cat /dev/kafka/local/topics/books | cut .title > /dev/kafka/local/topics/book_titles
```
Here's how cutting looks like in action:
Another problem that TypeStream solves is preventing you from writing faulty
pipelines by type-checking them before executing them. Here's type checking in
action:
Deploying a pipeline to production is as simple as running:
```sh
typestream run 'cat /dev/kafka/local/topics/books | cut .title > /dev/kafka/local/topics/book_titles'
```
See it in action here:
If you'd like to learn more about TypeStream, check out [the official
documentation](https://docs.typestream.io/).
## Getting started
If you use [Homebrew](https://brew.sh/):
```sh
brew install typestreamio/tap/typestream
$ typestream --version
```
if you see something like this:
```sh
typestream version 2023.08.31+3 42f7762daac1872416bebab7a34d0b79a838d40a (2023-09-02 09:20:52)
```
then you're good to go! You can now run:
```sh
typestream local start
typestream local seed
```
to start a local TypeStream server and seed it with some sample data. Now you're
ready to start writing your own pipelines!
Check out our [documentation](https://docs.typestream.io/) to learn more about
TypeStream.
## How to contribute
We love every form of contribution! Good entry points to the project are:
- Our [contributing guidelines](/CONTRIBUTING.md) document.
- Issues with the tag
[gardening](https://github.com/typestreamio/typestream/issues?q=is%3Aissue+is%3Aopen+label%3Agardening).
- Issues with the tag [good first
patch](https://github.com/typestreamio/typestream/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+patch%22).
If you're not sure where to start, open a [new
issue](https://github.com/typestreamio/typestream/issues/new) or hop on to our
[discord](https://discord.gg/Ha9sJWXb) server and we'll gladly help you get
started.
## Code of Conduct
You are expected to follow our [code of conduct](/CODE_OF_CONDUCT.md) when
interacting with the project via issues, pull requests, or in any other form.
Many thanks to the awesome [contributor
covenant](http://contributor-covenant.org/) initiative!
## License
[Apache 2.0](/LICENSE)