Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/laserdisc-io/tamer
Standalone alternatives to Kafka Connect Connectors
https://github.com/laserdisc-io/tamer
avro doobie fp fs2 functional-programming jdbc kafka s3 scala schema-registry zio zio-streams
Last synced: about 2 hours ago
JSON representation
Standalone alternatives to Kafka Connect Connectors
- Host: GitHub
- URL: https://github.com/laserdisc-io/tamer
- Owner: laserdisc-io
- License: mit
- Created: 2019-12-03T01:00:07.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2024-11-16T06:22:47.000Z (1 day ago)
- Last Synced: 2024-11-16T07:23:27.148Z (about 23 hours ago)
- Topics: avro, doobie, fp, fs2, functional-programming, jdbc, kafka, s3, scala, schema-registry, zio, zio-streams
- Language: Scala
- Homepage:
- Size: 1.35 MB
- Stars: 42
- Watchers: 9
- Forks: 15
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# Tamer
[](https://github.com/laserdisc-io/tamer/actions?query=workflow%3ACI+branch%3Amaster)
[](https://github.com/laserdisc-io/tamer/actions?query=workflow%3ARelease)
[](https://snyk.io/test/github/laserdisc-io/tamer?targetFile=build.sbt)
[](https://search.maven.org/artifact/io.laserdisc/tamer-core_2.13/)
[](https://scala-steward.org)
Tamer is a domesticated Kafka source connector.
It puts the user of this library in complete control of how data is ingested and what state is preserved (in a compacted Kafka topic).
As an example, it allows for a JDBC source to pull a window of data (e.g. 5 minutes), starting from some time in the past, as fast as possible.
At every "pull", the user can decide what to do next (e.g. should the window be increased/decreased? Should the pull slow down?).
Tamer currently supports 4 possible source types:
- any SQL data store Doobie can handle (e.g. Postgres, MySQL, etc)
- any cloud storage that is AWS S3 compatible
- OCI Object Storage
- any type of REST API, with AuthN, pagination and HATEOAS support
Some sensible defaults are provided out of the box but many customisations are possible.
## Usage
Add one of Tamer's modules as a dependency to your project:
```
// check the current version on Maven Central (or use the badge above)
libraryDependencies += "io.laserdisc" %% "tamer-db" % version
libraryDependencies += "io.laserdisc" %% "tamer-oci-objectstorage" % version
libraryDependencies += "io.laserdisc" %% "tamer-rest" % version
libraryDependencies += "io.laserdisc" %% "tamer-s3" % version
```
See [here](example/src/main/scala/tamer/db/DatabaseSimple.scala) for a sample application that makes use of Tamer's Db module for ingesting data from a JDBC datasource.
## End to end testing
### Database module
Basic manual testing is available for the code in the example module `tamer.db.DatabaseSimple` (and/or `tamer.db.DatabaseGeneralized`).
This code covers getting data from a synthetic Postgres database.
Make sure you have docker installed before proceeding.
From the `db/local` folder launch `docker compose up` (you can enter `docker compose down` if you want to start from scratch). After that you should be able to access the Kafka gui from [http://localhost:8000](http://localhost:8000).
Start the `runDatabaseSimple.sh` program which contains some example environment variables.
If Tamer works you should see messages appearing in the Kafka gui.
### S3 module
Basic manual testing is available for the code in the example module `tamer.s3.S3Simple`.
This code covers getting data from a synthetic S3 bucket.
Make sure you have docker installed before proceeding.
From the `s3/local` folder launch `docker compose up` (you can enter `docker compose down` if you want to start from scratch). After that you should be able to access the Kafka gui from [http://localhost:8000](http://localhost:8000).
Start the `runS3Simple.sh` program which contains some example environment variables.
If Tamer works you should see messages appearing in the Kafka gui.
## License
Tamer is licensed under the **[MIT License](LICENSE)** (the "License"); you may not use this software except in compliance with the License.
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and limitations under the License.