https://github.com/validus-risk-management/amazon-kinesis-client-rust
Amazon Kinesis Client Library for Rust
https://github.com/validus-risk-management/amazon-kinesis-client-rust
aws aws-kinesis-stream kcl kinesis rust
Last synced: 11 months ago
JSON representation
Amazon Kinesis Client Library for Rust
- Host: GitHub
- URL: https://github.com/validus-risk-management/amazon-kinesis-client-rust
- Owner: Validus-Risk-Management
- License: mit
- Created: 2022-11-04T13:12:57.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-12-06T09:17:57.000Z (over 3 years ago)
- Last Synced: 2025-05-13T16:37:40.134Z (about 1 year ago)
- Topics: aws, aws-kinesis-stream, kcl, kinesis, rust
- Language: Rust
- Homepage: https://crates.io/crates/kcl
- Size: 32.2 KB
- Stars: 6
- Watchers: 4
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Amazon Kinesis Client Library for Rust
[![crates-badge]](https://crates.io/crates/kcl)
[![docs-badge]](https://docs.rs/kcl)
[](LICENSE)
This package provides a Rust interface to the Amazon Kinesis Client Library (KCL) MultiLangDaemon,
which is part of the [Amazon KCL for Java][kinesis-github].
This interface manages the interaction with the MultiLangDaemon so that developers can focus on
implementing their record processor executable.
There is a provided Docker image that sets up the correct JARs using the [Amazon KCL for Python][kinesis-python].
A settings file is also required for the MultiLangDaemon to correctly set up your processor.
A sample of this can be found in the [examples][example-properties].
## Basic Usage
A more complete example can be found in the [example][example-consumer]
```rust no_run
use kcl::checkpointer::Checkpointer;
use kcl::reader::StdinReader;
use kcl::writer::StdoutWriter;
use kcl::{run, Processor, Record};
use serde::Deserialize;
#[derive(Deserialize)]
struct DummyPayload;
struct BaseApp;
impl Processor for BaseApp {
fn initialize(&mut self, _shard_id: &str) {}
fn process_records(
&mut self,
data: &[Record],
_checkpointer: &mut Checkpointer,
) {
for record in data {
match record.json::() {
Ok(data) => {}
Err(e) => {}
}
}
}
fn lease_lost(&mut self) {}
fn shard_ended(&mut self, _checkpointer: &mut Checkpointer) {}
fn shutdown_requested(&mut self, _checkpointer: &mut Checkpointer) {}
}
fn main() {
run(&mut BaseApp {});
}
```
## Docker
An example consumer of this Docker Image would be:
**Compile with al2 because amazoncoretto uses al2**
```dockerfile
FROM amazonlinux:2 as builder
RUN yum update -y && yum install -y gcc
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
COPY . .
RUN cargo build --release
FROM ghcr.io/validus-risk-management/amazon-kinesis-client-rust:latest as runner
COPY my-configs/app.properties app.properties
COPY --from=builder target/release/my-app target/release/my-app
```
The default entrypoint should meet most requirements:
```dockerfile
CMD ["java", "-cp", "/usr/local/lib/jars/*", "software.amazon.kinesis.multilang.MultiLangDaemon", "--properties-file", "app.properties"]
```
Additional configuration can be found [here][kcl-cli-params].
[amazon-kcl]: http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-app.html
[kinesis-github]: https://github.com/awslabs/amazon-kinesis-client
[kinesis-python]: https://github.com/awslabs/amazon-kinesis-client-python
[kcl-cli-params]: https://github.com/awslabs/amazon-kinesis-client-python/blob/v2.0.6/samples/amazon_kclpy_helper.py
[example-properties]: https://github.com/Validus-Risk-Management/amazon-kinesis-client-rust/blob/main/examples/sample.properties
[example-consumer]: https://github.com/Validus-Risk-Management/amazon-kinesis-client-rust/blob/main/examples/example_consumer/main.rs
[crates-badge]: https://img.shields.io/crates/v/kcl.svg
[docs-badge]: https://docs.rs/kcl/badge.svg