https://github.com/jimmystewpot/opentelemetry-datalake
A high performance OpenTelemetry receiver and pipeline that is focused on writing to open format datalakes at scale.
https://github.com/jimmystewpot/opentelemetry-datalake
apache-arrow datalake delta-lake hudi iceberg opentelemetry realtime rust
Last synced: about 1 month ago
JSON representation
A high performance OpenTelemetry receiver and pipeline that is focused on writing to open format datalakes at scale.
- Host: GitHub
- URL: https://github.com/jimmystewpot/opentelemetry-datalake
- Owner: jimmystewpot
- License: mpl-2.0
- Created: 2026-06-09T00:04:59.000Z (2 months ago)
- Default Branch: main
- Last Pushed: 2026-06-24T21:12:26.000Z (about 1 month ago)
- Last Synced: 2026-06-24T21:15:13.898Z (about 1 month ago)
- Topics: apache-arrow, datalake, delta-lake, hudi, iceberg, opentelemetry, realtime, rust
- Language: Rust
- Homepage:
- Size: 406 KB
- Stars: 2
- Watchers: 0
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Security: SECURITY.md
- Agents: AGENTS.md
Awesome Lists containing this project
README
# OpenTelemetry Datalake
[](https://codecov.io/github/jimmystewpot/opentelemetry-datalake)
[](https://opensource.org/licenses/MPL-2.0)
`opentelemetry-datalake` is an ultra-high-performance, horizontally scalable OpenTelemetry (OTLP) receiver pipeline written in Rust. It ingests OTLP metrics, traces, and logs, decodes them into memory-efficient **Apache Arrow** formats, and channels them through to a downstream data lake sink (such as Kafka or ACID table formats like Delta Lake, Iceberg, and Hudi).
Designed with zero-cost abstractions, lock-free concurrency, and zero-panic error handling, this pipeline is engineered to ingest telemetry at maximum throughput.
---
## Architecture Overview
```text
+--------------------------------------+
| OTLP Telemetry Ingress |
| (gRPC: 4317 / HTTP: 4318) |
+------------------+-------------------+
|
| (Protobuf / JSON Payload)
v
+------------------+-------------------+
| Arrow Codec |
| (Logs, Traces & Metrics Decoding) |
+------------------+-------------------+
|
| (Vectorized Arrow RecordBatches)
v
+------------------+-------------------+
| Signal Router |
| (Dedicated MPSC Channels) |
+--------+---------+---------+---------+
| | |
(Logs Channel) | | | (Metrics Channel)
| | (Traces Channel)
v v v
+----+ +----+ +----+
|Log | |Span| |Met | (No-Op Transformers)
+----+ +----+ +----+
| | |
v v v
+----+ +----+ +----+
|Sink| |Sink| |Sink| (Kafka Producer Sinks)
+----+ +----+ +----+
| | |
+---------+---------+
|
v
+------------------+-------------------+
| Apache Kafka / Data Lake |
+--------------------------------------+
```
---
## Workspace Layout
The project is structured as a Cargo virtual workspace consisting of the following crates:
* **`src/main.rs`**: The main entry point. Bootstraps config parsing, configures pipeline instrumentation, schedules the DAG execution, and handles graceful shutdown.
* **`crates/core`**: Core pipeline traits (`Source`, `Transform`, `Sink`), channel-based multiplexing routing (`Fanout`), and pipeline-wide observability/telemetry instrumentation.
* **`crates/arrow-codec`**: Deserialization modules mapping OTLP Protobuf and JSON metrics, traces, and logs payloads directly to columnar Apache Arrow `RecordBatch`es.
* **`crates/otlp-receiver`**: Ingest layer implementing a multi-protocol OTLP receiver with Tonic (gRPC) and Axum (HTTP/JSON).
* **`crates/noop-transformer`**: Implementation of `Transform` that passes signal record batches directly through to the next phase of the pipeline.
* **`crates/kafka-sink`**: High-performance sink implementing the `Sink` trait using `rdkafka` to stream Arrow IPC or JSON payloads to Kafka brokers.
---
## Configuration
Configuration is managed using `figment` and supports merging of file-based TOML configs and environment variable overrides.
### Example configuration (`config.toml`):
```toml
[server]
grpc_addr = "127.0.0.1:4317"
http_addr = "127.0.0.1:4318"
[kafka]
bootstrap_servers = "localhost:9092"
logs_topic = "otlp-logs"
traces_topic = "otlp-traces"
metrics_topic = "otlp-metrics"
serialization_format = "Ipc" # Options: "Ipc", "Json"
[kafka.options]
"queue.buffering.max.messages" = "100000"
"compression.codec" = "snappy"
[telemetry]
endpoint = "http://127.0.0.1:4317"
service_name = "opentelemetry-datalake"
cloud_region = "us-east-1"
```
### Environment Overrides:
Any configuration value can be overridden using the `OTEL_DATALAKE_` environment variable prefix. For example:
* `OTEL_DATALAKE_KAFKA__BOOTSTRAP_SERVERS="kafka-broker:9092"`
* `OTEL_DATALAKE_TELEMETRY__CLOUD_REGION="us-west-2"`
---
## Development & Operations
Developer tasks are automated via the root `Makefile`.
### Quality & Testing Gates
Run all quality checks (formatting, pedantic clippy linting, testing, and benchmarking):
```bash
make all
```
Individually execute development tasks:
* **Format code**:
```bash
cargo fmt
```
* **Run lints (strict pedantic rules)**:
```bash
make clippy
```
* **Run test suite**:
```bash
make test
```
* **Run micro-benchmarks**:
```bash
make bench
```
### Running the Application
Build the production release binary:
```bash
cargo build --release
```
Run the pipeline:
```bash
cargo run --bin opentelemetry-datalake -- --config config.toml
```