{"id":45817720,"url":"https://github.com/codernaut/kafka-to-fabric-streamer","last_synced_at":"2026-02-26T19:01:51.640Z","repository":{"id":330538543,"uuid":"1123080503","full_name":"codernaut/kafka-to-fabric-streamer","owner":"codernaut","description":"Bridge drilling telemetry from on-premises Kafka into Microsoft Fabric through Azure Event Hubs. This mirrors an oil-and-gas telemetry workflow: rigs stream rotary speed (RPM), weight on bit, and torque into local Kafka, and the app relays those events so Fabric can feed analytics, AI-driven drilling optimisation, and operational dashboards.","archived":false,"fork":false,"pushed_at":"2025-12-26T06:40:48.000Z","size":14,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-27T18:16:45.888Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codernaut.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-26T06:34:56.000Z","updated_at":"2025-12-26T06:40:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/codernaut/kafka-to-fabric-streamer","commit_stats":null,"previous_names":["codernaut/kafka-to-fabric-streamer"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/codernaut/kafka-to-fabric-streamer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codernaut%2Fkafka-to-fabric-streamer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codernaut%2Fkafka-to-fabric-streamer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codernaut%2Fkafka-to-fabric-streamer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codernaut%2Fkafka-to-fabric-streamer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codernaut","download_url":"https://codeload.github.com/codernaut/kafka-to-fabric-streamer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codernaut%2Fkafka-to-fabric-streamer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29868027,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-26T18:42:30.764Z","status":"ssl_error","status_checked_at":"2026-02-26T18:41:47.936Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-26T19:01:46.008Z","updated_at":"2026-02-26T19:01:51.630Z","avatar_url":"https://github.com/codernaut.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kafka-to-fabric-streamer\n\nBridge drilling telemetry from on-premises Kafka into Microsoft Fabric through Azure Event Hubs. This mirrors an oil-and-gas telemetry workflow: rigs stream rotary speed (RPM), weight on bit, and torque into local Kafka, and the app relays those events so Fabric can feed analytics, AI-driven drilling optimisation, and operational dashboards. The project boots a Quarkus application with two core components:\n\n- **The Rig** produces synthetic drilling parameters every 100 ms and writes them into a Kafka topic.\n- **The Bridge** consumes those messages and forwards them to Azure Event Hubs using SmallRye Reactive Messaging, applying backpressure when Event Hubs slows down.\n\n## Prerequisites\n\n- Java 17+\n- Maven 3.9+\n- Docker (for local Kafka via Redpanda)\n- Azure Event Hubs namespace and connection string for cloud forwarding\n\n## Local development\n\n1. **Start Redpanda and the app**\n\n   ```bash\n   docker compose up --build\n   ```\n\n   The Quarkus app runs on http://localhost:8080. Telemetry is published to the `drilling-parameters` topic.\n\n2. **Monitor Kafka**\n\n   Use `rpk topic consume drilling-parameters -b localhost:29092` in another shell to inspect the generated payloads.\n\n3. **Stop the stack**\n\n   ```bash\n   docker compose down -v\n   ```\n\n## Connecting to Azure Event Hubs\n\n1. Export the required configuration values before launching the app (or add them to `docker-compose.yml`).\n\n   ```bash\n   export EVENTHUBS_ENABLED=true\n   export EVENTHUBS_CONNECTION_STRING=\"Endpoint=sb://\u003cnamespace\u003e.servicebus.windows.net/;SharedAccessKeyName=\u003ckeyname\u003e;SharedAccessKey=\u003ckey\u003e;EntityPath=\u003cevent-hub-name\u003e\"\n   export EVENTHUBS_HUB_NAME=\u003cevent-hub-name\u003e\n   ```\n\n2. Start the application locally with Maven if you do not wish to use Docker.\n\n   ```bash\n   mvn quarkus:dev\n   ```\n\n3. Once the variables are set, the bridge will publish incoming messages into Event Hubs. Failures automatically trigger `nack` to apply backpressure until the send completes successfully.\n\n### Where to specify Azure configuration\n\n- **Docker Compose:** Update the `EVENTHUBS_*` entries under the `streamer` service in docker-compose.yml or provide a `.env` file that docker compose will load automatically.\n- **Local runs:** Export the same `EVENTHUBS_*` environment variables in your shell before running `mvn quarkus:dev`, or add them to application.properties for a permanent local default.\n\n## Configuration reference\n\n- `KAFKA_BOOTSTRAP_SERVERS` – Kafka bootstrap servers (default `localhost:29092`).\n- `EVENTHUBS_ENABLED` – Enable forwarding to Event Hubs (default `false`).\n- `EVENTHUBS_CONNECTION_STRING` – Event Hub connection string (required when enabled).\n- `EVENTHUBS_HUB_NAME` – Target Event Hub name (default `drilling-telemetry`).\n\n## Build\n\n```bash\nmvn clean package\n```\n\nThe resulting JVM runner `quarkus-run.jar` is packaged under `target/quarkus-app/` and is used by the Docker image.\n\n## Tests\n\n```bash\nmvn test\n```\n\nIntegration tests run with the in-memory SmallRye connector so they do not require external Kafka or Azure services.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodernaut%2Fkafka-to-fabric-streamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodernaut%2Fkafka-to-fabric-streamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodernaut%2Fkafka-to-fabric-streamer/lists"}