{"id":30692160,"url":"https://github.com/ncolesummers/subscribe-and-store","last_synced_at":"2026-04-18T01:01:57.475Z","repository":{"id":310400841,"uuid":"759594307","full_name":"ncolesummers/subscribe-and-store","owner":"ncolesummers","description":"High-performance MQTT to TimescaleDB service for robotic arm telemetry. Real-time data ingestion with automatic reconnection, processing 4x4 transformation matrices from multiple joints and fingers.","archived":false,"fork":false,"pushed_at":"2025-08-17T20:36:16.000Z","size":84,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-02T04:48:48.657Z","etag":null,"topics":["arm-robotics","async","data-ingestion","iot","mqtt","postgresql","real-time","robotics","robotics-telemetry","rust","sensor-data","telemetry","time-series","timescaledb","tokio"],"latest_commit_sha":null,"homepage":"https://ncolesummers.com","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ncolesummers.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"ncolesummers"}},"created_at":"2024-02-19T00:25:10.000Z","updated_at":"2025-08-17T20:36:20.000Z","dependencies_parsed_at":"2025-08-17T22:17:17.412Z","dependency_job_id":"6e1f345c-7067-4462-a69d-d1b9962006da","html_url":"https://github.com/ncolesummers/subscribe-and-store","commit_stats":null,"previous_names":["ncolesummers/subscribe-and-store"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ncolesummers/subscribe-and-store","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ncolesummers%2Fsubscribe-and-store","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ncolesummers%2Fsubscribe-and-store/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ncolesummers%2Fsubscribe-and-store/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ncolesummers%2Fsubscribe-and-store/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ncolesummers","download_url":"https://codeload.github.com/ncolesummers/subscribe-and-store/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ncolesummers%2Fsubscribe-and-store/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31952206,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"ssl_error","status_checked_at":"2026-04-18T00:39:20.671Z","response_time":62,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arm-robotics","async","data-ingestion","iot","mqtt","postgresql","real-time","robotics","robotics-telemetry","rust","sensor-data","telemetry","time-series","timescaledb","tokio"],"created_at":"2025-09-02T04:48:55.253Z","updated_at":"2026-04-18T01:01:57.437Z","avatar_url":"https://github.com/ncolesummers.png","language":"Rust","funding_links":["https://github.com/sponsors/ncolesummers"],"categories":[],"sub_categories":[],"readme":"# Subscribe and Store - MQTT to TimescaleDB Service\n\nA high-performance Rust service that subscribes to MQTT topics and stores robotic arm telemetry data in a TimescaleDB time-series database. Built for real-time data ingestion with automatic reconnection and error handling.\n\n## Overview\n\nThis service is designed to handle high-frequency telemetry data from robotic arms, subscribing to MQTT topics (`left_arm` and `right_arm`) and persisting the data to TimescaleDB for time-series analysis. The system processes transformation matrices and joint data from multiple robotic components.\n\n## Features\n\n- **Real-time MQTT Subscription**: Subscribes to multiple topics with configurable QoS levels\n- **TimescaleDB Integration**: Efficient time-series data storage with hypertables\n- **Automatic Reconnection**: Robust error handling with automatic MQTT reconnection\n- **High Performance**: Built with Rust and Tokio for asynchronous I/O\n- **JSON Data Processing**: Handles complex robotic arm data structures with multiple joint and finger matrices\n- **Environment Configuration**: Flexible configuration through environment variables\n\n## Architecture\n\n```\n┌─────────────┐      MQTT       ┌──────────────┐      SQL        ┌──────────────┐\n│ MQTT Broker │ ───────────────\u003e │ Rust Service │ ───────────────\u003e │ TimescaleDB  │\n│             │   (left_arm,     │              │   (JSONB data)   │              │\n│             │    right_arm)    │              │                  │              │\n└─────────────┘                  └──────────────┘                  └──────────────┘\n```\n\n## Data Model\n\n### ArmMessage Structure\n\n```rust\n{\n  timestamp: i64,              // Unix timestamp in milliseconds\n  matrices: {\n    J1-J9: Vec\u003cVec\u003cf64\u003e\u003e,     // Joint transformation matrices (9 joints)\n    F1-F6: Vec\u003cVec\u003cf64\u003e\u003e      // Finger transformation matrices (6 fingers)\n  }\n}\n```\n\n### Database Schema\n\n- **Tables**: `left_arm`, `right_arm`\n- **Columns**:\n  - `timestamp` (TIMESTAMPTZ): Time of data capture\n  - `data` (JSONB): Complete arm telemetry data\n- **Optimization**: TimescaleDB hypertables for efficient time-series queries\n\n## Prerequisites\n\n- Rust 1.70+ (2021 edition)\n- Docker and Docker Compose\n- PostgreSQL client libraries\n\n## Installation\n\n### 1. Clone the Repository\n\n```bash\ngit clone \u003crepository-url\u003e\ncd subscribe-and-store\n```\n\n### 2. Set Up TimescaleDB\n\n```bash\ncd db\ndocker-compose up -d\n```\n\nThis will:\n\n- Start a TimescaleDB instance on port 5432\n- Create the database `sabino-ts`\n- Initialize hypertables for `left_arm` and `right_arm`\n\n### 3. Configure Environment\n\nCreate a `.env` file in the project root:\n\n```env\n# Database Configuration\nDATABASE_URL=postgres://postgres:sabino@localhost:5432/sabino-ts\n\n# MQTT Configuration (optional, defaults shown)\nRUST_SUB_MQTT_URI=tcp://localhost:1883\nRUST_SUB_MQTT_CLIENT_ID=rust_sub_mqtt_tsdb_client\n```\n\n### 4. Build and Run\n\n```bash\n# Development\ncargo run\n\n# Production (optimized)\ncargo build --release\n./target/release/subscribe-and-store\n```\n\n## Configuration\n\n| Environment Variable      | Description                  | Default                     |\n| ------------------------- | ---------------------------- | --------------------------- |\n| `DATABASE_URL`            | PostgreSQL connection string | Required                    |\n| `RUST_SUB_MQTT_URI`       | MQTT broker URI              | `tcp://localhost:1883`      |\n| `RUST_SUB_MQTT_CLIENT_ID` | MQTT client identifier       | `rust_sub_mqtt_tsdb_client` |\n\n## Dependencies\n\n### Core Dependencies\n\n- **paho-mqtt** (0.12): MQTT client with bundled C library\n- **tokio** (1.x): Async runtime with full features\n- **sqlx** (0.7.3): Async PostgreSQL driver with compile-time query verification\n- **serde** (1.0): JSON serialization/deserialization\n- **anyhow** (1.0): Error handling with context\n\n### Database\n\n- **TimescaleDB**: PostgreSQL extension for time-series data\n- **PostgreSQL 16**: Latest stable release\n\n## Development\n\n### Project Structure\n\n```\nsubscribe-and-store/\n├── src/\n│   ├── main.rs      # Application entry point and message loop\n│   ├── mqtt.rs      # MQTT connection and subscription logic\n│   ├── sql.rs       # Database operations and connection pooling\n│   └── model.rs     # Data structures for arm messages\n├── db/\n│   ├── docker-compose.yml  # TimescaleDB container configuration\n│   └── init.sql            # Database initialization script\n├── Cargo.toml              # Rust dependencies\n└── README.md\n```\n\n### Testing\n\n```bash\n# Run tests\ncargo test\n\n# Run with logging\nRUST_LOG=debug cargo run\n```\n\n### Performance Considerations\n\n- **QoS 0**: Used for maximum throughput (at-most-once delivery)\n- **Connection Pooling**: 5 database connections for concurrent writes\n- **Stream Buffer**: 25 messages buffered for processing\n- **Keep-Alive**: 20-second interval for connection monitoring\n\n## Monitoring\n\nThe service provides console output for:\n\n- Connection status (MQTT and database)\n- Subscription confirmations\n- Reconnection attempts\n- Error messages with context\n\n## Error Handling\n\n- **MQTT Disconnection**: Automatic reconnection with exponential backoff\n- **Database Errors**: Logged with context, continues processing other messages\n- **Parse Errors**: Detailed error messages with problematic data\n- **Subscription Failures**: Retry up to 5 times with 5-second intervals\n\n## Docker Deployment\n\n### Building the Service Container\n\n```dockerfile\nFROM rust:1.70 as builder\nWORKDIR /app\nCOPY . .\nRUN cargo build --release\n\nFROM debian:bookworm-slim\nRUN apt-get update \u0026\u0026 apt-get install -y libssl-dev ca-certificates\nCOPY --from=builder /app/target/release/subscribe-and-store /usr/local/bin/\nCMD [\"subscribe-and-store\"]\n```\n\n### Docker Compose Full Stack\n\n```yaml\nversion: '3.8'\nservices:\n  app:\n    build: .\n    environment:\n      - DATABASE_URL=postgres://postgres:sabino@timescaledb:5432/sabino-ts\n      - RUST_SUB_MQTT_URI=tcp://mqtt:1883\n    depends_on:\n      - timescaledb\n    restart: unless-stopped\n\n  timescaledb:\n    image: timescale/timescaledb:latest-pg16\n    # ... (existing configuration)\n```\n\n## Troubleshooting\n\n### Common Issues\n\n1. **Connection Refused to Database**\n\n   - Verify TimescaleDB is running: `docker ps`\n   - Check DATABASE_URL in `.env`\n   - Ensure PostgreSQL port 5432 is not blocked\n\n2. **MQTT Connection Failed**\n\n   - Verify MQTT broker is running\n   - Check RUST_SUB_MQTT_URI configuration\n   - Ensure network connectivity to broker\n\n3. **Parse Errors**\n   - Verify incoming JSON structure matches ArmMessage format\n   - Check timestamp is in milliseconds\n   - Ensure all joint and finger matrices are present\n\n## License\n\n[MIT License](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fncolesummers%2Fsubscribe-and-store","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fncolesummers%2Fsubscribe-and-store","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fncolesummers%2Fsubscribe-and-store/lists"}