{"id":46696928,"url":"https://github.com/erans/lclq","last_synced_at":"2026-03-09T05:32:36.683Z","repository":{"id":320988467,"uuid":"1084006846","full_name":"erans/lclq","owner":"erans","description":"Drop-in local replacement for AWS SQS and GCP Pub/Sub Perfect for lightning-fast local development and CI/CD testing without cloud dependencies.","archived":false,"fork":false,"pushed_at":"2025-12-04T01:00:11.000Z","size":598,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-06T03:27:27.093Z","etag":null,"topics":["cicd","pubsub","rust","sqs"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/erans.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-27T04:48:56.000Z","updated_at":"2025-12-04T00:59:44.000Z","dependencies_parsed_at":"2025-10-27T06:26:07.482Z","dependency_job_id":null,"html_url":"https://github.com/erans/lclq","commit_stats":null,"previous_names":["erans/lclq"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/erans/lclq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erans%2Flclq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erans%2Flclq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erans%2Flclq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erans%2Flclq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/erans","download_url":"https://codeload.github.com/erans/lclq/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erans%2Flclq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30283917,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-09T02:57:19.223Z","status":"ssl_error","status_checked_at":"2026-03-09T02:56:26.373Z","response_time":61,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cicd","pubsub","rust","sqs"],"created_at":"2026-03-09T05:32:36.033Z","updated_at":"2026-03-09T05:32:36.669Z","avatar_url":"https://github.com/erans.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lclq - Local Cloud Queue\n\n**Lightning-fast local replacement for AWS SQS and GCP Pub/Sub.**\nZero configuration. Zero cloud dependencies. 100% SDK compatible.\n\n![Version](https://img.shields.io/badge/version-0.2.0-blue)\n![License](https://img.shields.io/badge/license-MIT%2FApache--2.0-green)\n![Tests](https://img.shields.io/badge/tests-408%20passing-success)\n\n---\n\n## Why lclq?\n\nTransform your development workflow with **blazing-fast local queues** that work seamlessly with your existing AWS and GCP code:\n\n### 🚀 **182x Faster Than Cloud**\n- **1.8M messages/sec** vs 10K on AWS SQS\n- **Sub-10µs latency** vs 10ms+ on cloud\n- Iterate faster. Test faster. Ship faster.\n\n### ⚡ **Zero Configuration**\n```bash\nlclq start  # That's it. Server running in \u003c100ms\n```\nNo accounts. No credentials. No setup. Just run.\n\n### 🎯 **100% SDK Compatible**\nWorks with **official AWS and GCP SDKs** out of the box. Change one line (the endpoint), keep everything else:\n\n```python\n# Production\nsqs = boto3.client('sqs')\n\n# Local dev - just add endpoint\nsqs = boto3.client('sqs', endpoint_url='http://localhost:9324')\n# Rest of your code stays EXACTLY the same\n```\n\n### 💰 **Zero Cloud Costs**\n- **Free CI/CD testing** - No AWS/GCP bills for integration tests\n- **Unlimited local development** - Test without rate limits or quotas\n- **Perfect for learning** - Explore SQS/Pub/Sub without cloud accounts\n\n---\n\n## Perfect For\n\n| Use Case | Why lclq? |\n|----------|-----------|\n| 🔧 **Local Development** | Test queue-based apps without 100ms+ cloud latency |\n| ✅ **CI/CD Pipelines** | Fast, isolated tests in GitHub Actions, GitLab CI, Jenkins |\n| 🧪 **Integration Testing** | Spin up in seconds, tear down instantly |\n| 📚 **Learning \u0026 Prototyping** | Experiment with SQS/Pub/Sub without AWS/GCP accounts |\n\n---\n\n## Quick Start\n\n### Installation\n\n**Option 1: Binary (Recommended)**\n```bash\n# Linux\ncurl -sSL https://github.com/erans/lclq/releases/latest/download/lclq-linux-x86_64 -o lclq\nchmod +x lclq\n./lclq start\n\n# macOS (Apple Silicon)\ncurl -sSL https://github.com/erans/lclq/releases/latest/download/lclq-macos-aarch64 -o lclq\nchmod +x lclq\n./lclq start\n\n# Windows\n# Download lclq-windows-x86_64.exe from releases page\n```\n\n**Option 2: Docker**\n```bash\ndocker run -p 9324:9324 -p 8085:8085 -p 9000:9000 erans/lclq:latest\n```\n\n**Option 3: From Source**\n```bash\ncargo install lclq\nlclq start\n```\n\n### Server Endpoints\n\nOnce started, lclq exposes multiple endpoints:\n\n| Service | Endpoint | Protocol |\n|---------|----------|----------|\n| AWS SQS | `http://localhost:9324` | HTTP/REST |\n| GCP Pub/Sub | `localhost:8085` | gRPC |\n| GCP Pub/Sub | `http://localhost:8086` | HTTP/REST |\n| Admin API | `http://localhost:9000` | HTTP/REST |\n| Metrics | `http://localhost:9090/metrics` | Prometheus |\n\n---\n\n## Usage Examples\n\n### AWS SQS (Python + boto3)\n\n```python\nimport boto3\n\n# Only change: point to localhost\nsqs = boto3.client(\n    'sqs',\n    endpoint_url='http://localhost:9324',\n    region_name='us-east-1',\n    aws_access_key_id='dummy',\n    aws_secret_access_key='dummy'\n)\n\n# Your existing code works unchanged\nqueue = sqs.create_queue(QueueName='my-queue')\nsqs.send_message(QueueUrl=queue['QueueUrl'], MessageBody='Hello World!')\nresponse = sqs.receive_message(QueueUrl=queue['QueueUrl'])\nprint(response['Messages'][0]['Body'])  # \"Hello World!\"\n```\n\n### AWS SQS (JavaScript + AWS SDK v3)\n\n```javascript\nimport { SQSClient, CreateQueueCommand, SendMessageCommand } from \"@aws-sdk/client-sqs\";\n\n// Only change: add endpoint\nconst client = new SQSClient({\n  region: \"us-east-1\",\n  endpoint: \"http://localhost:9324\",\n  credentials: { accessKeyId: \"dummy\", secretAccessKey: \"dummy\" }\n});\n\n// Your existing code works unchanged\nconst { QueueUrl } = await client.send(new CreateQueueCommand({ QueueName: \"my-queue\" }));\nawait client.send(new SendMessageCommand({ QueueUrl, MessageBody: \"Hello World!\" }));\n```\n\n### GCP Pub/Sub (Python)\n\n```python\nimport os\nos.environ['PUBSUB_EMULATOR_HOST'] = 'localhost:8085'  # Only change needed\n\nfrom google.cloud import pubsub_v1\n\npublisher = pubsub_v1.PublisherClient()\nsubscriber = pubsub_v1.SubscriberClient()\n\n# Your existing code works unchanged\ntopic_path = publisher.topic_path('my-project', 'my-topic')\npublisher.create_topic(request={\"name\": topic_path})\nfuture = publisher.publish(topic_path, b'Hello World!')\nprint(f\"Published message ID: {future.result()}\")\n```\n\n---\n\n## CI/CD Integration\n\n### GitHub Actions\n\n```yaml\nname: Integration Tests\non: [push, pull_request]\n\njobs:\n  test:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n\n      - name: Start lclq\n        run: |\n          curl -sSL https://github.com/erans/lclq/releases/latest/download/lclq-linux-x86_64 -o lclq\n          chmod +x lclq\n          ./lclq start \u0026\n          sleep 2\n\n      - name: Run tests\n        run: pytest tests/integration/\n```\n\n### Docker Compose\n\n```yaml\nversion: '3.8'\nservices:\n  lclq:\n    image: erans/lclq:latest\n    ports:\n      - \"9324:9324\"  # SQS\n      - \"8085:8085\"  # Pub/Sub\n      - \"9000:9000\"  # Admin\n\n  app:\n    build: .\n    environment:\n      AWS_ENDPOINT_URL: http://lclq:9324\n      PUBSUB_EMULATOR_HOST: lclq:8085\n    depends_on:\n      - lclq\n```\n\n---\n\n## Features\n\n### ✅ AWS SQS Compatibility\n\nFully compatible with official AWS SDKs (Python, JavaScript, Go, Rust, Java, Ruby).\n\n**Core Features:**\n- Standard and FIFO queues\n- Message attributes\n- Dead Letter Queues (DLQ)\n- Visibility timeout management\n- Long polling (up to 20 seconds)\n- Batch operations (10 messages)\n- Delay queues (0-900 seconds)\n- Message retention (60s - 14 days)\n- Content-based deduplication\n- Redrive policies\n\n**Tested:** 7/7 tests passing with Python, JavaScript, Go, and Rust SDKs.\n\n### ✅ GCP Pub/Sub Compatibility\n\nFully compatible with official Google Cloud SDKs (gRPC + HTTP/REST).\n\n**Core Features:**\n- Topics and subscriptions\n- Message publishing with attributes\n- Message ordering with ordering keys\n- Pull and StreamingPull subscriptions\n- Acknowledgment deadline modification\n- Dead letter topics\n- Message retention\n- Subscription filtering\n\n**Tested:** 53/53 tests passing with Python, JavaScript, and Go SDKs.\n\n### 🔔 GCP Pub/Sub Push Subscriptions\n\n**Automatic HTTP delivery** - Messages are pushed to your webhook endpoints automatically. No polling required!\n\n**Key Features:**\n- **HTTP/HTTPS webhook delivery** - Receive messages via HTTP POST to your endpoints\n- **Exponential backoff retry** - Configurable retry policy (default: 10s-600s, 5 attempts)\n- **Dead letter topics** - Failed deliveries automatically sent to DLT after max retries\n- **GCP-compatible format** - JSON payload matches Google Cloud Pub/Sub push format\n- **Lightweight workers** - Default 2 workers (configurable via `LCLQ_PUSH_WORKERS`)\n\n**Quick Example:**\n```python\nfrom google.cloud import pubsub_v1\n\nsubscriber = pubsub_v1.SubscriberClient()\n\n# Create push subscription pointing to your webhook\nsubscription = subscriber.create_subscription(\n    request={\n        \"name\": \"projects/my-project/subscriptions/my-push-sub\",\n        \"topic\": \"projects/my-project/topics/my-topic\",\n        \"push_config\": {\n            \"push_endpoint\": \"http://your-app.local:8080/webhook\"\n        },\n        \"retry_policy\": {\n            \"minimum_backoff\": {\"seconds\": 10},\n            \"maximum_backoff\": {\"seconds\": 600}\n        }\n    }\n)\n\n# Messages automatically delivered to your webhook!\n# No pull() calls needed\n```\n\nYour webhook receives:\n```json\n{\n  \"message\": {\n    \"data\": \"SGVsbG8sIFdvcmxkIQ==\",\n    \"attributes\": {\"key\": \"value\"},\n    \"messageId\": \"123\",\n    \"publishTime\": \"2025-12-03T10:00:00Z\"\n  },\n  \"subscription\": \"projects/my-project/subscriptions/my-push-sub\"\n}\n```\n\nReturn HTTP 2xx to acknowledge, 4xx/5xx or timeout triggers retry with exponential backoff.\n\n**Environment Variables:**\n```bash\nLCLQ_PUSH_WORKERS=4  # Number of push delivery workers (default: 2)\n```\n\nSee [Push Subscriptions Documentation](docs/push-subscriptions.md) for complete guide.\n\n### 📦 Storage Options\n\n**In-Memory (Default)**\n- 1.8M+ messages/sec throughput\n- Zero dependencies\n- Perfect for testing\n- Configurable eviction policies (LRU, FIFO, RejectNew)\n\n**SQLite (Persistent)**\n- Full ACID guarantees\n- Single-file database\n- Automatic maintenance\n- Great for local development with data persistence\n\n---\n\n## Performance\n\nReal benchmark results (in-memory backend):\n\n| Operation | Throughput | P50 Latency | P99 Latency |\n|-----------|------------|-------------|-------------|\n| Send (single) | 1.82M/sec | 7.4µs | 34µs |\n| Send (batch 10) | 4.67M/sec | 21µs | 85µs |\n| Receive (single) | 1.66M/sec | 8.0µs | 31µs |\n| Send+Receive cycle | 894K/sec | 14µs | 57µs |\n\n**Result:** 182x faster than AWS SQS, 286x better latency.\n\nSee [docs/benchmarks.md](docs/benchmarks.md) for detailed analysis.\n\n---\n\n## Configuration\n\nlclq works with **zero configuration**, but supports customization:\n\n### Command Line\n```bash\nlclq start --sqs-port 9324 --bind-address 0.0.0.0\n```\n\n### Environment Variables\n```bash\n# Server ports\nLCLQ_SQS_PORT=9324\nLCLQ_PUBSUB_GRPC_PORT=8085\nLCLQ_PUBSUB_REST_PORT=8086\nLCLQ_BIND_ADDRESS=0.0.0.0\n\n# Push subscriptions\nLCLQ_PUSH_WORKERS=2  # Number of push delivery workers (default: 2)\n\n# Receipt handle security (required for production)\nLCLQ_RECEIPT_SECRET=your-secret-key-here\n\n# Start server\nlclq start\n```\n\n### Config File (`lclq.toml`)\n```toml\n[server]\nsqs_port = 9324\npubsub_grpc_port = 8085\nbind_address = \"127.0.0.1\"\n\n[storage.backend]\ntype = \"InMemory\"        # or \"Sqlite\"\nmax_messages = 100000\neviction_policy = \"Lru\"  # or \"Fifo\" or \"RejectNew\"\n\n[logging]\nlevel = \"info\"\nformat = \"text\"\n```\n\nStart with config: `lclq start --config lclq.toml`\n\n---\n\n## CLI Commands\n\n```bash\n# Server management\nlclq start                         # Start all services\nlclq health                        # Health check\n\n# Queue operations\nlclq queue list                    # List all queues\nlclq queue create my-queue         # Create a queue\nlclq queue delete my-queue         # Delete a queue\nlclq queue stats my-queue          # Queue statistics\nlclq queue purge my-queue          # Clear all messages\n```\n\n---\n\n## Monitoring\n\n### Prometheus Metrics (`http://localhost:9090/metrics`)\n\nTrack key metrics:\n- `lclq_messages_sent_total` - Messages sent by queue\n- `lclq_messages_received_total` - Messages received\n- `lclq_queue_depth` - Current queue depth\n- `lclq_send_latency_seconds` - Send latency histogram\n- `lclq_receive_latency_seconds` - Receive latency histogram\n\n### Admin API (`http://localhost:9000`)\n\n- `GET /health` - Server health status\n- `GET /stats` - System statistics\n- `GET /queues` - List all queues\n- `GET /queues/{name}` - Queue details\n\n---\n\n## FAQ\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eDo I need AWS or GCP accounts?\u003c/b\u003e\u003c/summary\u003e\n\nNo! Use dummy credentials. lclq doesn't verify signatures or require real accounts.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eIs lclq production-ready?\u003c/b\u003e\u003c/summary\u003e\n\nlclq is designed for **local development and CI/CD testing**. For production workloads, use actual cloud services (AWS SQS, GCP Pub/Sub).\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eWhich SDKs work with lclq?\u003c/b\u003e\u003c/summary\u003e\n\nAll official AWS and GCP SDKs. **Tested:** Python, JavaScript, Go, Rust.\n**Should work:** Java, Ruby, .NET (not yet tested).\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eDoes lclq persist data?\u003c/b\u003e\u003c/summary\u003e\n\n- **In-memory (default):** Data lost on restart. Perfect for tests.\n- **SQLite:** Fully persistent with ACID guarantees.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eHow does lclq compare to LocalStack/ElasticMQ?\u003c/b\u003e\u003c/summary\u003e\n\n- **LocalStack:** Full AWS emulation (100+ services). lclq focuses on queues only = faster, simpler.\n- **ElasticMQ:** SQS-only. lclq adds Pub/Sub + better performance (182x faster).\n\u003c/details\u003e\n\n---\n\n## Building from Source\n\n```bash\n# Clone repository\ngit clone https://github.com/erans/lclq.git\ncd lclq\n\n# Build\ncargo build --release\n\n# Run tests (408 unit tests, 66% coverage)\ncargo test\n\n# Run benchmarks\ncargo bench\n\n# Run integration tests\ncd tests/integration/python \u0026\u0026 poetry install \u0026\u0026 poetry run pytest\ncd tests/integration/javascript \u0026\u0026 npm install \u0026\u0026 npm test\n```\n\n---\n\n## Project Status\n\n| Component | Status | Tests |\n|-----------|--------|-------|\n| AWS SQS Core | ✅ Complete | 7/7 passing |\n| GCP Pub/Sub Core | ✅ Complete | 53/53 passing |\n| In-Memory Backend | ✅ Complete | 408 unit tests |\n| SQLite Backend | ✅ Complete | 5 integration tests |\n| CLI \u0026 Admin API | ✅ Complete | Full coverage |\n\nSee [docs/TODO.md](docs/TODO.md) for detailed roadmap.\n\n---\n\n## Documentation\n\n- **[Quick Start Guide](docs/quickstart.md)** - Get started in 5 minutes\n- **[Benchmarks](docs/benchmarks.md)** - Performance analysis\n- **[Changelog](CHANGELOG.md)** - Release history and version changes\n- **[Product Roadmap](docs/prd.md)** - Feature planning\n- **[Technical Roadmap](docs/tech-prd.md)** - Implementation details\n\n---\n\n## Support \u0026 Contributing\n\n- 🐛 **Bug Reports:** [GitHub Issues](https://github.com/erans/lclq/issues)\n- 💬 **Discussions:** [GitHub Discussions](https://github.com/erans/lclq/discussions)\n- 🤝 **Contributing:** Pull requests welcome!\n\n---\n\n## License\n\nDual-licensed under:\n- **Apache License 2.0** ([LICENSE-APACHE](LICENSE-APACHE))\n- **MIT License** ([LICENSE-MIT](LICENSE-MIT))\n\nChoose whichever works best for your project.\n\n---\n\n## Acknowledgments\n\nInspired by excellent local development tools:\n- [LocalStack](https://localstack.cloud/) - Local AWS emulation\n- [Google Cloud Pub/Sub Emulator](https://cloud.google.com/pubsub/docs/emulator) - Official GCP emulator\n- [ElasticMQ](https://github.com/softwaremill/elasticmq) - SQS-compatible queue\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n**⭐ Star this repo if lclq makes your development faster!**\n\n[Get Started](https://github.com/erans/lclq/releases) • [Report Bug](https://github.com/erans/lclq/issues) • [Request Feature](https://github.com/erans/lclq/issues)\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ferans%2Flclq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ferans%2Flclq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ferans%2Flclq/lists"}