{"id":17856044,"url":"https://github.com/yudhasubki/blockqueue","last_synced_at":"2026-01-27T00:25:27.037Z","repository":{"id":210681039,"uuid":"727176568","full_name":"yudhasubki/blockqueue","owner":"yudhasubki","description":"Cost-effective Job Queue with Pub/Sub. Embeddable in Go applications, or runnable as a standalone HTTP service. Supports SQLite, Turso, and PostgreSQL.","archived":false,"fork":false,"pushed_at":"2026-01-10T07:07:10.000Z","size":669,"stargazers_count":77,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-11T01:53:32.785Z","etag":null,"topics":["background-jobs","distributed-systems","go","libsql","libsqlite3","message-broker","message-queue","nutsdb","queue","sqlite3"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yudhasubki.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-12-04T10:42:20.000Z","updated_at":"2026-01-10T07:17:30.000Z","dependencies_parsed_at":"2023-12-18T14:39:57.939Z","dependency_job_id":"8e16af10-62af-4cfb-bc51-64062c6cb35d","html_url":"https://github.com/yudhasubki/blockqueue","commit_stats":null,"previous_names":["yudhasubki/blockqueue"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/yudhasubki/blockqueue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yudhasubki%2Fblockqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yudhasubki%2Fblockqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yudhasubki%2Fblockqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yudhasubki%2Fblockqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yudhasubki","download_url":"https://codeload.github.com/yudhasubki/blockqueue/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yudhasubki%2Fblockqueue/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28792959,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T21:49:50.245Z","status":"ssl_error","status_checked_at":"2026-01-26T21:48:29.455Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-jobs","distributed-systems","go","libsql","libsqlite3","message-broker","message-queue","nutsdb","queue","sqlite3"],"created_at":"2024-10-28T03:00:44.668Z","updated_at":"2026-01-27T00:25:27.024Z","avatar_url":"https://github.com/yudhasubki.png","language":"Go","readme":"# Block Queue\n\n**Block Queue** is a lightweight message queue with pub/sub mechanism for a cheap, robust, reliable, and durable messaging system.\n\nBuilt on [SQLite3](https://www.sqlite.org/index.html) with support for [Turso Database](https://turso.tech/) and [PostgreSQL](https://www.postgresql.org/).\n\n## Why BlockQueue\n\nWhile Kafka, Redis, or SQS are excellent products, they are complex and require significant resources. BlockQueue is built for simplicity, low resource usage, and cost-effectiveness.\n\n## Features\n- Cost-Effective: Budget-friendly solution for messaging needs\n- Pub/Sub Mechanism: Easy communication and real-time updates\n- Low Latency: Minimized network latency with SQLite as default storage\n- Multiple Drivers: SQLite, Turso, and PostgreSQL support\n\n## Installation\n\n### Binary\nDownload from releases or build from source:\n```bash\ngo build -o blockqueue ./cmd/blockqueue\n```\n\n### As Library\n```bash\ngo get -u github.com/yudhasubki/blockqueue\n```\n\n## Usage\n\nBlockQueue can be used in two ways:\n\n### 1. HTTP Server Mode\n\nStart the server:\n```bash\n./blockqueue http -config=config.yaml\n```\n\nExample config.yaml:\n```yaml\nhttp:\n  port: 8080\n  shutdown: \"30s\"\n  driver: \"sqlite\"\nsqlite:\n  db_name: \"blockqueue\"\n  busy_timeout: 5000\nwrite_buffer:\n  batch_size: 100\n  flush_interval: \"100ms\"\n  buffer_size: 10000\n```\n\nThen use HTTP API:\n```bash\n# Create topic with subscriber\ncurl -X POST http://localhost:8080/topics \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"name\": \"orders\",\n    \"subscribers\": [{\"name\": \"processor\", \"option\": {\"max_attempts\": 5, \"visibility_duration\": \"5m\"}}]\n  }'\n\n# Publish message\ncurl -X POST http://localhost:8080/topics/orders/messages \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"message\": \"order created\"}'\n\n# Read message (long-polling)\ncurl http://localhost:8080/topics/orders/subscribers/processor?timeout=5s\n\n# Acknowledge message\ncurl -X DELETE http://localhost:8080/topics/orders/subscribers/processor/messages/{message_id}\n```\n\n### 2. Library Mode (Embedded)\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"fmt\"\n    \"log\"\n    \"time\"\n\n    \"github.com/yudhasubki/blockqueue\"\n    \"github.com/yudhasubki/blockqueue/pkg/io\"\n    \"github.com/yudhasubki/blockqueue/pkg/sqlite\"\n)\n\nfunc main() {\n    // Initialize SQLite driver\n    db, err := sqlite.New(\"queue.db\", sqlite.Config{BusyTimeout: 5000})\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    ctx := context.Background()\n\n    // Create BlockQueue instance\n    stream := blockqueue.New(db, blockqueue.BlockQueueOption{\n        WriteBufferConfig: blockqueue.WriteBufferConfig{\n            BatchSize:     100,\n            FlushInterval: 100 * time.Millisecond,\n            BufferSize:    1000,\n        },\n    })\n\n    if err := stream.Run(ctx); err != nil {\n        log.Fatal(err)\n    }\n    defer stream.Close()\n\n    // Create topic and subscriber\n    request := io.Topic{\n        Name: \"orders\",\n        Subscribers: io.Subscribers{\n            {\n                Name: \"processor\",\n                Option: io.SubscriberOpt{\n                    MaxAttempts:        3,\n                    VisibilityDuration: \"1m\",\n                },\n            },\n        },\n    }\n    topic := request.Topic()\n    stream.AddJob(ctx, topic, request.Subscriber(topic.Id))\n\n    // Start consumer goroutine\n    go func() {\n        for {\n            messages, err := stream.Read(ctx, topic, \"processor\")\n            if err != nil {\n                log.Printf(\"read error: %v\", err)\n                continue\n            }\n            for _, msg := range messages {\n                log.Printf(\"received: %s\", msg.Message)\n                stream.Ack(ctx, topic, \"processor\", msg.Id)\n            }\n        }\n    }()\n\n    // Publish messages\n    for i := 0; i \u003c 10; i++ {\n        stream.Publish(ctx, topic, io.Publish{\n            Message: fmt.Sprintf(\"order-%d\", i),\n        })\n    }\n}\n```\n\n## Drivers\n\n### SQLite (Recommended)\nBest for single-node deployments. Highest throughput with minimal latency.\n\n```go\ndb, _ := sqlite.New(\"queue.db\", sqlite.Config{\n    BusyTimeout: 5000,\n    CacheSize:   -4000,  // 4MB cache\n    MmapSize:    0,      // Disable mmap for minimal memory\n})\n```\n\n### PostgreSQL\nFor multi-client scenarios or when you already have PostgreSQL infrastructure.\n\n```go\ndb, _ := postgre.New(postgre.Config{\n    Host:     \"localhost\",\n    Username: \"user\",\n    Password: \"pass\",\n    Name:     \"blockqueue\",\n    Port:     5432,\n})\n```\n\n### Turso\nFor edge deployments with LibSQL.\n\n```go\ndb, _ := turso.New(\"libsql://your-db.turso.io?authToken=TOKEN\")\n```\n\n## API Reference\n\n| Endpoint | Method | Description |\n|----------|--------|-------------|\n| /topics | POST | Create topic with subscribers |\n| /topics/{topic}/messages | POST | Publish message |\n| /topics/{topic}/subscribers/{sub} | GET | Read messages (long-polling) |\n| /topics/{topic}/subscribers/{sub}/messages/{id} | DELETE | Acknowledge message |\n| /topics/{topic}/subscribers | POST | Add subscribers |\n| /topics/{topic}/subscribers/{sub} | DELETE | Remove subscriber |\n\n### Subscriber Options\n| Option | Example | Description |\n|--------|---------|-------------|\n| max_attempts | 5 | Maximum redelivery attempts |\n| visibility_duration | 5m | Time before unacked message is redelivered |\n\n## Benchmark\n\nMacBook Pro M1, 8GB RAM\n\n### SQLite (100 VUs, 10s)\n```\nhttp_reqs..................: 388908  38885/s\nhttp_req_duration..........: med=1.19ms p(95)=7.02ms p(99.9)=30.47ms\n```\n\n### PostgreSQL (100 VUs, 10s)\n```\nhttp_reqs..................: 113626  11340/s\nhttp_req_duration..........: med=4.87ms p(95)=18.26ms p(99.9)=275.74ms\n```\n\n## Architecture\n\n![Publish Architecture](https://github.com/yudhasubki/blockqueue/blob/main/docs/img/publisher_architecture.png)\n\n![Consumer Architecture](https://github.com/yudhasubki/blockqueue/blob/main/docs/img/consumer_architecture.png)\n\n## Roadmap\n- [x] HTTP Protocol\n- [x] Metrics (Prometheus)\n- [x] SQLite WAL Mode\n- [x] PostgreSQL Support\n- [ ] TCP Protocol\n- [ ] Go SDK\n- [ ] PHP SDK\n\n## Acknowledgment\nInspired by [Redis](https://redis.io), [Kafka](https://kafka.apache.org/), and [Amazon SQS](https://aws.amazon.com/sqs/).\n\n## License\nApache 2.0 License","funding_links":[],"categories":["Apps and Communities buidling on SQLite"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyudhasubki%2Fblockqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyudhasubki%2Fblockqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyudhasubki%2Fblockqueue/lists"}