{"id":13492044,"url":"https://github.com/sandglass/sandglass","last_synced_at":"2025-09-27T07:32:27.611Z","repository":{"id":48493784,"uuid":"104253644","full_name":"sandglass/sandglass","owner":"sandglass","description":"Sandglass is a distributed, horizontally scalable, persistent, time sorted message queue.","archived":true,"fork":false,"pushed_at":"2018-08-25T20:11:00.000Z","size":4994,"stargazers_count":1528,"open_issues_count":8,"forks_count":45,"subscribers_count":43,"default_branch":"dev","last_synced_at":"2025-09-13T18:21:48.036Z","etag":null,"topics":["distributed","distributed-systems","go","message-queue","messaging","taskqueue"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sandglass.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-20T18:33:28.000Z","updated_at":"2025-09-09T07:38:00.000Z","dependencies_parsed_at":"2022-09-10T16:32:21.004Z","dependency_job_id":null,"html_url":"https://github.com/sandglass/sandglass","commit_stats":null,"previous_names":["celrenheit/sandglass"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/sandglass/sandglass","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandglass%2Fsandglass","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandglass%2Fsandglass/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandglass%2Fsandglass/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandglass%2Fsandglass/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sandglass","download_url":"https://codeload.github.com/sandglass/sandglass/tar.gz/refs/heads/dev","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sandglass%2Fsandglass/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275333380,"owners_count":25446119,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-15T02:00:09.272Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed","distributed-systems","go","message-queue","messaging","taskqueue"],"created_at":"2024-07-31T19:01:02.594Z","updated_at":"2025-09-27T07:32:22.573Z","avatar_url":"https://github.com/sandglass.png","language":"Go","funding_links":[],"categories":["Go","Queues","go"],"sub_categories":["Vectors"],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"Sandglass Logo\" src=\"https://raw.githubusercontent.com/celrenheit/gifs/master/sandglass/brand%2Blogo-sm.png\"  /\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\nSandglass is a distributed, horizontally scalable, persistent, time ordered message queue. It was developed to support asynchronous tasks and message scheduling which makes it suitable for usage as a task queue.\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/sandglass/sandglass/releases/latest\"\u003e\n        \u003cimg alt=\"Release\" src=\"https://img.shields.io/github/release/sandglass/sandglass.svg?style=flat-square\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://travis-ci.org/sandglass/sandglass\"\u003e\n        \u003cimg alt=\"Build Status\" src=\"https://img.shields.io/travis/sandglass/sandglass.svg?style=flat-square\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"LICENSE\"\u003e\n        \u003cimg alt=\"License\" src=\"https://img.shields.io/badge/license-apache-blue.svg?style=flat-square\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n## Table of contents\n\n- [Features](#features)\n- [Installation](#installation)\n- [Getting started](#getting-started)\n- [Documentation](#documentation)\n- [Motivation](#motivation)\n- [Clients](#clients)\n  - [Go](#go)\n    - [Installation](#installation-1)\n    - [Documentation](#documentation)\n    - [Usage](#usage)\n      - [Producer](#producer)\n      - [Consumer](#consumer)\n  - [Java, Python, Node.js, Ruby](#java-python-nodejs-ruby)\n- [Architecture](#architecture)\n  - [General](#general)\n  - [Topic](#topic)\n  - [Offset Tracking](#offset-tracking)\n  - [Technologies used](#technologies-used)\n- [Contributing](#contributing)\n- [License](#license)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n## Features\n\n* Horizontal scalability\n* Highly available\n* Persistent storage\n* Time ordered\n* Multiple consumers per group for a partition\n* Produce message to be consumed in the future\n* Acknowledge/NotAcknowledge each message individualy\n* Automatic redelivery and commit offset tracking\n* Language agnostic\n\n**EXPERIMENTAL**: This is a prototype of a side project. This should not be used in production in its current form as things may change quickly without notice.\n\n## Installation\n\nOn MacOS using [Homebrew](https://brew.sh):\n```shell\nbrew install celrenheit/taps/sandglass\n```\n\nFor other platforms, you can grab binaries [here](https://github.com/sandglass/sandglass/releases). \n\n## Getting started\n\n\u003e NOTE: All data will be stored in /tmp/node1. If you wish to change this, copy `demo/node1.yaml` and modify it accordingly.\n\nFirst, let's launch sandglass server:\n\n```shell\nsandglass --config https://raw.githubusercontent.com/sandglass/sandglass/master/demo/node1.yaml --offset_replication_factor 1\n```\n\nIn a second terminal window, create a _emails_ topic:\n\n```shell\nsandctl topics create emails --num_partitions 3 --replication_factor 1\n```\n\n...produce 10,000 messages:\n\n```shell\nsandctl produce emails '{\"dest\" : \"hi@example.com\"}' -n 10000\n```\n\n...and consume from the _emails_ topic:\n\n```shell\nsandctl consume emails\n```\n\n(or if you wish to watch you can use `sandctl consume -f emails` to see messages coming live)\n\n\u003e We are using a single node cluster, this is not recommended for production.\n\nAdd a second node to the cluster:\n```shell\nsandglass --config https://raw.githubusercontent.com/sandglass/sandglass/master/demo/node2.yaml\n```\n\nand repeat the same steps described above for another topic and increasing the replication factor to 2.\n\n## Motivation\n\nAs previously asked ([#4](https://github.com/sandglass/sandglass/issues/4)), the purpose of this project might not seem clear. In short, there is two goals. \n\nThe first is to be able to track each message individually (i.e. not using a single commit offset) to make suitable for asynchronous tasks.\n\nThe second is the ability to schedule messages to be consumed in the future. This make it suitable for retries.\n\n## Documentation\n\n\u003e Documentation is a work in progress and still lacking.\n\n* [API docs](https://docs.sandglass.stream/)\n* Overview - TODO\n* Clients development guides - TODO\n* Documentation the different ways of starting a cluster\n\n## Clients\n\n### Go\n\n#### Installation\n\n```shell\ngo get -u github.com/sandglass/sandglass-client/go/sg\n```\n\n#### Documentation\n\nThe documentation is available on [godoc](https://godoc.org/github.com/sandglass/sandglass-client/go/sg).\n\n\n#### Usage\n\n##### Producer\n\n```go\n// Let's first create a client by providing adresses of nodes in the sandglass cluster\nclient, err := sg.NewClient(\n    sg.WithAddresses(\":7170\"),\n)\nif err != nil {\n    panic(err)\n}\ndefer client.Close()\n\n// Now we produce a new message\n// Notice the empty string \"\" in the 3th argument, meaning let sandglass choose a random partition\nerr := client.Produce(context.Background(), \"emails\", \"\", \u0026sgproto.Message{\n    Value: []byte(\"Hello, Sandglass!\"),\n})\nif err != nil {\n    panic(err)\n}\n```\n\nIn order to produce message in the future, you need to specify a custom offset:\n\n```go\ninOneHour := time.Now().Add(1 * time.Hour)\ngen := sandflake.NewFixedTimeGenerator(inOneHour)\n\nmsg := \u0026sgproto.Message{\n\tOffset: gen.Next(),\n\tValue:  []byte(\"Hello\"),\n}\n\nerr := client.ProduceMessage(context.Background(), \"emails\", \"\", msg)\nif err != nil {\n\treturn err\n}\n```\n\nThis will produce a message that will be available for consumption in 1h.\n\n##### Consumer\n\n\n1. High-level\n\n```go\n// Let's first create a client by providing adresses of nodes in the sandglass cluster\nclient, err := sg.NewClient(\n    sg.WithAddresses(\":7170\"),\n)\nif err != nil {\n    panic(err)\n}\ndefer client.Close()\n\nmux := sg.NewMux()\nmux.SubscribeFunc(\"emails\", func(msg *sgproto.Message) error {\n    // handle message\n    log.Printf(\"received: %s\\n\", string(msg.Value))\n    return nil\n})\n\nm := \u0026sg.MuxManager{\n    Client:               c,\n    Mux:                  mux,\n    ReFetchSleepDuration: dur,\n}\n\nerr = m.Start()\nif err != nil {\n    log.Fatal(err)\n}\n```\n\n2. Low level\n\n```go\n// Let's first create a client by providing adresses of nodes in the sandglass cluster\nclient, err := sg.NewClient(\n    sg.WithAddresses(\":7170\"),\n)\nif err != nil {\n    panic(err)\n}\ndefer client.Close()\n\n\n// Listing partitions in order to choose one to consume from\npartitions, err := client.ListPartitions(context.Background(), topic)\nif err != nil {\n    panic(err)\n}\n\n// we are choosing only one partition for demo purposes\npartition := partitions[0]\n\n// Create a new consumer using consumer group emails-sender and consumer name consumer1\nconsumer := client.NewConsumer(topic, partition, \"emails-sender\", \"consumer1\")\n\n// and consume messages\nmsgCh, err := consumer.Consume(context.Background())\nif err != nil {\n    panic(err)\n}\n\nfor msg := range msgCh {\n    // do an amazing amount of work\n    log.Printf(\"received: %s\\n\", string(msg.Value))\n\n    // when we are done, we Acknowledge the message\n    // but we can also NotAcknowledge to trigger a redelivery of the message\n    if err := consumer.Acknowledge(context.Background(), msg); err!=nil {\n        panic(err)\n    }\n}\n```\n\n### Java, Python, Node.js, Ruby\n\nInterested in having client for one the following languages ? \n\nSupport is planned but there is no specific schedule. So, if you are interested to quickly have a client in your language, **help is welcome!**\n\nCheck the raw generated code available on https://github.com/sandglass/sandglass-grpc and feel free to submit your through a pull request to https://github.com/sandglass/sandglass-client.\n\n\n## Architecture\n\n### General\n\n![general architecture illustration](img/architecture-general.png)\n\n### Topic\n\nThere is two kinds of topics:\n* Timer:\n   * Fixed number of partitions (set up-front, could change)\n   * Time ordered using [sandflake IDs](https://github.com/celrenheit/sandflake)\n   * Can produce messages in the future\n\n* KV:\n   * Fixed number of partitions (set up-front, cannot change)\n   * Behaves like a distributed key value store\n\n\nA topic has a number of partitions.\nData is written into a single partition. Either the destination partition is specified by the producer. Otherwise, we fallback to choosing the destination partition using a consistent hashing algorithm.\n\nEach produced message to a partition writes a message to a Write Ahead Log (WAL) and to the View Log (VL).\nThe WAL is used for the replication logic, it is sorted in the order each message was produced.\nThe View Log is used for message consumption, it is mainly sorted by time (please refer to [sandflake ids](https://github.com/celrenheit/sandflake) for the exact composition) for a Timer topics and by keys for KV topics.\n\n\nA message is composed of the following fields:\n\n        index                       \u003c- position in the WAL\n\n        offset                      \u003c- position in the view log for timer topics\n        key and clusteringKey       \u003c- position in the view log for key for kv topics (key is used for partitioning)\n\n        value                       \u003c- your payload\n\n### Offset Tracking\n\nSandglass is responsible for maintaining two offsets for each consumer group: \n* Commited: the offset below which all messages have been ACKed\n* Consumed: the last consumed message\n\nWhen consuming sandglass starts from the last commited until the last consumed message to check the redelivery of messages. And from the last consumed offset until the last produced message to deliver the new messages. These two actions are done in parallel.\n\n### Technologies used\n\n* Leader Election: [hashicorp/raft](https://github.com/hashicorp/raft)\n* Persitance: [RocksDB](http://rocksdb.org/) or [Badger](https://github.com/dgraph-io/badger) \n* Communication: [GRPC](https://grpc.io)\n* Gossip: [Serf](https://github.com/hashicorp/serf)\n* and many more\n\n## Contributing\n\nWant to contribute to Sandglass ? Awesome! Feel free to submit an issue or a pull request.\n\nHere are some ways you can help:\n\n* Report bugs\n* Your language is not supported ? Feel free to [build a client](#java-python-nodejs-ruby). Trust me, it should not take you long :)\n* Improve code/documentation\n* Propose new features\n* and more...\n\n## License\n\nThis project is licensed under the Apache License 2.0 available [here](LICENSE).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsandglass%2Fsandglass","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsandglass%2Fsandglass","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsandglass%2Fsandglass/lists"}