https://github.com/breqdev/snowcloud
Service for generating unique, time-ordered IDs across distributed worker processes.
https://github.com/breqdev/snowcloud
flask redis
Last synced: about 1 year ago
JSON representation
Service for generating unique, time-ordered IDs across distributed worker processes.
- Host: GitHub
- URL: https://github.com/breqdev/snowcloud
- Owner: breqdev
- License: mit
- Created: 2021-07-13T02:04:05.000Z (almost 5 years ago)
- Default Branch: main
- Last Pushed: 2022-09-03T18:16:20.000Z (over 3 years ago)
- Last Synced: 2025-02-03T12:53:13.407Z (about 1 year ago)
- Topics: flask, redis
- Language: Python
- Homepage: https://breq.dev/projects/snowflake
- Size: 25.4 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# snowcloud
Snowcloud is a service for generating unique, time-ordered, 64-bit IDs across distributed worker processes. This format is commonly called "snowflake" IDs, and is used by [Twitter](https://blog.twitter.com/engineering/en_us/a/2010/announcing-snowflake) and [Discord](https://discord.com/developers/docs/reference#snowflakes).
## Snowflake ID Format
| Timestamp | Worker ID | Increment |
| --------- | --------- | --------- |
| 42 bits | 10 bits | 12 bits |
| milliseconds since Jan 1, 2020 | allocated by delegator | incremented by generator |
## Snowcloud Deployment Structure
A Snowcloud deployment consists of a *delegator* and many *generators*.
### Delegator
The delegator (located in `snowcloud/delegator.py`, run with e.g. `python3 -m snowcloud.delegator`) keeps track of assigning worker IDs to different generator processes. It uses [Redis](https://redis.io) to keep track of a pool of worker IDs. It then allows clients to register for a worker ID or renew their existing worker ID.
### Generators
Generators obtain a worker ID from the delegator and use it to generate their own Snowflake IDs. These can be integrated into Flask applications using the `snowcloud.flask_ext` module.
An example Flask server is located in `snowcloud/server.py`. However, more commonly, the Flask application would use the Snowflake ID internally instead. For instance, it could generate a Snowflake ID to use as the primary key column whenever it inserts a new row into the database.