{"id":29715039,"url":"https://github.com/longlene/snow","last_synced_at":"2025-07-24T05:02:33.881Z","repository":{"id":304578076,"uuid":"1018660650","full_name":"longlene/snow","owner":"longlene","description":null,"archived":false,"fork":false,"pushed_at":"2025-07-14T00:24:48.000Z","size":20,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-07-14T02:52:33.244Z","etag":null,"topics":["erlang","snowflake"],"latest_commit_sha":null,"homepage":"","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/longlene.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-12T18:42:24.000Z","updated_at":"2025-07-14T00:24:51.000Z","dependencies_parsed_at":"2025-07-14T02:52:36.887Z","dependency_job_id":"88a0f83a-c113-4993-8396-bd610c9925c1","html_url":"https://github.com/longlene/snow","commit_stats":null,"previous_names":["longlene/snow"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/longlene/snow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/longlene%2Fsnow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/longlene%2Fsnow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/longlene%2Fsnow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/longlene%2Fsnow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/longlene","download_url":"https://codeload.github.com/longlene/snow/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/longlene%2Fsnow/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266796332,"owners_count":23985471,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-24T02:00:09.469Z","response_time":99,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["erlang","snowflake"],"created_at":"2025-07-24T05:01:05.496Z","updated_at":"2025-07-24T05:02:33.871Z","avatar_url":"https://github.com/longlene.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Snow - High-Performance Lock-Free Erlang Snowflake ID Generator\n\nA high-performance, lock-free Snowflake ID generation library for Erlang, built with `persistent_term` and `atomics` for truly concurrent operation.\n\n## Features\n\n- 🚀 **Lock-Free Design**: Uses Erlang atomics and CAS operations for true lock-free concurrency\n- ⚡ **High Performance**: \n  - Single-thread: 2M+ IDs/sec\n  - Concurrent (20 processes): 3.7M+ IDs/sec with 1.8x speedup\n- 🌍 **Distributed-Friendly**: Supports region and worker ID configuration for multi-node deployments\n- ⏰ **Clock Protection**: Automatic detection and handling of clock drift/backwards\n- 🔧 **Configurable**: Custom epoch, region, worker ID, and compile-time bit allocation\n- 🎯 **Aggressive CAS**: Direct retry with CAS return value optimization for maximum throughput\n- 📦 **Zero Dependencies**: Pure Erlang implementation\n\n## ID Structure (64-bit, can be changed via rebar.config)\n\n```\n| 1 bit | 41 bits | 4 bits | 6 bits | 12 bits |\n|-------|---------|--------|--------|---------|\n|   0   |timestamp| region | worker |sequence |\n```\n\n- **Reserved** (1 bit): Always 0\n- **Timestamp** (41 bits): Millisecond timestamp, ~69 years from epoch\n- **Region** (4 bits): Region ID (0-15, configurable)\n- **Worker** (6 bits): Worker node ID (0-63, configurable)  \n- **Sequence** (12 bits): Per-millisecond sequence (0-4095, auto-calculated)\n\n## Quick Start\n\n### Add Dependency\n\n```erlang\n{deps, [\n    {snow, {git, \"https://github.com/longlene/snow.git\", {branch, \"main\"}}}\n]}.\n```\n\n### Configuration\n\nConfigure in `sys.config`:\n\n```erlang\n[\n  {snow, [\n    {epoch, 1640995200000}, % 2022-01-01 00:00:00 UTC\n    {region, 0},            % 0-15\n    {worker, 0}             % 0-63\n  ]}\n].\n```\n\n### Usage Examples\n\n```erlang\n%% Start application\napplication:ensure_all_started(snow).\n\n%% Generate single ID\nId = snow:next_id().\n% 7128834371969425408\n\n%% Batch generate IDs (optimized for bulk operations)\nIds = snow:next_ids(1000).\n\n%% Decode ID\n#{timestamp := Ts, region := R, worker := W, sequence := Seq} = snow:decode_id(Id).\n\n%% Custom initialization (global worker)\nsnow:init(1640995200000, 5, 10).\n\n%% Multi-worker API - create independent workers\nWorker1 = snow:start_worker(1640995200000, 1, 2),\nWorker2 = snow:start_worker(1640995200000, 1, 3),\n\n%% Generate IDs using specific workers\nId1 = snow:next_id(Worker1),\nId2 = snow:next_id(Worker2),\nIds = snow:next_ids(Worker1, 1000),\n\n%% Get configuration info\nsnow:info(),                    % Global worker info\nsnow:worker_info(Worker1).      % Specific worker info\n```\n\n## API Reference\n\n### snow:init/3\nInitialize the global generator with custom configuration.\n\n**Parameters:**\n- `Epoch :: non_neg_integer()` - Start timestamp in milliseconds\n- `Region :: 0..15` - Region ID\n- `Worker :: 0..63` - Worker node ID\n\n### snow:start_worker/3\nCreate a new independent worker instance (recommended for multi-worker scenarios).\n\n**Parameters:**\n- `Epoch :: non_neg_integer()` - Start timestamp in milliseconds\n- `Region :: 0..15` - Region ID\n- `Worker :: 0..63` - Worker node ID\n\n**Returns:** `worker_handle()` - Opaque worker handle for ID generation\n\n### snow:next_id/0\nGenerate a single Snowflake ID using the global worker.\n\n**Returns:** `non_neg_integer()`\n\n### snow:next_id/1\nGenerate a single Snowflake ID using a specific worker.\n\n**Parameters:**\n- `WorkerHandle :: worker_handle()` - Worker handle from `start_worker/3`\n\n**Returns:** `non_neg_integer()`\n\n### snow:next_ids/1\nEfficiently generate multiple IDs in batch using the global worker.\n\n**Parameters:**\n- `Count :: pos_integer()` - Number of IDs to generate\n\n**Returns:** `[non_neg_integer()]`\n\n### snow:next_ids/2\nEfficiently generate multiple IDs in batch using a specific worker.\n\n**Parameters:**\n- `WorkerHandle :: worker_handle()` - Worker handle from `start_worker/3`\n- `Count :: pos_integer()` - Number of IDs to generate\n\n**Returns:** `[non_neg_integer()]`\n\n### snow:decode_id/1\nDecode a Snowflake ID into its components.\n\n**Parameters:**\n- `Id :: non_neg_integer()` - Snowflake ID to decode\n\n**Returns:**\n```erlang\n#{\n    timestamp := non_neg_integer(),\n    region := non_neg_integer(),\n    worker := non_neg_integer(),\n    sequence := non_neg_integer()\n}\n```\n\n### snow:info/0\nGet global worker configuration and bit allocation.\n\n**Returns:**\n```erlang\n#{\n    epoch := non_neg_integer(),\n    region := non_neg_integer(),\n    worker := non_neg_integer(),\n    bits := #{\n        timestamp := pos_integer(),\n        region := pos_integer(),\n        worker := pos_integer(),\n        sequence := pos_integer()\n    }\n}\n```\n\n### snow:worker_info/1\nGet specific worker configuration and bit allocation.\n\n**Parameters:**\n- `WorkerHandle :: worker_handle()` - Worker handle from `start_worker/3`\n\n**Returns:**\n```erlang\n#{\n    epoch := non_neg_integer(),\n    region := non_neg_integer(),\n    worker := non_neg_integer(),\n    bits := #{\n        timestamp := pos_integer(),\n        region := pos_integer(),\n        worker := pos_integer(),\n        sequence := pos_integer()\n    }\n}\n```\n\n## Performance\n\nBenchmarks on modern hardware (Erlang/OTP 26):\n\n| Scenario | Performance | Notes |\n|----------|------------|--------|\n| Single-thread | 2.07M IDs/sec | Maximum single-threaded performance |\n| 10 processes | 2.26M IDs/sec | Good concurrent performance |\n| 20 processes | 3.24M IDs/sec | Excellent concurrent scalability |\n| 50 processes | 2.75M IDs/sec | Strong performance under high load |\n\n### Performance Optimizations\n\n1. **Pre-computed Base IDs**: Region and worker bits calculated once at initialization\n2. **Compile-time Constants**: Bit shifts resolved at compile time\n3. **CAS Return Value Optimization**: Uses actual values from failed CAS operations for immediate retry\n4. **Batch Sequence Reservation**: Single CAS operation reserves multiple sequence numbers\n5. **Lock-Free Design**: Pure atomic operations without any locking mechanisms\n\n## Compile-time Configuration\n\nCustomize bit allocation by modifying `rebar.config`:\n\n```erlang\n{erl_opts, [\n    debug_info,\n    {d, timestamp_bits, 41},  % Timestamp bits (supports ~69 years)\n    {d, region_bits, 4},      % Region bits (16 regions)\n    {d, worker_bits, 6}       % Worker bits (64 workers)\n    % sequence_bits automatically calculated: 64 - 1 - 41 - 4 - 6 = 12\n]}.\n```\n\n**Example configurations:**\n- High regions: `{d, region_bits, 5}` + `{d, worker_bits, 5}` = 32 regions, 32 workers, 2048/ms\n- High workers: `{d, region_bits, 3}` + `{d, worker_bits, 7}` = 8 regions, 128 workers, 2048/ms\n\n## Architecture\n\n### Lock-Free Design\n- Uses `atomics:compare_exchange/4` for atomic sequence updates\n- No GenServer or process-based state management\n- Configuration stored in `persistent_term` for fast access\n\n### CAS Optimization Strategy\nMaximizes throughput through immediate CAS retry:\n\n```erlang\n%% Uses return value from failed CAS operations\ncase atomics:compare_exchange(AtomicRef, 1, OldVal, NewVal) of\n    ok -\u003e\n        %% Success - return generated ID\n        construct_final_id(Timestamp, BaseId, Sequence);\n    CurrentVal -\u003e\n        %% Failed - retry immediately with current value\n        generate_id_loop(Epoch, BaseId, AtomicRef, CurrentVal)\nend\n```\n\n### Multi-Worker Architecture\n\n**Single Worker per Node (Traditional):**\n```erlang\n%% Node 1\nsnow:init(1640995200000, 0, 1).\n\n%% Node 2  \nsnow:init(1640995200000, 0, 2).\n\n%% Node 3\nsnow:init(1640995200000, 0, 3).\n```\n\n**Multiple Workers per Node (Recommended):**\n```erlang\n%% Create multiple independent workers on same node\nWorker1 = snow:start_worker(1640995200000, 1, 1),  % Region 1, Worker 1\nWorker2 = snow:start_worker(1640995200000, 1, 2),  % Region 1, Worker 2  \nWorker3 = snow:start_worker(1640995200000, 2, 1),  % Region 2, Worker 1\n\n%% Use workers for different purposes\nOrderId = snow:next_id(Worker1),      % Order service\nUserId = snow:next_id(Worker2),       % User service  \nPaymentId = snow:next_id(Worker3),    % Payment service\n```\n\n**Key Benefits of Multi-Worker:**\n- **Zero contention**: Each worker has independent atomic state\n- **Better performance**: Avoid global persistent_term lookups  \n- **Service isolation**: Different services can use different workers\n- **Flexible deployment**: Support any number of workers per node\n\n## Testing\n\n```bash\n# Run unit tests\nrebar3 ct\n\n# Run performance benchmarks\nmake bench\n\n# Run with custom compiler optimizations\nrebar3 as bench compile\n```\n\n## Error Handling\n\n- **Clock backwards**: Throws `{clock_backwards, OldTime, NewTime}` \n- **Invalid configuration**: Validates region/worker bounds at initialization\n- **Sequence exhaustion**: Automatically waits for next millisecond\n\n## License\n\nMIT License - see LICENSE file for details.\n\n## Contributing\n\nContributions welcome! Please ensure:\n- All tests pass (`rebar3 ct`)\n- Performance benchmarks show no regression\n- Code follows existing style conventions\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flonglene%2Fsnow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flonglene%2Fsnow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flonglene%2Fsnow/lists"}