{"id":28546133,"url":"https://github.com/alexnthnz/consistent-hashing","last_synced_at":"2025-08-02T12:06:28.583Z","repository":{"id":297757901,"uuid":"997807431","full_name":"alexnthnz/consistent-hashing","owner":"alexnthnz","description":"A lightweight implementation of consistent hashing for efficient data distribution in distributed systems. Supports dynamic node addition/removal and virtual nodes for balanced load distribution. Ideal for distributed databases, load balancers, and caching systems.","archived":false,"fork":false,"pushed_at":"2025-06-07T09:31:00.000Z","size":31,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-07T06:43:04.805Z","etag":null,"topics":["consistent-hashing","go"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alexnthnz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-07T08:17:26.000Z","updated_at":"2025-06-07T09:31:02.000Z","dependencies_parsed_at":"2025-06-07T09:40:05.326Z","dependency_job_id":null,"html_url":"https://github.com/alexnthnz/consistent-hashing","commit_stats":null,"previous_names":["alexnthnz/consistent-hashing"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/alexnthnz/consistent-hashing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexnthnz%2Fconsistent-hashing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexnthnz%2Fconsistent-hashing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexnthnz%2Fconsistent-hashing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexnthnz%2Fconsistent-hashing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alexnthnz","download_url":"https://codeload.github.com/alexnthnz/consistent-hashing/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexnthnz%2Fconsistent-hashing/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268385726,"owners_count":24242101,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-02T02:00:12.353Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consistent-hashing","go"],"created_at":"2025-06-09T23:09:00.818Z","updated_at":"2025-08-02T12:06:28.537Z","avatar_url":"https://github.com/alexnthnz.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Consistent Hashing\n\nA lightweight, high-performance implementation of consistent hashing for efficient data distribution in distributed systems. Supports dynamic node addition/removal, weighted nodes, multiple hash functions, and comprehensive monitoring. Ideal for distributed databases, load balancers, and caching systems.\n\n## ✨ Features\n\n- **🔄 Hash Ring Implementation**: Maps keys to nodes using a hash ring with minimal data movement\n- **⚖️ Weighted Nodes**: Support for weighted consistent hashing based on node capacity\n- **🔧 Pluggable Hash Functions**: Choose between FNV (fast) and SHA-256 (secure) hash functions\n- **🚀 Dynamic Scaling**: Handles dynamic node addition and removal seamlessly\n- **⚡ High Performance**: Optimized with binary search for O(log n) key lookups\n- **🔄 Replication Support**: Built-in support for data replication across multiple nodes\n- **🧵 Thread Safety**: Full concurrent access support with RWMutex protection\n- **📊 Monitoring \u0026 Analytics**: Built-in load distribution analysis and ring statistics\n- **🧪 Comprehensive Testing**: Includes unit tests, benchmarks, and concurrency tests\n- **🏭 Production Ready**: Extensively tested for reliability and performance\n\n## 📦 Installation\n\n```bash\ngo get github.com/alexnthnz/consistent-hashing\n```\n\n## 🚀 Quick Start\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"github.com/alexnthnz/consistent-hashing\"\n)\n\nfunc main() {\n    // Create a hash ring with 100 virtual nodes per physical node\n    ring := consistenthashing.NewHashRing(100)\n    \n    // Add nodes with different weights\n    ring.AddNode(\u0026consistenthashing.Node{\n        ID:     \"server1\",\n        Host:   \"192.168.1.10\",\n        Port:   8080,\n        Weight: 1, // Standard capacity\n    })\n    \n    ring.AddNode(\u0026consistenthashing.Node{\n        ID:     \"server2\", \n        Host:   \"192.168.1.11\",\n        Port:   8080,\n        Weight: 2, // Double capacity\n    })\n    \n    // Find which node should handle a key\n    node := ring.GetNode(\"user:1001\")\n    fmt.Printf(\"Key 'user:1001' -\u003e Node: %s (%s)\\n\", node.ID, node.String())\n    \n    // Get multiple nodes for replication\n    nodes := ring.GetNodes(\"user:1001\", 2)\n    fmt.Printf(\"Replication nodes: %v\\n\", nodes)\n}\n```\n\n## 🏗️ Project Structure\n\n```\nconsistent-hashing/\n├── 📄 consistent_hash.go          # Core implementation\n├── 🧪 consistent_hash_test.go     # Comprehensive tests  \n├── 📁 examples/\n│   ├── 🚀 basic_usage.go         # Basic usage demonstration\n│   ├── 🏗️ distributed_cache.go   # Advanced distributed cache example\n│   └── ⚡ advanced_features.go   # Advanced features showcase\n├── ⚙️ Makefile                   # Build and development commands\n├── 📦 go.mod                     # Go module definition\n├── 📖 README.md                  # This file\n└── 🚫 .gitignore                # Git ignore rules\n```\n\n## 📚 API Reference\n\n### Types\n\n#### `Node`\nRepresents a physical node in the distributed system.\n\n```go\ntype Node struct {\n    ID     string  // Unique identifier\n    Host   string  // Host address\n    Port   int     // Port number\n    Weight int     // Weight for load balancing (default: 1)\n}\n```\n\n#### `HashRing`\nThe main consistent hash ring structure with thread-safe operations.\n\n#### `HashFunction`\nInterface for pluggable hash functions.\n\n```go\ntype HashFunction interface {\n    Hash(key string) uint32\n}\n```\n\n### Functions\n\n#### `NewHashRing(virtualReplicas int, opts ...Option) *HashRing`\nCreates a new hash ring with options.\n\n```go\n// Default ring with FNV hash\nring := consistenthashing.NewHashRing(100)\n\n// Ring with SHA-256 hash\nring := consistenthashing.NewHashRing(100, \n    consistenthashing.WithHashFunction(\u0026consistenthashing.SHA256Hasher{}))\n```\n\n#### Core Operations\n- `AddNode(node *Node)` - Adds a node (thread-safe)\n- `RemoveNode(nodeID string)` - Removes a node (thread-safe)\n- `GetNode(key string) *Node` - Gets responsible node (thread-safe)\n- `GetNodes(key string, count int) []*Node` - Gets multiple nodes for replication\n\n#### Utility Methods\n- `HasNode(nodeID string) bool` - Checks if node exists\n- `GetNodeByID(nodeID string) *Node` - Gets node by ID\n- `GetAllNodes() []*Node` - Gets all nodes\n- `Size() int` - Number of physical nodes\n- `VirtualSize() int` - Number of virtual nodes\n\n#### Analytics \u0026 Monitoring\n- `GetLoadDistribution(keys []string) map[string]int` - Analyzes key distribution\n- `GetRingInfo() map[string]interface{}` - Gets ring statistics\n\n## 🎯 Examples\n\n### Basic Usage\n```bash\nmake run-basic\n```\n\n### Distributed Cache\n```bash\nmake run-cache\n```\n\n### Advanced Features\n```bash\nmake run-advanced\n```\n\nThis demonstrates:\n- Hash function comparison (FNV vs SHA-256)\n- Weighted consistent hashing\n- Thread safety and concurrent operations\n- Monitoring and analytics\n\n## 🛠️ Development\n\n### Running Tests\n```bash\n# Run all tests\nmake test\n\n# Run tests with race detection\nmake test-verbose\n\n# Run benchmarks\nmake benchmark\n\n# Generate coverage report\nmake coverage\n```\n\n### Code Quality\n```bash\n# Format code\nmake fmt\n\n# Run static analysis\nmake vet\n\n# Run all quality checks\nmake check\n```\n\n## ⚡ Performance\n\nThe implementation is optimized for high performance:\n\n- **O(log n)** key lookup time using binary search\n- **O(n log n)** node addition time (with sorting)\n- **O(n)** node removal time (in-place filtering)\n- **Thread-safe** with minimal lock contention\n- **Memory efficient** data structures\n\n### Benchmarks\n\nRecent benchmark results on Apple M1:\n\n```\nBenchmarkGetNode-8              12864074    92.21 ns/op    13 B/op    1 allocs/op\nBenchmarkGetNodeFNV-8           13238444    91.45 ns/op    13 B/op    1 allocs/op\nBenchmarkGetNodeSHA256-8         8215366   142.9 ns/op     13 B/op    1 allocs/op\nBenchmarkConcurrentGetNode-8     7179759   170.4 ns/op     13 B/op    1 allocs/op\n```\n\n**Key Insights:**\n- **FNV hash is ~56% faster** than SHA-256 for lookups\n- **Excellent concurrent performance** with minimal overhead\n- **Low memory allocation** per operation\n\n## 🎛️ Configuration\n\n### Hash Functions\n\nChoose the right hash function for your use case:\n\n```go\n// Fast hash (default) - good for most applications\nring := consistenthashing.NewHashRing(100)\n\n// Secure hash - for cryptographic requirements\nring := consistenthashing.NewHashRing(100, \n    consistenthashing.WithHashFunction(\u0026consistenthashing.SHA256Hasher{}))\n```\n\n### Virtual Nodes\n\nBalance performance vs. distribution:\n\n- **Low (10-50)**: Faster operations, acceptable imbalance\n- **Medium (50-200)**: Good balance of performance and distribution  \n- **High (200+)**: Better distribution, slightly slower operations\n\n### Weighted Nodes\n\nDistribute load based on node capacity:\n\n```go\nnodes := []*consistenthashing.Node{\n    {ID: \"small\", Host: \"host1\", Port: 8080, Weight: 1},   // 1x capacity\n    {ID: \"large\", Host: \"host2\", Port: 8080, Weight: 4},   // 4x capacity\n}\n```\n\n## 🎯 Use Cases\n\n### 🗄️ Distributed Caching\n- **Redis Cluster**: Distribute cache keys across Redis instances\n- **Memcached**: Load balance requests across Memcached servers\n- **CDN**: Route requests to geographically distributed cache servers\n\n### 🗃️ Database Sharding\n- **Horizontal Partitioning**: Distribute data across database shards\n- **Read Replicas**: Route read queries to appropriate replica servers\n- **Time-series Data**: Distribute time-series data across multiple databases\n\n### ⚖️ Load Balancing\n- **API Gateway**: Route requests to backend services\n- **Microservices**: Distribute service calls across instances\n- **Message Queues**: Assign message processing to worker nodes\n\n### 💾 Distributed Storage\n- **Object Storage**: Distribute files across storage nodes\n- **Blockchain**: Distribute blockchain data across nodes\n- **P2P Networks**: Route data in peer-to-peer systems\n\n## 🔍 Monitoring\n\nThe library provides comprehensive monitoring capabilities:\n\n```go\n// Get ring statistics\ninfo := ring.GetRingInfo()\nfmt.Printf(\"Physical nodes: %d\\n\", info[\"physical_nodes\"])\nfmt.Printf(\"Virtual nodes: %d\\n\", info[\"virtual_nodes\"])\n\n// Analyze load distribution\nkeys := []string{\"key1\", \"key2\", \"key3\"} // your keys\ndistribution := ring.GetLoadDistribution(keys)\nfor nodeID, count := range distribution {\n    fmt.Printf(\"Node %s: %d keys\\n\", nodeID, count)\n}\n```\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Make your changes\n4. Add tests for new functionality\n5. Run the test suite (`make test`)\n6. Commit your changes (`git commit -m 'Add amazing feature'`)\n7. Push to the branch (`git push origin feature/amazing-feature`)\n8. Open a Pull Request\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## 🙏 Acknowledgments\n\n- Inspired by the original consistent hashing paper by Karger et al.\n- Built with Go's excellent standard library and concurrent primitives\n- Tested extensively for production use cases\n- Performance optimized based on real-world usage patterns","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexnthnz%2Fconsistent-hashing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexnthnz%2Fconsistent-hashing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexnthnz%2Fconsistent-hashing/lists"}