{"id":29100521,"url":"https://github.com/kolesnikova-dev/distributed-rate-limiter","last_synced_at":"2025-12-02T12:05:16.769Z","repository":{"id":280536818,"uuid":"940091741","full_name":"kolesnikova-dev/distributed-rate-limiter","owner":"kolesnikova-dev","description":"This repository contains a Redis-based rate limiter implementing Token Bucket algorithm","archived":false,"fork":false,"pushed_at":"2025-03-06T20:06:59.000Z","size":59,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-10T09:13:16.294Z","etag":null,"topics":["atomicity","docker","go","lua","optimized-performance","rate-limiter","redis"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kolesnikova-dev.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-27T15:48:53.000Z","updated_at":"2025-03-06T20:07:02.000Z","dependencies_parsed_at":null,"dependency_job_id":"a42a0975-5da2-437c-afda-e75272fe42ba","html_url":"https://github.com/kolesnikova-dev/distributed-rate-limiter","commit_stats":null,"previous_names":["kweeuhree/distributed-rate-limiter","kolesnikova-dev/distributed-rate-limiter"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kolesnikova-dev/distributed-rate-limiter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kolesnikova-dev%2Fdistributed-rate-limiter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kolesnikova-dev%2Fdistributed-rate-limiter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kolesnikova-dev%2Fdistributed-rate-limiter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kolesnikova-dev%2Fdistributed-rate-limiter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kolesnikova-dev","download_url":"https://codeload.github.com/kolesnikova-dev/distributed-rate-limiter/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kolesnikova-dev%2Fdistributed-rate-limiter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27452374,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-02T02:00:06.387Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atomicity","docker","go","lua","optimized-performance","rate-limiter","redis"],"created_at":"2025-06-28T18:14:15.703Z","updated_at":"2025-12-02T12:05:16.485Z","avatar_url":"https://github.com/kolesnikova-dev.png","language":"Go","readme":"# 🚦Distributed Rate Limiter\n\nA distributed rate limiter implemented in Go using the Token Bucket algorithm and Redis for storing tokens. This program runs in a Docker container, and is designed to handle bursts of requests efficiently while ensuring atomicity and scalability.\n\n**Third-Party packages**\n\n- The `httprouter` package is used for fast and efficient routing.\n- The `alice` package is used for clear and readable middleware chaining.\n- The `go-redis` is a Redis client for Go.\n- The `toolkit` custom reusable module with logic commonly used in web development, such as logging, JSON handling, and error management.\n\n---\n\n## ⚙️ Features\n\n- **Token Bucket Algorithm:** Handles bursts of requests by allowing a fixed number of tokens to be consumed within a time window.\n\n- **Distributed:** Uses Redis as a centralized store for tokens, making it suitable for distributed systems.\n\n- **Atomic Operations:** Ensures concurrent-safe operations using Lua scripting for Redis.\n\n- **Performance:** The Lua script is loaded and cached on Redis' server, ensuring fast execution.\n\n- **Modular Design:** Code is organized into reusable components for better maintainability.\n\n- **Custom Toolkit:** Reduces code duplication by centralizing logging and utility functions.\n\n---\n\n## 🚦 Rate Limiter Middleware\n\nEach client is identified by their IP address, and Redis key is created for each client:\n\n```go\n\"rate_limit:\u003cclientIP\u003e\"\n```\n\nTokens are stored in Redis with an expiration time. Tokens are replenished based on the elapsed time since the last request:\n\n```lua\nmath.floor(elapsed * max_tokens / expiration_in_micros)\n```\n\nThe Lua script checks if enough tokens are available using `redis.call()`. If tokens are available, the request is allowed, otherwise, it is rejected.\n\n---\n\n## 🔍 Prerequisites\n\n- Go 1.20+\n\n- Redis 6.0+\n\n- Redis connection secrets (Public endpoint: `REDIS_CONN_ADDRESS`, Password to the database: `REDIS_PASSWORD`)\n\n## 🖥️ Run the program\n\n- **Pull the docker image**\n\n```bash\ndocker pull ghcr.io/kweeuhree/distributed-rate-limiter:latest\n```\n\n- **Copy `compose.yaml` from the repository into the folder with the docker image.**\n\n- **Create the Secrets File**\n\nEnsure the `redisSecrets.txt` file exists in the same directory as the compose.yaml, and contains the correct Redis connection endpoint and password.\nYour file should look like so:\n\n```go\nREDIS_CONN_ADDRESS=redis-XXXXX.cXXX.us-centralX-X.gce.redns.redis-cloud.com:XXXXX;REDIS_PASSWORD=1234567890qwerty\n```\n\n- **Run the container:**\n\n```bash\ndocker-compose up\n```\n\nConsole output upon successful launch:\n\n\u003cp align=\"center\"\u003e\n\u003cimg alt=\"Console output upon successful launch\" src=\"assets/screenshots/console.png\" /\u003e\n\u003c/p\u003e\n---\n\n## 📚 Challenges and learning\n\nI started this project wanting to learn concurrency in Go a little better, as well as trying out a distributed solution for a common real-world problem, such as the necessity to limit the amount of requests a single user can make.\n\nAfter finishing the initial Go implementation, I encountered the challenge of ensuring atomicity. Although combining Go transactions and `Exec()` method was a valid and working solution, it was bulky and less efficient. Originally, I resisted Lua scripting, because my primary focus was on learning Go. But at the same time I was committed to leveraging Redis for a distributed solution.\n\nRedis clearly recommends Lua scripting when atomicity is concerned. So the final code is refactored to use a cached Lua script. Redis provides an option to load and cache a script, so it can be run on demand using a SHA string provided by Redis.\n\nThis approach not only ensures atomicity but also improves performance by reducing the number of calls to Redis. Instead of making multiple Redis calls, the program executes a single cached Lua script, which handles all the necessary logic in one atomic operation.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkolesnikova-dev%2Fdistributed-rate-limiter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkolesnikova-dev%2Fdistributed-rate-limiter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkolesnikova-dev%2Fdistributed-rate-limiter/lists"}