{"id":28826092,"url":"https://github.com/olesyastorchakprojects/todo_app","last_synced_at":"2026-04-16T17:34:02.541Z","repository":{"id":298129447,"uuid":"998515303","full_name":"olesyastorchakprojects/todo_app","owner":"olesyastorchakprojects","description":"Production-grade Axum REST API with Sled, OpenTelemetry, rate limiting and benchmarking","archived":false,"fork":false,"pushed_at":"2025-06-17T17:16:45.000Z","size":4013,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-06-17T18:26:29.236Z","etag":null,"topics":["axum","backend","k6","observability","opentelemetry","rate-limiting","rust","sled"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/olesyastorchakprojects.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-08T19:16:29.000Z","updated_at":"2025-06-17T17:16:43.000Z","dependencies_parsed_at":"2025-06-09T15:51:15.159Z","dependency_job_id":null,"html_url":"https://github.com/olesyastorchakprojects/todo_app","commit_stats":null,"previous_names":["olesyastorchakprojects/todo_app"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/olesyastorchakprojects/todo_app","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olesyastorchakprojects%2Ftodo_app","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olesyastorchakprojects%2Ftodo_app/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olesyastorchakprojects%2Ftodo_app/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olesyastorchakprojects%2Ftodo_app/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/olesyastorchakprojects","download_url":"https://codeload.github.com/olesyastorchakprojects/todo_app/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olesyastorchakprojects%2Ftodo_app/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260676916,"owners_count":23045111,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["axum","backend","k6","observability","opentelemetry","rate-limiting","rust","sled"],"created_at":"2025-06-19T03:02:06.670Z","updated_at":"2025-10-25T18:13:38.475Z","avatar_url":"https://github.com/olesyastorchakprojects.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# To-Do Service • Axum | Sled | OpenTelemetry\r\n\r\nA compact, production-minded REST service demonstrating:\r\n\r\n* Clean three-layer architecture: **Storage → Service → HTTP Handlers**.\r\n* **Service** uses **Storage** though traits → concrete implementation of storage can be replaced\r\n* Available **Storage** impl - embedded key-value storage on **Sled** with **Bincode** encoding.\r\n* **JWT authentication**, role-based authorization, server-side sessions.\r\n* **Full OpenTelemetry instrumentation** via the `opentelemetry` crate\r\n  → traces \u0026 metrics exported to **Tempo / Prometheus / Grafana**.\r\n* CI-friendly unit, integration **and** load tests (k6).\r\n* **Docker-first** workflow with hot-reload via `cargo watch`.\r\n\r\n---\r\n# Dockerized workflow\r\n\r\n    # Spin up observability stack (Tempo, Prometheus, Grafana, OTel-Collector, Todo app)\r\n    docker compose -f compose/docker-compose.observability.yml -p observability up --build -d\r\n\r\n    # Spin up todo_app in dev mode: debug build with cargo watch\r\n    docker compose -f compose/docker-compose.dev.yml -p dev up --build -d\r\n\r\n---\r\n\r\n# Content\r\n\r\n1. [Project layout](#1--project-layout)\r\n2. [Http API](#2--http-api-axum-08)\r\n3. [AuthN \u0026 AuthZ](#3--authn--authz)\r\n4. [Storage layer](#4--storage-layer--sled--bincode)\r\n5. [Observability Stack](#5--observability-stack)\r\n6. [Configuration \u0026 Tuning](#6--configuration--tuning)\r\n7. [Testing matrix and coverage](#7-testing-matrix-and-coverage)\r\n8. [Bench tests](#8-bench-tests)\r\n9. [Rate limiting strategy](#9-rate-limiting-strategy)\r\n10. [Roadmap](#10-roadmap)\r\n\r\n## 1  Project Layout\r\n\r\n    crate\r\n    ├── src\r\n        ├── storage/ # storage traits, sled storage impl\r\n        ├── service/ # business rules, password hashing/verification, jwt tokens generation\r\n        ├── handlers/ # thin Axum handlers -\u003e Result\u003c_, StatusCode\u003e\r\n        ├── middleware/ # JWT validation, role gate, rate limiters, update http request metric, create tracing span root\r\n        ├── init/ # functions to initialize tracing/metrics providers, storage\r\n        ├── utils/ # app metrics definitions, root span wrapper, blocking tasks gauge wrapper\r\n    ├── bench/ # k6 scripts + docker compose to run k6 load tests\r\n    ├── compose/ # docker compose files to run observability stack and application for development\r\n    ├── config/ # default.toml • production.toml • …\r\n    ├── tests/ # integration tests\r\n    ├── app.rs # assemble axum router with swagger-ui wrapper\r\n    ├── lib.rs # exposes bare minimum public types to use in main.rs and integration tests (under cfg guard)\r\n    ├── main.rs # entry point, setup tokio runtime, flushes tracing/metrics providers, storage\r\n\r\n\r\n---\r\n\r\n## 2  HTTP API (Axum 0.8)\r\n\r\n| Route                              | Method               | Auth                  | Description                   |\r\n|------------------------------------|----------------------|-----------------------|-------------------------------|\r\n| `/auth/register`                   | POST                 | –                     | Create user                   |\r\n| `/auth/login`                      | POST                 | –                     | Issue Access + Refresh tokens |\r\n| `/auth/refresh`                    | POST                 | **Session / Refresh** | Rotate tokens                 |\r\n| `/auth/logout`                     | POST                 | **User**              | Invalidate session            |\r\n| `/todos`                           | GET / POST / DELETE  | **User**              | List / create / bulk delete   |\r\n| `/todos/{id}`                      | GET / PATCH / DELETE | **User**              | CRUD single To-Do             |\r\n| `/admin/users`                     | GET                  | **Admin**             | List all users                |\r\n| `/admin/user/{id}` / `…/email/{e}` | GET / DELETE         | **Admin**             | Inspect / remove              |\r\n| `/admin/user/{id}/role`            | PATCH                | **Admin**             | Promote / demote              |\r\n| `/health`                          | GET                  | –                     | Liveness-probe                |\r\n\r\nAll handlers are annotated with **`#[utoipa::path]`** → Swagger UI is exposed at `/swagger-ui`\r\n\r\n![](docs/images/Swagger.png)\r\n\r\n---\r\n\r\n## 3  AuthN \u0026 AuthZ\r\n\r\n| Concern        | Implementation |\r\n|----------------|----------------|\r\n| Password hash  | 2 possible impl: **`argon2`** and **`ring::pbkdf2`** |\r\n| Tokens         | **JWT** HS256. 10 min Access, 10 days Refresh (configurable) |\r\n| Sessions       | Server side session, TTL configurable |\r\n| Roles          | `Role::{User, Admin}` checked by `require_role` middleware. |\r\n| Error handling | Invalid/Expired token → `401`; forbidden role → `403`. |\r\n\r\nBy default argon2 is used with these parameters:\r\n```toml\r\n# excerpt default.toml\r\n[auth.argon2]\r\nmemory_cost = 32768\r\ntime_cost = 2\r\nparallelism = 1\r\n```\r\nGiven parameters were selected based on bench tests run under 30 rps registration load, targeting SLO of 250 ms per hash.\r\nWhy 30 rps? 4 hashes per second (argon2 utilizes 1 thread regardless of parallelism parameter) * 8 cpu cores = 32. Rounded to 30.\r\n\r\nk6 test details: 1 minutes warmup with 30 rps and 2 minutes bench with 30 rps load.\r\n\r\nResults on Intel(R) Core(TM) i7-6820HK CPU * 32 GB RAM (results in each row are averages from 5 runs):\r\n\r\n| `m_cost` (MiB) | `t_cost` | p50 (ms) | p90 (ms) | p95 (ms) | p99 (ms) | % \u003e 250 ms  | max run queue len             |\r\n|----------------|----------|----------|----------|----------|----------|-------------|-------------------------------|\r\n| 19             | 1        | 19       |   34     |  43      |    59    |     0       | 0                             |\r\n| 19             | 2        | 31       |   47     |  49      |    69    |     0       | 1                             |\r\n| 19             | 3        | 41       |   65     |  72      |    121   |     0.07    | 6                             |\r\n| 19             | 4        | 66       |   130    |  214     |    444   |     12      | 5                             |\r\n| 32             | 1        | 29       |   47     |  49      |    69    |     0       | 5                             |\r\n| 32             | 2        | 59       |   111    |  144     |    263   |     2.4     | 5 (average from 8, 6, 1, 8, 1)|\r\n| 32             | 3        | 105      |   300    |  360     |    550   |     28      | 35                            |\r\n\r\nCompared to weaker settings (`t_cost=1`, `m_cost=19 MiB`), selected configuration (`t_cost=2`, `m_cost = 32 MiB`):\r\n- Improves defense against GPU/ASIC attacks by increasing memory usage\r\n- Preserves latency guarantees (`p95 ≈ 144 ms`, `p99 ≈ 263 ms`)\r\n- Maintains stable load: only ~2.4% of hashes exceed 250 ms\r\n- Keeps blocking task queue shallow (`max queue = 5`)\r\n- Avoids unpredictable latency spikes seen at `t=3` (`p99 \u003e 500 ms`, queue up to 35)\r\n\r\nThis makes it a balanced default for production scenarios where user-facing latency is critical but password hashes must resist offline brute-force attempts.\r\n![](docs/images/argon2_hashing.png)\r\n\r\nResults for pbkdf2 for **10 rps** load:\r\n| `iterations`   | p50 (ms) | p90 (ms) | p95 (ms) | p99 (ms) | % \u003e 250 ms  |\r\n|----------------|----------|----------|----------|----------|-------------|\r\n| 310000         |  180     | 244      |   302    |  456     |    11.1     |\r\n\r\n![](docs/images/pbkdf2_hashing.png)\r\n\r\n---\r\n\r\n## 4  Storage Layer — `sled` + `bincode`\r\n\r\n| Aspect                  | Choice | Motivation |\r\n|-------------------------|--------|------------|\r\n| Engine                  | **`sled` 0.34** | Zero-config, embedded LSM tree; crash-safe; single-binary deployment (no external DB for PoC / edge nodes). |\r\n| Serialization           | **`bincode` 2** | Compact (\u003c 1 B overhead per value); zero-alloc; Serde-driven. |\r\n| Key scheme              | `\"\u003cprefix\u003e:\u003cuuid\u003e\"` | Prefix keeps related keys adjacently on disk → fast range scans for pagination. |\r\n| Separation of stored entities |  Dedicated `user`/`todo`/`session` trees| Storage load spread |\r\n| Durability              | `sled::transaction` + explicit `flush()` on graceful shutdown. | Prevent loosing any data |\r\n| Implementation dependency isolation| Upper `service` layer uses storage via UserStorage/TodoStorage/SessionStorage traits | Easy to change storage impl from `sled` to for ex. `Postgres`\r\n\r\n**Only methods with transaction are wrapped into `spawn_blocking`**\r\n\r\nk6 benchmarks (1 k rps mixed CRUD) show _P99 ≤ 10 ms_ for single sled call; therefore synchronous I/O stays inside latency SLO and avoids thread-pool context switches.\r\n\r\np99 from 5 minute full bench run (registration + login + crud) for sled single call methods:\r\n![](docs/images/full_storage_p99_5_min_run.png)\r\n\r\np99 from 5 minute full bench run (registration + login + crud) for methods with transactions:\r\n![](docs/images/full_storage_p99_transactions_5_min_run.png)\r\n\r\n---\r\n\r\n## 5  Observability Stack\r\n\r\n| Component   | Ports | Purpose |\r\n|-------------|-------|---------|\r\n| **otel-collector** | 4317 | Fan-out OTLP → Tempo (traces) + Prometheus (metrics). |\r\n| **tempo**          | 3200/4317 | Trace backend; compaction retention 1 h. |\r\n| **prometheus**     | 9090      | Metric storage \u0026 alert rules. |\r\n| **grafana**        | 3001      | Dashboards |\r\n\r\n### Tracing configuration\r\n\r\n* Sampling rate: `telemetry.tracing_sampling_rate` - default `0.1`.\r\n* **Spans containing an error are _always_ exported** — middleware sets `sampling.priority = 1`.\r\n* High-cardinality labels (`enduser_id`, `todo_id`, `session_id`, `email` etc) live only on the **root span**; Tempo indexes these labels explicitly to keep search \u003c 100 ms.\r\n\r\n```yaml\r\n# excerpt tempo.yaml\r\nstorage:\r\n  trace:\r\n    backend: local\r\n    local:\r\n      path: /tmp/tempo/blocks\r\n    wal:\r\n      path: /tmp/tempo/wal\r\n    block:\r\n      version: vParquet4\r\n      parquet_dedicated_columns:\r\n        - name: enduser_id\r\n          type: string\r\n          scope: span\r\n        - name: enduser_email\r\n          type: string\r\n          scope: span\r\n        - name: target_user_id\r\n          type: string\r\n          scope: span\r\n```\r\nExample of trace\r\n\r\n![](docs/images/trace.png)\r\n\r\nThere are 4 exported to json dashboards in `/compose/grafana/dashboards` folder which automatically loaded in Grafana when observability stack starts:\r\n1. auth # panels to track performance of password hashing/verifying\r\n2. todo_app # panels to track overall performance of the application with breakdown into handlers/service/storage p_50/p_90/p_99 metrics.\r\n3. k6_bench # metrics exported by k6 during bench testing. It has `phase` variable to be able to separate `warmup` and `bench` phases performance.\r\n4. windows_node_exporter # to track Windows system performance\r\n\r\nauth dashboard\r\n![](docs/images/auth.png)\r\n\r\ntodo_app dashboard\r\n![](docs/images/todo_app.png)\r\n\r\nwindows_node_exporter dashboard\r\n![](docs/images/windows_node_exporter.png)\r\n\r\nk6_bench\r\n![](docs/images/k6_bench.png)\r\n\r\n---\r\n\r\n## 6  Configuration \u0026 Tuning\r\n\r\nAll settings live in `config/*.toml` and can be overridden with `APP__…` environment variables (double underscores map to dots).\r\n\r\n| Section       | Default              | Purpose |\r\n|---------------|----------------------|---------|\r\n| `storage`     | `sled`               | selection of storage implementation and impl parameters|\r\n| `jwt`         | `10min/10days/30days`| JWT access/refresh-token/session TTLs |\r\n| `telemetry`   | -                    | Enables tracing/metrics/stdout_tracing; tracing/metrics endpoints; tracing sampling rate |\r\n| `server`      | `0.0.0.0:3400`       | Application server address |\r\n| `auth`        | `argon2` - default   | Selection of kdf algo (argon2 or pbkdf2); parameters of kdf algo; credentials of admins |\r\n| `rate_limiter`| -                    | limits for endpoints/group of endpoints of 2 kind: global and per_ip |\r\n\r\n```toml\r\n# config/default.toml  (excerpt)\r\n[storage]\r\nbackend = \"sled\"\r\n\r\n[storage.sled]\r\npath = \"/app/sled_data\"\r\n# in delete_all we delete items in batches\r\ndelete_batch_size = 100\r\n\r\n[jwt]\r\n# 10 min\r\naccess_token_ttl_sec = 600\r\n# 10 days\r\nrefresh_token_ttl_sec = 864000\r\n# 30 days\r\nsession_ttl_sec = 2592000\r\n\r\n[telemetry]\r\ntracing_endpoint = \"http://otel-collector:4317\"\r\nmetrics_endpoint = \"http://otel-collector:4317\"\r\n```\r\n\r\n---\r\n\r\n## 7 Testing matrix and coverage\r\n\r\n| Tier                                     | \tRunner / Tooling | Location\r\n|-----------------------------------------|------|----------|\r\n|`Unit`                                   |cargo test | `#[cfg(test)]` modules |\r\n|`Integration`                            |Tokio + Axum|`tests/*.rs` (spins full app)|\r\n|`Load`                                   |k6 + Docker|`bench/scripts/`|\r\n\r\nCode coverage with existing tests\r\n\r\n![](docs/images/coverage.png)\r\n\r\n---\r\n\r\n## 8 Bench tests\r\n\r\nTested on Intel(R) Core(TM) i7-6820HK CPU * 32 GB RAM machine.\r\n\r\nbench folder layout\r\n\r\n    crate\r\n    ├── bench\r\n      ├── bench_scripts/ # k6 test scripts\r\n      ├── gentokens/     # tiny rust project to generate jwt tokens\r\n      ├── build.sh       # script to build docker containers for bench testing\r\n      ├── run.sh         # main script to run bench tests\r\n      ├── docker-compose.bench.yml # containers for bench testing: gentokens, todo_app in release build, k6\r\n      ├── Dockerfile_bench # dockerfile to build todo_app in release mode\r\n      ├── Dockerfile_gentokens # dockerfile to build gentokens\r\n\r\ntests:\r\n* `registration.js` - load tests for `/auth/register` endpoint\r\n* `login.js` - load tests for `/auth/login` endpoint. These tests use tokens generated by gentokens\r\n* `promote_user.js` - load tests for `/admin/user/{id}/role` endpoint. These tests use user login data generated by gentokens\r\n* `crud.js` - load tests for `/todos` endpoints. These tests use tokens generated by gentokens\r\n   CRUD scenarios:\r\n    - create todo\r\n    - delete todo (get users todos, delete 1st one; if user doesn't have todos yet, create one and delete it)\r\n    - update todo (get users todos, update 1st one; if user doesn't have todos yet, create one and update it)\r\n    - get all user todos\r\n    - delete all user todos\r\n* `full.js` - all scenarios together: registration, login, promote, crud.\r\n\r\nEach test includes a warm-up phase followed by a benchmark phase to ensure stable metrics.\r\n\r\n# Bench workflow:\r\n\r\n      # run scripts from bench folder\r\n      cd bench\r\n\r\n      # build containers: todo_app in release mode, gentokens app, k6\r\n      ./build.sh\r\n\r\n      # run bench test for registration\r\n      # --kdf=argon2 - password hashing function - argon2\r\n      # --ips=20 - test script will send requests from 20 random client ips\r\n      ./run.sh --bench_test=registration --kdf=argon2 --ips=20\r\n\r\n      # run bench test for registration\r\n      # --kdf=pbkdf2 - password hashing function - pbkdf2\r\n      # --iterations=310000 - iterations to run hash function by pbkdf2\r\n      # --ips=20 - test script will send requests from 20 random client ips\r\n      ./run.sh --bench_test=registration --kdf=pbkdf2 --iterations=310000 --ips=20\r\n\r\n      # run bench test for login\r\n      # --kdf=argon2 - password hashing function - argon2\r\n      # --ips=20 - test script will send requests from 20 random client ips\r\n      # --tokens=50 - gentokens app will register 50 users and save user data (email, password, token) into known file\r\n      ./run.sh --bench_test=login --kdf=argon2 --tokens=50 --ips=20\r\n\r\n      # run bench test for crud\r\n      # --kdf=argon2 - password hashing function - argon2\r\n      # --ips=100 - test script will send requests from 100 random client ips\r\n      # --tokens=300 - gentokens app will register 300 users and save user data (email, password, token) into known file\r\n      ./run.sh --bench_test=crud --kdf=argon2 --tokens=300 --ips=100\r\n\r\n      # run full bench test\r\n      # --kdf=argon2 - password hashing function - argon2\r\n      # --ips=50 - test script will send requests from 50 random client ips\r\n      # --tokens=300 - gentokens app will register 300 users and save user data (email, password, token) into known file\r\n      ./run.sh --bench_test=full --kdf=argon2 --tokens=300 --ips=50\r\n\r\nrun.sh parameters:\r\n  - --app_config # config file to use, `default - bench`\r\n  - --bench_test # one of: registration, login, crud, full, promote\r\n  - --kdf # password hashing function: argon2 or pbkdf2 (pbkdf2 needs also --iterations parameter, default is 1000)\r\n  - --iterations # parameter to tune pbkdf2 password hashing function\r\n  - --ips # number of IP addresses script will send requests from. This is needed to prevent rate limeter from rejection requests based on per_ip limits. This parameter also needs config option `rate_limiter.x_forwarded_for = true`. Which is set by default in `bench.toml` config file. Script sets random ip in `x_forwarded_for` header, and app reads client ip from this header (enabled only in bench.toml with rate_limiter.x_forwarded_for = true). This is done only for bench testing to simulate load from different clients.\r\n  - --tokens # number of tokens to pre-generate before running tests. `gentokens` app writes token into `/tokens/tokens.json` file. Test scripts read tokens from this file.\r\n\r\nBench testing emits tracing/metrics. By default tracing sampling rate is 0.1. Increasing it to more substential value will impact performance.\r\nThere is additional dashboard for k6 metrics. It shows picture of app performance from k6 perspective. Because scenarios in tests are marked with tags (`warmup`/`bench`) it's possible to see performance for different phases. Dashboard has `phase` variable.\r\n\r\n![](docs/images/k6.png)\r\n\r\n---\r\n\r\n## 9. Rate limiting strategy\r\n\r\nThe application implements a fail-fast rate limiter powered by the `governor` crate. It works at two levels:\r\n- Global limiters restrict the total request rate for each endpoint group\r\n- Per-IP limiters enforce fairness and isolate noisy clients\r\n\r\nAll excess requests are rejected immediately, with no buffering or queuing, returning HTTP `429 Too Many Requests`.\r\n\r\n### How the limits were calculated:\r\n\r\nRate limits are based on measured performance from isolated k6 benchmarks:\r\n1. Each endpoint group was tested separately to determine its maximum sustainable p99 latency.\r\n    - registration and login were capped at 250ms p99, stable up to 30 rps combined\r\n    - crud operations showed 5ms p99 up to ~400 rps (without concurrent load)\r\n\r\n2. CPU budget planning:\r\n    - Target: ≤ 80% system CPU usage\r\n    - Argon2 operations at 18 rps (login + registration) consume ≈ 4.5 cores\r\n    - Remaining ~1.9 cores allocated to crud and admin → yields ≈ 380 rps for writes/reads\r\n\r\n3. Under real-world mixed load, storage p99 spiked to 60–75 ms at times. In response, crud limits were split into light vs heavy categories to better protect latency-sensitive operations.\r\n\r\n\r\n### Light vs Heavy CRUD separation\r\n\r\n  |Type\t|Endpoints\t|Global Limit\t|Per-IP Limit|\r\n  |-----|-----------|-------------|------------|\r\n  |Light CRUD|\tget_todo, get_all_todos, delete_todo|\t320 rps|\t15 rps|\r\n  |Heavy CRUD|\tupdate_todo, delete_all_todos\t|60 rps\t|10 rps|\r\n\r\n  This design protects the blocking thread pool and sled engine from overload, while allowing light, frequent operations to proceed smoothly.\r\n\r\n### Benchmark strategy\r\n\r\nBenchmarks are intentionally run with 10–15% higher RPS than the configured limits, simulating:\r\n  - real-world overuse by aggressive clients\r\n  - behavior behind a load balancer or edge throttle\r\n  - validation of graceful failure (fail-fast)\r\n\r\nAll 429 rejections are observable in metrics and traces (e.g. Prometheus, Tempo), providing full visibility into overload conditions.\r\n\r\n### SLO Compliance\r\n\r\nA 20-minute benchmark with full mixed load confirmed that the system meets its latency SLO target:\r\n  - 99th percentile of storage operations under 5 ms (SLO compliance consistently ≥ 98.8%)\r\n  - 99th percentile of password hashing/verification under 250 ms\r\n  - All overload conditions are handled via fail-fast limiting\r\n\r\nThis confirms that the system behaves predictably under stress, and the rate limits isolate heavy workloads without degrading core latency.\r\n\r\nPanel with p99 for storage operations and SLO \u003c 5ms line.\r\n![](docs/images/SLO_5ms.png)\r\n\r\nPanels with SLO \u003c 250ms lines for hashing and verifying password.\r\n![](docs/images/SLO_250ms.png)\r\n\r\nPanel with max run queue length.\r\n![](docs/images/run_queue.png)\r\n\r\nk6 output from running mixed load:\r\n```\r\nk6  |      ✗ delete_all_todos\r\nk6  |       ↳  99% — ✓ 39380 / ✗ 221\r\nk6  |      ✗ get_todo\r\nk6  |       ↳  96% — ✓ 96554 / ✗ 3048\r\nk6  |      ✗ created\r\nk6  |       ↳  83% — ✓ 132950 / ✗ 25844\r\nk6  |      ✗ fallback deleted\r\nk6  |       ↳  59% — ✓ 22662 / ✗ 15445\r\nk6  |      ✗ register 201 or 409\r\nk6  |       ↳  87% — ✓ 10539 / ✗ 1462\r\nk6  |      ✗ updated\r\nk6  |       ↳  68% — ✓ 15173 / ✗ 7013\r\nk6  |      ✗ get_todos\r\nk6  |       ↳  99% — ✓ 59945 / ✗ 222\r\nk6  |      ✓ promote_user\r\nk6  |      ✗ deleted\r\nk6  |       ↳  98% — ✓ 35007 / ✗ 569\r\nk6  |      ✗ login 200\r\nk6  |       ↳  99% — ✓ 11955 / ✗ 46\r\nk6  |      ✗ login token exists\r\nk6  |       ↳  99% — ✓ 11955 / ✗ 46\r\nk6  |\r\nk6  |      █ setup\r\nk6  |\r\nk6  |        ✓ login 200\r\nk6  |        ✓ login token exists\r\nk6  |\r\nk6  |      checks.....................: 89.26% ✓ 448123     ✗ 53916\r\nk6  |      data_received..............: 86 MB  65 kB/s\r\nk6  |      data_sent..................: 232 MB 176 kB/s\r\nk6  |      http_req_blocked...........: avg=7.06µs  min=1.1µs      max=67.24ms  p(90)=7.8µs   p(95)=12.1µs   p(99)=23.8µs\r\nk6  |      http_req_connecting........: avg=1.8µs   min=0s         max=67.15ms  p(90)=0s      p(95)=0s       p(99)=0s\r\nk6  |      http_req_duration..........: avg=4.27ms  min=-933549ns  max=397.86ms p(90)=2.77ms  p(95)=16.43ms  p(99)=67.18ms\r\nk6  |      http_req_failed............: 10.99% ✓ 53870      ✗ 436167\r\nk6  |      http_req_receiving.........: avg=81.49µs min=8.7µs      max=28.73ms  p(90)=127.5µs p(95)=153.81µs p(99)=254.51µs\r\nk6  |      http_req_sending...........: avg=38.56µs min=-1909708ns max=33.17ms  p(90)=67.4µs  p(95)=83.1µs   p(99)=143µs\r\nk6  |      http_req_tls_handshaking...: avg=0s      min=0s         max=0s       p(90)=0s      p(95)=0s       p(99)=0s\r\nk6  |      http_req_waiting...........: avg=4.15ms  min=0s         max=396.54ms p(90)=2.6ms   p(95)=16.17ms  p(99)=67.04ms\r\nk6  |      http_reqs..................: 490037 371.224953/s\r\nk6  |      iteration_duration.........: avg=6.23ms  min=348.12µs   max=398.44ms p(90)=6.44ms  p(95)=48.98ms  p(99)=73.89ms\r\nk6  |      iterations.................: 355209 269.086711/s\r\nk6  |      vus........................: 1      min=0        max=9\r\nk6  |      vus_max....................: 410    min=410      max=410\r\nk6  |\r\nk6  |\r\nk6  | running (22m00.1s), 0000/0410 VUs, 355209 complete and 0 interrupted iterations\r\nk6  | warm_up                   ✓ [ 100% ] 000/100 VUs  2m0s   500.00 iters/s\r\nk6  | warm_up_delete_all_todos  ✓ [ 100% ] 000/030 VUs  2m0s   30.00 iters/s\r\nk6  | warm_up_reg               ✓ [ 100% ] 000/010 VUs  2m0s   20.00 iters/s\r\nk6  | warm_up_update_todo       ✓ [ 100% ] 000/030 VUs  2m0s   30.00 iters/s\r\nk6  | create_todo               ✓ [ 100% ] 000/050 VUs  20m0s  50.00 iters/s\r\nk6  | delete_all_todos          ✓ [ 100% ] 000/030 VUs  20m0s  30.00 iters/s\r\nk6  | delete_todo_with_fallback ✓ [ 100% ] 000/050 VUs  20m0s  50.00 iters/s\r\nk6  | get_all_todos             ✓ [ 100% ] 000/050 VUs  20m0s  50.00 iters/s\r\nk6  | login                     ✓ [ 100% ] 000/010 VUs  20m0s  10.00 iters/s\r\nk6  | promote_user              ✓ [ 100% ] 000/010 VUs  20m0s  10.00 iters/s\r\nk6  | registration              ✓ [ 100% ] 000/010 VUs  20m0s  8.00 iters/s\r\nk6  | update_todo               ✓ [ 100% ] 000/030 VUs  20m0s  30.00 iters/s\r\n\r\n```\r\nPanels with global and per_ip rate limiter rejections and stable rps\r\n![](docs/images/rate_limiter_rejection.png)\r\n\r\n---\r\n\r\n## 10. Roadmap\r\n\r\n    1. Postgres adapter implementing the same TodoStorage / UserStorage / SessionStorage traits.\r\n\r\n    2. CI: fmt + clippy -D warnings + cargo audit + test → Docker Buildx → GHCR.\r\n\r\n    3. Signed release binaries; JWT secret pulled from Vault on start-up.\r\n\r\n    4. Additional Grafana dashboards (capacity, WAL growth) and Alertmanager rules.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folesyastorchakprojects%2Ftodo_app","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Folesyastorchakprojects%2Ftodo_app","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folesyastorchakprojects%2Ftodo_app/lists"}