{"id":27603209,"url":"https://github.com/evalece/bench-the-query","last_synced_at":"2026-04-19T05:33:12.983Z","repository":{"id":285177653,"uuid":"957287608","full_name":"evalece/bench-the-query","owner":"evalece","description":"Benchmarking GraphQL vs various REST API tool in a container setting for minimal noise in response time analysis, while simulating cold start for all queries on database. Both API endpoint and database benchmark are recored as primary source of response time analysis. ","archived":false,"fork":false,"pushed_at":"2025-04-20T23:30:04.000Z","size":7958,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-22T19:13:52.389Z","etag":null,"topics":["benchmark","containerization","graphql","rest-api"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/evalece.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-30T01:40:19.000Z","updated_at":"2025-04-20T23:30:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"a551f703-8710-4550-8d7c-729b7b672ca7","html_url":"https://github.com/evalece/bench-the-query","commit_stats":null,"previous_names":["evalece/bench-the-query"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evalece%2Fbench-the-query","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evalece%2Fbench-the-query/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evalece%2Fbench-the-query/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evalece%2Fbench-the-query/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/evalece","download_url":"https://codeload.github.com/evalece/bench-the-query/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250306640,"owners_count":21408926,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","containerization","graphql","rest-api"],"created_at":"2025-04-22T19:13:55.742Z","updated_at":"2025-10-18T06:58:55.962Z","avatar_url":"https://github.com/evalece.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Bench-the-query\n![Version](https://img.shields.io/badge/version-v1.0.0-blue)\n![License](https://img.shields.io/github/license/evalece/bench-the-query)\n![Last Commit](https://img.shields.io/github/last-commit/evalece/bench-the-query)\n![Dockerized](https://img.shields.io/badge/Dockerized-yes-blue?logo=docker)\n![Tech Stack](https://img.shields.io/badge/Made%20With-K6%20%7C%20Redis%20%7C%20FastAPI%20%7C%20GraphQL%20%7C%20Grafana-informational)\n\n\n![schematic](in-dev-screen/final_result.png)\n\n## What is Bench the Query?\n\n ![schematic](proxy_pipeline.jpeg) \n- Bench the Query helps developers to choose client servers based on traffic load and context size.\n- The containerized Benchmark tool are light-weighted and can be tested seperately.\n- A versatile tool to benchmark various client servers with minimal code. \n- Automates deployment for database, query protocols and load test in one docker compile. \n- Furthermore, benchmark results are made accessible in a pre-configured Grafana Dashboard. \n- Users use this tool by entering a list of payload size used in querying and setting desire loads in load test the same way as K6 load test. \n\n##  Installation and Setup \n\n[Bench the Query Setup Instructions](setup.md)\n\n\n## Benchmark Options\n\n#### Datastore Options  \n\n| Name | Description | Type | Setup |  Default value |\n| -------------- | --------------------  | ------------------ | ---- | ------ |\n| `STRING_SIZES` | String count dataset; \u003cbr\u003eloaded to DB at compose up | env var, deploy before \u003cbr\u003e pipeline runtime | ./env | 3,5,10,\u003cbr\u003e,15,30,50\u003cbr\u003e,75,100,500,\u003cbr\u003e750,1000,1500,\u003cbr\u003e2000 |\n| `NUM_USER` | Number of users to be loaded in DB | env var, deploy before pipeline runtime  | ./env | `10` |\n\n\n#### Load Test Options \n\n Bench the Query currently supports cold cache DB + GET request on load test  via K6. To utilize automated pre-configured Grafana uptream live monitoring dashboard it is encouraged the user follows the recoomended options:\n  - Enabling parallel and sequntial load testing with virtual user (VUs) as thread to simulate users.\n  - Customize a sheduled traffic surge with K6's format [1]. \n  - Stick with K6 load test metrics at reference [2] for efficient upstream processing.\n\n- Reference \u003cbr\u003e\n  [1]https://grafana.com/docs/k6/latest/set-up/set-up-distributed-k6/usage/scheduling-tests/ \u003cbr\u003e\n  [2]https://grafana.com/docs/k6/latest/using-k6/execution-context-variables/ \u003cbr\u003e\n\n| Name | Description | Type | Setup |  Default value |\n| :-------------- | :----------------  | ----------- | ------- | ----------------------- |\n| `vus` | K6 virtual users by \u003cbr\u003e host machine threads | java script \u003cbr\u003e, deploy at pipeline runtime | ./load_test/scripts/k6_options.js| `20` |\n| `iterations` | K6 VUs' task iterations |java script, deploy at pipeline runtime | ./load_test/scripts/k6_options.js| `100` |\n\n#### Toxy Proxy by Shopify Options  \n  - Toxy Proxy by Shopfiy is injected between client server and RedisDB to similate rate limiting, RTT, traffic surges and other unforeseen challenges. \n  - The goal is to examine client server strategy and adpatation under a degaradated system. \n  - TCP pakcets and traffics are simulated by K6.\n- Reference \u003cbr\u003e\n [1]https://github.com/Shopify/toxiproxy?tab=readme-ov-file#http-api \u003cbr\u003e\n\n| Name | Description | Type | Remark for simulation |  Default value |\n| :-------------- | :----------------  | :-------------- | :----------------  | :-------------- |\n| `latency` | A delay to all data going through the proxy in ms. | HTTP field | RTT simulation; can apply seperately to up and downstream | `0` |\n| `jitter` | The delay is equal to latency +/- jitter ms. | HTTP field | set 0 |`0` |\n| `limit_data` |Closes connection if data transmitted exceeded limit.  | HTTP field | Rate limiting | `0` |\n| `rate` | Maximum connection rate on Proxy in KB/s. | HTTP field | -- | `0` |\n| `slicer` | Slice to fit specific packet size in Bytes, allowing latency addition between packet on silicer attribute| HTTP field | -- | `0` |\n\n\n#### Bench the Query under the hood\n\nA containerized benchmarking suite to compare GraphQL and REST API (FastAPI for now, more later) performance under simulated load using K6 while enabling realtime tracking in Grafana.\n  If -\u003e desinates dataflow, we have: \n  Lua script -\u003e Redis \u003c-\u003e GraphQL \u0026 FastAPI \u003c-\u003e K6 -\u003e InfluxDB-\u003e Grafana. \n\n#### What Bench the Query is not?\n- While more client servers are to be added more a more inclusive benchmark test, the client protocols and security checks are not made to be serve production codes. \n\n\n## Quick Start \nGit clone the project\n### Step 1 - Place Payload Size Variabilities \n```bash \ngit clone https://github.com/evalece/bench-the-query.git \n```\n\nIf needed, adjust .env for desired payload lenghth. These are contents in string sizes and will be requested by K6 load test user via GET method. You can also set a desired number of users holding these values in Redis. If not, we will use defualt: \n\n- In case Redis Stack fail to generate datasets automatically, open the container terminal and at entrypoint, enter the following:\n\n\n[Redis Docker Image Entry Point](./redis/entrypoint.sh)\n\n\n```bash\nexport json_ss=\"[$STRING_SIZES]\"\nredis-cli --eval /0.lua , \"$NUM_USER\" \"$json_ss\"\n```\n\n```bash\nredis-cli --eval 0.lua user:1 user:2 user:3\nredis-cli --eval 1.lua user:1 user:2 user:3\nredis-cli --eval 2.lua user:1 user:2 user:3\n```\n\n[Datastore and Client-server schema Environment Variable](./env)\n\n```bash\n# .env, otherwise set by user during docker compose up \nSTRING_SIZES=3,5,10,15,30,50,75,100,500,750,1000,1500,2000\nNUM_USER=10\n\n```\n\n### Step 2 - Stress Client Servers with Toxic Proxy, TCP Backpressure, High RTT and MORE!!!\n\nCustomize your K6 load test and Toxy Proxy [11] to control traffic flow and latencies to further test client server's handling ability. see [11] repo should you need to customize toxics for Toxy Proxy.\n\n- To do so, we will first compose up Toxy Proxy:\n```bash \ndocker-compose -f docker-compose.infra.yml up\n```\n- Next, at a seperate termianl, pass the following to route listenig ports with upcoming client server ports \n```bash \ncurl -X POST http://localhost:6380/proxies -d '{\n  \"name\": \"grphQL_redis_proxy\",\n  \"listen\": \"0.0.0.0:4000\",\n  \"upstream\": \"redis:6379\"\n}'\n```\n\n```bash \ncurl -X POST http://localhost:6380/proxies -d '{\n  \"name\": \"fastAPI_redis_proxy\",\n  \"listen\": \"0.0.0.0:8000\",\n  \"upstream\": \"redis:6379\"\n}'\n```\n\n[K6 customizable Java Script ](load_test/scripts/k6_options.js)\n\n(For best use of pre-config dashboard, do not delete tags or rename them). \n\n\n### Step 3 - Boot up Client Server, Load tests and Grafana Dashboard(Last step)\nTo comppose:\n\n- Client servers: Docker-compose.client.yml up\n- Load test with Grafana: Docker-compose.load.yml up\n\n```bash \ndocker-compose -f docker-compose.client.yml up\ndocker-compose -f docker-compose.load.yml up\n\n```\nAnd accessing results later by loading pre-config dashboard at \n\n```bash\n load_test/dashboard/bechmark_dashboard.json\n```\nResult at:\n```bash\nhttp://localhost:3000\n```\n\n### (Optional Step) Accessing Results\nFor accessing the results the first time in Grafana, use the following credential:\n\nLoad Live Grafana Dashboard (if using dashboard 2587 with K6 default metrics) \n![schematic](dashboard.png)\n\n```bash\n http://localhost:3000\n\n User:admin\n Passward:admin\n ```\n\nAlternatively, see instructions at the following if you would like to setup your own\n[InfluxDB Implementation](influxDB/readme.md)\n\n## More?\n\nsee   [Technical Blogs](technical_blogs/README.md)  for more development story!\n\n## Reference\n 1. Dataset: https://www.kaggle.com/datasets/rohanadagouda/cleaned-dataset \n 2. Batch Redis data loading using Riot: https://redis.github.io/riot/#_importing \n 3. Docker compose dependency and other options: https://docs.docker.com/compose/how-tos/startup-order/  \n 4. Redis Pipeline: https://redis.io/docs/latest/develop/use/pipelining/ Our case: Allow a toggle on/off to observe RTT effect\n 5. Redis SSL/TLS: https://redis.io/docs/latest/operate/rs/security/encryption/tls/enable-tls/ (Unable to test for now due to Enterprise)\n 6. A. D. Birrell, B. J. Nelson, and Xerox Palo Alto Research Center, “Implementing remote procedure calls,” Feb. 1984. [Online].   Available: http://birrell.org/andrew/papers/ImplementingRPC.pdf\n 7. GraphQL as API endpoint https://graphql.org/faq/best-practices/ \n 8. For REST; Java+Spring- https://redis.io/learn/develop/java/redis-and-spring-course/lesson_2 \n 9. Redis Dataset: https://github.com/redis-developer/redis-datasets \n 10. riot https://redis.github.io/riot/#_datagen_struct and fake data gen : https://www.datafaker.net/ \n 11. Toxy Proxy by Shopify https://github.com/Shopify/toxiproxy \n\n\n\n\n\n###  License\n\nAuthor: Eva Liu; lj2liu@uwaterloo.ca\nInitially for job seeking + reviewing \u0026 learning DB \u0026 distributed sys purpose\nChatGPT assistance in:\n 1. Grammar and proof read of document writing \n 2. Readability of ReadMe and comments. \n 3. Paraphrasing 80% first version of readme after techincal discussion, 0% on README across all since 3rd push \n\n\n\n\nCC BY-NC 4.0\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevalece%2Fbench-the-query","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevalece%2Fbench-the-query","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevalece%2Fbench-the-query/lists"}