https://github.com/fiam/testing-federation
https://github.com/fiam/testing-federation
Last synced: 3 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/fiam/testing-federation
- Owner: fiam
- License: mit
- Created: 2023-09-20T16:28:55.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2023-09-20T17:58:10.000Z (over 1 year ago)
- Last Synced: 2025-01-11T17:30:35.440Z (5 months ago)
- Language: TypeScript
- Size: 1.71 MB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# GraphQL Gateways Benchmark
This repository is a collection is different scenarios and tests performed on different implementations of GraphQL gateways.
This goals of this repo:
1. **Provide a transparent, accurate and descriptive benchmark testing for different tools:**
➡️ All tests are running in Docker containers, with the same configuration.
➡️ During the test, tools like [cadvisor](https://github.com/google/cadvisor) are monitoring stats.
➡️ Promethues and Grafana are stores results and stats for every run, and we export and embed it as PNG.
➡️ Every scenario have a different report, based on what it measures.
➡️ We are using a dedicated GitHub Actions runners, with `concurrency=1` to make sure tests are running as standalone.
2. **Always be up to date:**
➡️ This allows tools to improve over time.
➡️ All code in the repository is open and can be changed by the community.
➡️ Scenarios are executed for every PR, and reports are generated automatically per scenario.
➡️ [Renovate](https://github.com/renovatebot/renovate) keeps all dependencies up-to-date.
3. **Various scenarios**
➡️ We are trying to create real-life scenarios based on [our experience and our customers](the-guild.dev).
➡️ Each scenario has differnt setup and measure different stats.
# Scenarios
## `fed-v1-constant-vus-over-time`
[Latest Results](./federation-v1/scenarios/constant-vus-over-time/README.md)
This scenario runs the following:
1. 4 GraphQL subgraphs in dedicated services
2. A GraphQL gateway compatible with the Apollo Federation v1 spec
3. Constant rate of VUs over fixed time span
This measures the following:
1. RPS (requests per second) rate
2. Request duration (average, p95)
3. Request failures (count)
4. CPU usage during the entire execution
5. RAM usage during the entire execution
6. HTTP layer timings
> This scenario uses Federation v1 spec with all gateways that supports this kind of specification (not all gateways supports v2 spec).
## `fed-v1-constant-vus-subgraphs-delay`
[Latest Results](./federation-v1/scenarios/constant-vus-subgraphs-delay/README.md)
This scenario runs the same flow as `fed-v1-constant-vus-over-time` but with an intentional delay on each upstream Subgraph. This creates more stress and increases memory in the server due to the more inflight requests.
## `fed-v1-constant-vus-subgraphs-delay-resources`
[Latest Results](./federation-v1/scenarios/constant-vus-subgraphs-delay-resources/README.md)
This scenario runs the same flow as `fed-v1-constant-vus-subgraphs-delay` but with more resources (CPU and RAM) allocated for the gateway.
## `fed-v1-ramping-vus`
[Latest Results](./federation-v1/scenarios/ramping-vus/README.md)
This scenario runs the following:
1. 4 GraphQL subgraphs in dedicated services
2. A GraphQL gateway compatible with the Apollo Federation spec
3. Gradually ramping VUs to a high number, to demo a stress scenario
This measures the following:
1. RPS (requests per second) rate
2. Request duration (average, med, max, p95)
3. Request failures (count)
4. CPU usage during the entire execution
5. RAM usage during the entire execution
6. HTTP layer timings
> This scenario uses Federation v1 spec with all gateways that supports this kind of specification (not all gateways supports v2 spec).