{"id":13550423,"url":"https://github.com/hasura/graphql-bench","last_synced_at":"2025-04-06T12:10:59.908Z","repository":{"id":37873062,"uuid":"129732387","full_name":"hasura/graphql-bench","owner":"hasura","description":"A super simple tool to benchmark GraphQL queries","archived":false,"fork":false,"pushed_at":"2024-01-25T21:08:19.000Z","size":16950,"stargazers_count":270,"open_issues_count":43,"forks_count":40,"subscribers_count":24,"default_branch":"master","last_synced_at":"2025-03-30T11:07:10.973Z","etag":null,"topics":["benchmark","graphql","graphql-server"],"latest_commit_sha":null,"homepage":"","language":"TSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hasura.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-16T11:29:52.000Z","updated_at":"2025-03-11T07:36:25.000Z","dependencies_parsed_at":"2024-01-19T07:06:28.377Z","dependency_job_id":"89c59e6f-921e-4318-b203-74519322a04f","html_url":"https://github.com/hasura/graphql-bench","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hasura%2Fgraphql-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hasura%2Fgraphql-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hasura%2Fgraphql-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hasura%2Fgraphql-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hasura","download_url":"https://codeload.github.com/hasura/graphql-bench/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247478324,"owners_count":20945266,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","graphql","graphql-server"],"created_at":"2024-08-01T12:01:32.894Z","updated_at":"2025-04-06T12:10:59.888Z","avatar_url":"https://github.com/hasura.png","language":"TSQL","funding_links":[],"categories":["TSQL"],"sub_categories":[],"readme":"# GraphQL Bench\n\n- [GraphQL Bench](#graphql-bench)\n  - [Introduction](#introduction)\n  - [Usage](#usage)\n    - [CLI](#cli)\n      - [Commands Overview](#commands-overview)\n      - [Queries/Mutations](#queriesmutations)\n        - [Config](#config)\n        - [Run with Docker](#run-with-docker)\n        - [Run Locally](#run-locally)\n        - [Usage Guide](#usage-guide)\n      - [Subscriptions](#subscriptions)\n        - [Config](#config-1)\n        - [Note: Required Table](#note-required-table)\n        - [Run with Docker](#run-with-docker-1)\n        - [Run Locally](#run-locally-1)\n        - [Usage Guide](#usage-guide-1)\n    - [Programmatic API](#programmatic-api)\n      - [Queries/Mutations](#queriesmutations-1)\n      - [Subscriptions](#subscriptions-1)\n\n## Introduction\n\nGraphQL Bench is a versatile tool for benchmarking and load-testing GraphQL Services.\nIt can be run as a CLI application (local or Docker), and also provides a programmatic API.\nBoth HTTP (Queries/Mutations) and Websocket (Subscriptions) tests are supported.\n\nHTTP tests can be configured to run with your choice of:\n- Autocannon\n- K6\n- wrk2\n\nEach benchmark tool will produce live output you can monitor while running your tests:\n\n| Autocannon                                 | K6                                 |\n| ------------------------------------------ | ---------------------------------- |\n| ![](./readme_images/autocannon-output.png) | ![](./readme_images/k6-output.png) |\n\nThe output is standardized internally across tools using `HDRHistograms` and can be viewed in a web app:\n\n![](./app/hasura-bench-report.gif)\n\n## Usage\n\n### CLI\n\n#### Commands Overview\n```sh\n❯ graphql-bench --help\n\nUSAGE\n  $ graphql-bench [COMMAND]\n\nCOMMANDS\n  help          display help for graphql-bench\n  query         benchmark queries or mutations\n  subscription  benchmark subscriptions\n```\n```sh\n❯ graphql-bench query --help\nbenchmark queries or mutations\n\nUSAGE\n  $ graphql-bench query\n\nOPTIONS\n  -c, --config=config    (required) Filepath to YAML config file for query benchmarks\n  -h, --help             show CLI help\n  -o, --outfile=outfile  Filepath to output JSON file containing benchmark stats\n  --url=url              URL to direct graphql queries; may override 'url' from the YAML config, which is optional if this flag is passed\n\nEXAMPLE\n  $ graphql-bench query --config ./config.query.yaml --outfile results.json\n```\n```sh\n❯ graphql-bench subscription --help\nbenchmark subscriptions\n\nUSAGE\n  $ graphql-bench subscription\n\nOPTIONS\n  -c, --config=config  (required) Filepath to YAML config file for subscription benchmarks\n  -h, --help           show CLI help\n\nEXAMPLE\n  $ graphql-bench subscription --config ./config.subscription.yaml\n```\n\n---\n\n#### Queries/Mutations\n\nWhen running locally, add executable permission to the \n* run binary at `app/cli/bin` by running `chmod +x run`\n* k6 binary at `app/queries/bin/k6/` by running `chmod +x k6`\n* wrk binary at `app/queries/bin/wrk/` by running `chmod +x wrk`\n##### Config\n\nThe Query/Mutation CLI bench expects a YAML config of the following format:\n\n```yaml\nurl: 'http://localhost:8085/v1/graphql'\nheaders:\n  X-Hasura-Admin-Secret: my-secret\n# \"Debug\" mode enables request and response logging for Autocannon and K6\n# This lets you see what is happening and confirm proper behavior.\n# This should be disabled for genuine benchmarks, and only used for debugging/visibility.\ndebug: false\nqueries:\n    # Name: Unique name for the query\n  - name: SearchAlbumsWithArtist\n    # Tools: List of benchmarking tools to run: ['autocannon', 'k6', 'wrk2']\n    tools: [autocannon, k6]\n    # Execution Strategy: the type of the benchmark to run. Options are: \n    # REQUESTS_PER_SECOND: Fixed duration, fixed rps. Example parameters:\n    #   duration: 10s\n    #   rps: 500\n    # FIXED_REQUEST_NUMBER: Complete requests as fast as possible, no duration. Example parameters:\n    #   requests: 10000\n    # MAX_REQUESTS_IN_DURATION: Make as many requests as possible in duration. Example parameters:\n    #   duration: 10s\n    # MULTI_STAGE: (K6 only currently) Several stages of REQUESTS_PER_SECOND benchmark. Example parameters:\n    #   initial_rps: 0\n    #   stages:\n    #     - duration: 5s\n    #       target: 100\n    #     - duration: 10s\n    #       target: 1000\n    # CUSTOM: Pass completely custom options to each tool (see full API spec for all supported options, very large)\n    execution_strategy: REQUESTS_PER_SECOND\n    rps: 2000\n    duration: 10s\n    connections: 50\n    query: |\n      query SearchAlbumsWithArtist {\n        albums(where: {title: {_like: \"%Rock%\"}}) {\n          id\n          title\n          artist {\n            name\n            id\n          }\n        }\n      }\n  - name: AlbumByPK\n    tools: [autocannon, k6]\n    execution_strategy: FIXED_REQUEST_NUMBER\n    requests: 10000\n    query: |\n      query AlbumByPK {\n        albums_by_pk(id: 1) {\n          id\n          title\n        }\n      }\n  - name: AlbumByPKMultiStage\n    tools: [k6]\n    execution_strategy: MULTI_STAGE\n    initial_rps: 0\n    stages:\n      - duration: 5s\n        target: 100\n      - duration: 5s\n        target: 1000\n    query: |\n      query AlbumByPK {\n        albums_by_pk(id: 1) {\n          id\n          title\n        }\n      }\n```\n\n##### Run with Docker\n\n*Note: to be updated when image published to Dockerhub*\n\nThe `Makefile` contains steps to automate building/tagging/running the image. You can run `make build_local_docker_image` and then `make run_docker_query_bench`.\n\nThe configuration used by these `make` commands lives in `/docker-run-test/config.(query|subscription).yaml`, so edit those files to change the run parameters.\n\nTo manually execute, run the following (where the current directory contains the file `config.query.yaml`):\n\n```sh\ndocker run --net=host -v \"$PWD\":/app/tmp -it \\\n  graphql-bench-local query \\\n  --config=\"./tmp/config.query.yaml\" \\\n  --outfile=\"./tmp/report.json\"\n```\n\n##### Run Locally\n\n```sh\ncd cli\nyarn install\n./bin/run query  --config=\"\u003cquery YAML config file path here\u003e\" --outfile=\"report.json\"\n```\n\n##### Usage Guide\n\n- Watch the tool-specific output during the benchmark to view live metrics\n- Save the output to a file, IE `report.json` \n- Inspect `report.json` to view detailed statistics and histograms\n- Open the `report.json` in the web viewer app (hosted at https://hasura.github.io/graphql-bench/app/web-app/) for visual metrics\n\n---\n\n#### Subscriptions\n\n##### Config\n\nThe Subscription CLI bench expects a YAML config of the following format:\n\n```yaml\nurl: 'http://localhost:8085/v1/graphql'\ndb_connection_string: postgres://postgres:postgrespassword@localhost:5430/postgres\nheaders:\n  X-Hasura-Admin-Secret: my-secret\nconfig:\n  # Label must be unique per-run, it identifiers the run in the DB\n  label: SearchAlbumsWithArtistUpdated\n  # Total number of websocket connections to open\n  max_connections: 20\n  # New connections to make per second until target reached\n  connections_per_second: 10\n  # Whether or not to insert the subscription payload data into the DB at the end\n  insert_payload_data: true\n  # The subscription to run\n  query: |\n    subscription AlbumByIDSubscription($artistIds: [Int!]!) {\n      albums(where: {artist_id: { _in: $artistIds}}) {\n        id\n        title\n        updated_at\n      }\n    }\n  # Optional variables (if subscription uses variables)\n  variables:\n    some_value: a_string\n    # Ranges will loop repeatedly from \"start\" to \"end\" and increment by one for each new subscription\n    some_range: { start: 1, end: 10 }\n    another_range: { start: 50, end: 100 }\n    some_number: 10\n    artistIds: [1, 2, 3, 4]\n    some_object:\n      a_key: a_value\n```\n\n##### Note: Required Table\n\nFor the Subscriptions test to record data in the DB, it requires a table `events` present with the following schema:\n\n```sql\nCREATE TABLE public.events (\n  -- unique label to identify benchmark\n  label text NOT NULL\n  -- connection_id represents the n'th connection\n  connection_id int NOT NULL\n  operation_id int NOT NULL\n  -- event_number represents the nth event that was received by the client\n  event_number int NOT NULL\n  -- event_data stores the payload that was received\n  event_data jsonb NOT NULL\n  -- event_time stores the time at which the event was received by the client\n  event_time timestamptz NOT NULL\n  -- is_error represents whether the event was error or not\n  is_error boolean NOT NULL\n  --  latency is not populated by the benchmark tool, but this can be populated by calculating event_time - \u003cevent_triggered_time\u003e\n  latency int\n)\n```\n\n##### Run with Docker\n\n*Note: to be updated when image published to Dockerhub*\n\nThe `Makefile` contains steps to automate building/tagging/running the image. You can run `make build_local_docker_image` and then `make run_docker_subscription_bench`.\n\nThe configuration used by this `make` commands lives in `/docker-run-test/config.(query|subscription).yaml`, so edit those files to change the run parameters.\n\nTo manually execute, run the following (where the current directory contains the file `config.query.yaml`):\n\n```sh\ndocker run --net=host -v \"$PWD\":/app/tmp -it \\\n  graphql-bench-local subscription \\\n  --config=\"./tmp/config.subscription.yaml\" \\\n```\n\n##### Run Locally\n\n```sh\ncd cli\nyarn install\n./bin/run subscription --config=\"\u003csubscription YAML config file path here\u003e\"\n```\n\n##### Usage Guide\n\n- Create events by making changes in the subscribed table\n- As you create changes, you should notice the number of data events increasing in `stdout` output:\n  ![](./subscriptions/example-stdout-output.png)\n- Stop the benchmark with `ctrl + c`\n- The script should say it has inserted the event data:\n\n  ```\n  ❯ Executing Teardown Process\n  ❯ Starting to close socket connections\n  ❯ Sockets closed, attempting to insert event data\n  ❯ Inserted total of 10 events for label SearchAlbumsWithArtistUpdated\n  ❯ Trying to close DB connection pool\n  ❯ Database connection destroyed\n  ❯ Now exiting the process\n  ```\n\n---\n\n### Programmatic API\n\n*Note: A good reference/usage demo exists at `/queries/src/tests.ts` and `/subscriptions/src/tests.ts`.*\n\n\n#### Queries/Mutations\n\nThe easiest way of interacting with this library is to use the exported `BenchmarkRunner` class.\n\nIt exposes a single method, `.runBenchmarks()` which takes a `GlobalConfig` object defining the benchmarks to be run. Here's an example of running a `REQUESTS_PER_SECOND` benchmark for a query using all of `autocannon`, `k6`, and `wrk2`:\n\n```ts\nimport { BenchmarkRunner } from './main'\nimport {\n  GlobalConfig,\n  BenchmarkTool,\n  MaxRequestsInDurationBenchmark,\n  FixedRequestNumberBenchmark,\n  RequestsPerSecondBenchmark,\n  MultiStageBenchmark,\n  CustomBenchmark,\n} from './executors/base/types'\n\nconst queries = {\n  searchAlbumsWithArtist: `\n    query SearchAlbumsWithArtist {\n      albums(where: {title: {_like: \"%Rock%\"}}) {\n        id\n        title\n        artist {\n          name\n          id\n        }\n      }\n    }`,\n}\n\nconst rpsBench: RequestsPerSecondBenchmark = {\n  tools: [BenchmarkTool.AUTOCANNON, BenchmarkTool.K6, BenchmarkTool.WRK2],\n  name: 'AlbumsArtistTrackGenreAll',\n  execution_strategy: 'REQUESTS_PER_SECOND',\n  duration: '3s',\n  rps: 500,\n  query: queries.albumsArtistTracksGenreAll,\n}\n\nconst tests: GlobalConfig = {\n  url: 'http://localhost:8085/v1/graphql',\n  headers: { 'X-Hasura-Admin-Secret': 'my-secret' },\n  queries: [rpsBench],\n}\n\nasync function main() {\n  const runner = new BenchmarkRunner(tests)\n  const results = await runner.runBenchmarks()\n  console.log('Test results:', results)\n}\n\nmain()\n```\n\n#### Subscriptions\n\nThe `main` exported method from the Subscriptions repo is what runs the benchmarks. Here's an example of programmatically running a subscription test:\n\n```ts\nimport { SubscriptionBenchConfig } from './utils'\nimport { main as runSubscriptionBenchmark } from './main'\n\nconst testConfig: SubscriptionBenchConfig = {\n  url: 'http://localhost:8085/v1/graphql',\n  db_connection_string:\n    'postgres://postgres:postgrespassword@localhost:5430/postgres',\n  headers: {\n    'X-Hasura-Admin-Secret': 'my-secret',\n  },\n  config: {\n    label: 'SearchAlbumsWithArtist',\n    max_connections: 20,\n    connections_per_second: 10,\n    insert_payload_data: true,\n    query: `\n      subscription AlbumByIDSubscription($artistIds: [Int!]!) {\n        albums(where: {artist_id: { _in: $artistIds}}) {\n          id\n          title\n          updated_at\n        }\n      }\n    `,\n    variables: {\n      artistIds: [1, 2, 3, 4],\n    },\n  },\n}\n\nasync function main() {\n  await runSubscriptionBenchmark(testConfig)\n}\n\nmain()\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhasura%2Fgraphql-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhasura%2Fgraphql-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhasura%2Fgraphql-bench/lists"}