{"id":18492976,"url":"https://github.com/graphprotocol/qlog","last_synced_at":"2025-09-23T17:19:44.926Z","repository":{"id":43019998,"uuid":"302759514","full_name":"graphprotocol/qlog","owner":"graphprotocol","description":"A tool to summarize and analyze graph-node query logs","archived":false,"fork":false,"pushed_at":"2022-03-22T23:53:10.000Z","size":68,"stargazers_count":10,"open_issues_count":0,"forks_count":3,"subscribers_count":24,"default_branch":"main","last_synced_at":"2025-04-01T01:05:26.728Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/graphprotocol.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-09T21:44:46.000Z","updated_at":"2025-01-31T10:37:00.000Z","dependencies_parsed_at":"2022-09-10T00:01:15.587Z","dependency_job_id":null,"html_url":"https://github.com/graphprotocol/qlog","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graphprotocol%2Fqlog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graphprotocol%2Fqlog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graphprotocol%2Fqlog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graphprotocol%2Fqlog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/graphprotocol","download_url":"https://codeload.github.com/graphprotocol/qlog/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247931510,"owners_count":21020247,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T13:12:07.167Z","updated_at":"2025-09-23T17:19:39.864Z","avatar_url":"https://github.com/graphprotocol.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"This repo contains a tool to process and summarize the query logs that\n[graph-node](https://github.com/graphprotocol/graph-node) generates. It can\nbe used to\n\n* transform the textual log files that `graph-node` poduces into JSONL\n  files with one JSON object for each query. The input can either be the\n  raw log from `graph-node` or the log entries that Google Cloud's\n  Stackdriver emits. The JSONL files can then be used as input to\n  [agora](https://github.com/graphprotocol/agora/)\n* summarize query performance for similar queries to calculate basic\n  statistics like average and maximum query duration\n* sample query logs to generate a random sample of a larger logfile\n\n## Installation\n\nYou will need to have a [Rust toolchain installed ](https://rustup.rs/)\nCheck out this repository, `cd` into the checkout and run\n\n```\ncargo install --bins --path .\n```\n\nThe resulting binary will be placed into `~/.cargo/bin`. If you want to\nupdate an existing installation, you will also have to pass `--force` to\n`cargo`.\n\n\n## Gathering query logs\n\nWhen you run `graph-node`, set `GRAPH_LOG_QUERY_TIMING` to `gql`. That will\nmake `graph-node` log lines of the form\n\n```\nSep 22 17:01:52.521 INFO Query timing (GraphQL), block: 1234567, query_time_ms: 489, variables: null, query: query things { things(first: 10) { id name } } , query_id: f3e751f1852e62e6-cf1efbcd35771aeb, subgraph_id: Qmsubgraph, component: GraphQlRunner\n```\n\n`qlog` can then be used to transform and summarize these logs in a number\nof ways\n\n## Processing query logs\n\nIn its simplest form, `qlog process` reads a textfile containing log\nentries and transforms them into line-separated JSON (JSONL). To do that,\nrun\n```\ngrep 'Query timing' queries.log | qlog process --text --output queries.jsonl\n```\n\nIn addition, `qlog` can also summarize queries by grouping similar queries\nby their 'shape hash' by passing `--graphql summary.jsonl`, for example to\njust summarize queries, run\n\n```\ngrep 'Query timing' queries.log | qlog process --text --graphql summary.jsonl\n```\n\nIf there is already a file with queries in JSONL form, `qlog process` can\nsummarize this with the following command. Note that we do not pass\n`--text` to `qlog process` since the input file is already in JSONL form:\n\n```\ncat queries.jsonl | qlog process --graphql summary.jsonl\n```\n\nFinally, `qlog process` can also be used to take fixed-size samples of a\nlogfile; running `qlog process` like this will produce a file\n`samples.jsonl` that contains 1,000 samples per subgraph taken\nuniformly from `queries.jsonl`:\n\n```\ncat queries.jsonl | qlog process --samples 1000 --sample-file samples.jsonl\n```\nSampling can also be restricted to certain subgraphs with the\n`--sample-subgraphs` option which expects a comma-separated list of\nsubgraph identifiers, i.e. identifiers in the form `Qmsubgraph`.\n\nThe options for converting a logfile to JSONL, to summarize a logfile, and\nto sample a logfile can be combined so that `qlog process` only needs to be\nrun over a logfile once, which saves a significant amount of time for large\nlogfiles:\n\n```\ngrep 'Query timing' queries.log | \\\n  qlog process --text --graphql summary.jsonl \\\n     --output queries.jsonl \\\n     --samples 1000 --sample-file samples.jsonl\n```\n\n### Using `qlog` with Google Cloud\n\nGoogle Cloud's logging infrastructure, Stackdriver, wraps log messages in\nJSON objects. It is possible to have these JSON objects delivered into\nfiles in Google Cloud storage by defining a sink in Google Cloud Logging.\n\nThe command `qlog extract` can be used to parse such files and turn them\ninto JSONL log files, similar to what `qlog process` with the `--output`\noption produces. Run\n\n```\ncat stackdriver.log | qlog extract --graphql queries.jsonl -\n```\nto turn a StackDriver logfile into a JSONL logfile that `qlog process` can\nsummarize and sample.\n\n\n## Combining query summaries\n\nThe command `qlog combine` can be used to combine multiple summary files\ninto one by running `qlog combine summary1.jsonl summary2.jsonl ... \u003e\nsummary.jsonl`\n\n\n## Analysing query logs\n\nWith a summary file `summary.jsonl` produced by `qlog process` in hand, it\nis possible to perform simple analysis tasks with `qlog stats` and `qlog\nquery`. For example, the command\n```\nqlog stats -s total -f summary.jsonl\n\n```\nwill list query details sorted by their total time.\n\n`qlog stats` can also provide an overview of queries in tabular form, for\nexample by running `qlog stats summary.json | head -n 10`:\n\n```console\n|   QID   |  calls   |    total     |  avg   | stddev |  max   |  slow  |\n|---------+----------+--------------+--------+--------+--------+--------|\n| Q000019 |   933741 |    430758440 |    461 |    284 |   7998 |  30440 |\n| Q000511 |  3153187 |    412610852 |    131 |    137 |   3991 |  13506 |\n| Q000534 |   360948 |    152649245 |    423 |    282 |  10689 |   9162 |\n| Q000343 |  1291109 |     35364900 |     27 |     40 |   2918 |     19 |\n| Q000829 |     4269 |     33130650 |   7761 |   1116 |  17533 |   4269 |\n| Q000527 |   785074 |     32840740 |     42 |     45 |   1974 |     27 |\n| Q001673 |    24366 |     27497870 |   1129 |    492 |   7168 |  10180 |\n| Q001453 |   106500 |     24405334 |    229 |    361 |   6884 |   2827 |\n```\n\nThe `query` subcommand can be used to print more details about a specific\nquery (the leading `0` in the `QNNN` identifier are optional):\n\n```console\nhost:qlog\u003eqlog query summary.json Q558 Q333\n================================ Q558 ================================\n# subgraph:      QmaTK1m8VszFp7iijbWrX65iN8e5zogvJYvUAck7HEvAtQ\n# calls:                      1\n# slow_count:                 1\n# slow_percent:          100.00 %\n# total_time:              11.5 m\n# avg_time:              692850 ms\n# stddev_time:                0 ms\n# max_time:              692850 ms\n# max_uuid:      160e5f6b-f230-428a-bd3b-8cae6cbb5d89\n\nquery getTokens($pageSize: Int = 50) { tokens(first: $pageSize) { address name symbol decimals events { ... on TransferEvent { amount sender destination } } } }\n\n================================ Q333 ================================\n# subgraph:      QmVEoWSQ8eNnVkL1uw2toDSc1xXKi1ADYUm36p961Q1EzT\n# calls:                    692\n# slow_count:                10\n# slow_percent:            1.45 %\n# total_time:              18.7 m\n# avg_time:                1625 ms\n# stddev_time:            14798 ms\n# max_time:              158087 ms\n# max_uuid:      e956ea45-807d-43bb-ad4b-4adcd04bc281\n\nquery getSubdomains($id: ID!) { domain(id: $id) { id labelName subdomains { id labelName labelhash name owner { id __typename } __typename } __typename } }\n```\n\n### Using `jq` for simple analysis\n\nIf we have a directory with summary files `YYYY-MM-DD.jsonl` for each day,\nwe can list the total number of queries per day using `jq`:\n\n```bash\nfor f in *.jsonl\ndo\n  day=$(basename $f .jsonl)\n  gql=$(jq -s 'map(.calls) | add ' \u003c $f)\n  printf \"%s %10s\\n\" \"$day\" \"$gql\"\ndone\n```\n\nThis saves the number of queries per subgraph per day to a csv:\n```bash\nsummaries=\"\"\nfor f in *.jsonl\ndo\n  day=$(basename $f .json)\n  day_summary=$(jq -s 'group_by(.subgraph) |\n    map({\"subgraph\":(.[0].subgraph), \"calls\":(reduce .[].calls as $calls (0; . + $calls)), \"date\":(\"'$day'\") }) |\n    map([.subgraph, .calls, .date] | join(\", \")) | join(\" \\r\\n \")' \u003c $f)\n  summaries+=\"${day_summary//\\\"} \\r\\n\"\ndone\nprintf \"${summaries//\\\"}\" \u003e subgraph_daily_summary.csv\n```\n\n### Format of the JSONL file\n\nThe JSONL files that `qlog process --graphql` produces contain a list of\nJSON objects, one for each query that `graph-node` responded to with the\nfollowing entries:\n\n* `subgraph`: the IPFS hash of the subgraph against which the query was run\n* `query_id`: the entry is in the form `\u003cshape hash\u003e-\u003cquery hash\u003e` where\n  the shape hash is the hash of the query text when disregarding concrete\n  filter values in the query, so that a query with `first: 10` and one with\n  `first: 100` produce the same shape hash. THe `query hash` is the hash of\n  the query text and the variables passed to the query.\n* `block`: the number of the block against which the query was executed\n* `time`: how long processing the query took in ms\n* `query`: the text of the GraphQL query\n* `variables`: the variables used in the query as a string that is a JSON\n  object in its own right\n* `timestamp`: the server time when the query was run\n\nThe summary JSONL files produced with `qlog process --summary` contain JSON\nobjects with the following entries:\n\n* `query`: an example of the query being summarized. Queries are summarized\n  by their shape hash, i.e. the summary file contains one entry for each\n  unique shape in the original log file\n* `subgraph`: the IPFS hash of the subgraph\n* `calls`: the number of times this query shape was executed\n* `slow_count`: the number of executions of this query shape that took more\n  than 1s\n* `total_time`: the sum of the execution time of all queries being\n  summarized in ms\n* `time_squared`: the sum of the square of the execution time of queries in\n  ms^2\n* `max_time`: the time that the slowest execution of this query shape took\n  in ms\n* `max_uuid`: the `query_id` of a query that took `max_time` (it's called\n  `uuid` for historical reasons)\n* `max_variables`: the variables that were passed to the query when it took\n  `max_time`\n* `max_complexity`: meaningless; only there for historical reasons\n* `id`, `hash`: used by `qlog` for internal bookkeeping\n\nAverage query execution time can be calculated from this data as\n`total_time / calls` and the standard deviation as `sqrt(time_squared /\ncalls - (total_time/calls)^2)`\n\n\n### Write to Postgres\n\nFor more complex analysis, it is best to load summaries into a dedicated\nPostgres database. In the following, we assume that the environment is set\nup in such a way that running `psql` will connect to that dedicated\ndatabase. The command `./bin/load -c` will create a database schema `qlog`\nfor that purpose.\n\nAssuming there is a directory `daily-summaries` with one summary file\ncalled `YYYY-MM-DD.jsonl` for each day, simply run\n```\n./bin/load daily-summaries/*.jsonl\n```\n\nSubsequent days can be added to the database with `./bin/load\ndaily-summaries/2020-11-01.jsonl`\n\n#### View data\n\n\nOnce the summary data is loaded into Postgres, we can run even more complex\nqueries, for example, to find the subgraphs causing the most work:\n```console\nhost:qlog\u003e psql -c '\n\u003e select subgraph,\n\u003e        sum(total_time::numeric)/(24*60*60*1000) as total_time_days,\n\u003e        sum(calls) as calls\n\u003e   from qlog.data\n\u003e  group by subgraph\n\u003e  order by 2 desc\n\u003e  limit 5;'\n```\n\n# Copyright\n\nCopyright \u0026copy; 2021 The Graph Foundation\n\nLicensed under the [MIT license](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraphprotocol%2Fqlog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgraphprotocol%2Fqlog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraphprotocol%2Fqlog/lists"}