{"id":48625218,"url":"https://github.com/asyncswap/lsp-bench","last_synced_at":"2026-05-04T01:10:17.928Z","repository":{"id":337926011,"uuid":"1155833573","full_name":"asyncswap/lsp-bench","owner":"asyncswap","description":"A benchmarking framework for Language Server Protocol (LSP) servers. Measures latency, correctness, and memory usage over JSON-RPC stdio.","archived":false,"fork":false,"pushed_at":"2026-05-03T23:26:55.000Z","size":405,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-03T23:33:10.074Z","etag":null,"topics":["benchmark","lsp"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/asyncswap.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-12T00:33:59.000Z","updated_at":"2026-05-03T23:07:08.000Z","dependencies_parsed_at":"2026-02-18T12:00:50.169Z","dependency_job_id":null,"html_url":"https://github.com/asyncswap/lsp-bench","commit_stats":null,"previous_names":["mmsaki/solidity-lsp-benchmarks","mmsaki/lsp-bench","asyncswap/lsp-bench"],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/asyncswap/lsp-bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asyncswap%2Flsp-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asyncswap%2Flsp-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asyncswap%2Flsp-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asyncswap%2Flsp-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/asyncswap","download_url":"https://codeload.github.com/asyncswap/lsp-bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asyncswap%2Flsp-bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32590517,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T22:12:39.696Z","status":"ssl_error","status_checked_at":"2026-05-03T22:09:10.534Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","lsp"],"created_at":"2026-04-09T04:30:31.326Z","updated_at":"2026-05-04T01:10:17.923Z","avatar_url":"https://github.com/asyncswap.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lsp-bench\n\nA benchmarking framework for [Language Server Protocol](https://microsoft.github.io/language-server-protocol/) (LSP) servers. Measures latency, correctness, and memory usage for any LSP server that communicates over JSON-RPC stdio.\n\n## Install\n\n```sh\ncargo install lsp-bench\n```\n\nOr build from source:\n\n```sh\ngit clone --recursive https://github.com/mmsaki/solidity-lsp-benchmarks.git\ncd solidity-lsp-benchmarks\ncargo build --release\n```\n\n## Quick Start\n\n```sh\nlsp-bench init          # generates benchmark.yaml\n# edit benchmark.yaml with your project and servers\nlsp-bench               # run benchmarks\n```\n\n## What It Measures\n\n| Benchmark | What it tests |\n|-----------|---------------|\n| `initialize` | Cold-start time (fresh process per iteration) |\n| `textDocument/diagnostic` | Time to analyze a file and return diagnostics |\n| `textDocument/definition` | Go to Definition latency |\n| `textDocument/declaration` | Go to Declaration latency |\n| `textDocument/typeDefinition` | Go to Type Definition latency |\n| `textDocument/implementation` | Go to Implementation latency |\n| `textDocument/hover` | Hover information latency |\n| `textDocument/references` | Find References latency |\n| `textDocument/completion` | Completion suggestions latency |\n| `textDocument/signatureHelp` | Signature Help latency |\n| `textDocument/rename` | Rename symbol latency |\n| `textDocument/prepareRename` | Prepare Rename latency |\n| `textDocument/documentSymbol` | Document Symbols latency |\n| `textDocument/documentLink` | Document Links latency |\n| `textDocument/formatting` | Document Formatting latency |\n| `textDocument/foldingRange` | Folding Ranges latency |\n| `textDocument/selectionRange` | Selection Ranges latency |\n| `textDocument/codeLens` | Code Lens latency |\n| `textDocument/inlayHint` | Inlay Hints latency |\n| `textDocument/semanticTokens/full` | Semantic Tokens latency |\n| `textDocument/documentColor` | Document Color latency |\n| `workspace/symbol` | Workspace Symbol search latency |\n\nEach benchmark records per-iteration latency (p50, p95, mean), the full LSP response, and resident memory (RSS).\n\n## Configuration\n\nCreate a `benchmark.yaml`:\n\n```yaml\nproject: my-project\nfile: src/main.rs\nline: 45\ncol: 12\n\niterations: 10\nwarmup: 2\ntimeout: 10\nindex_timeout: 15\n\nbenchmarks:\n  - all\n\nservers:\n  - label: my-server\n    cmd: my-language-server\n    args: [\"--stdio\"]\n\n  - label: other-server\n    cmd: other-lsp\n    args: [\"--stdio\"]\n```\n\n### Per-Method Overrides\n\nUse `methods:` to set different positions or trigger characters for specific LSP methods. Methods not listed fall back to the global `line`/`col`.\n\n```yaml\nmethods:\n  textDocument/completion:\n    trigger: \".\"\n  textDocument/hover:\n    line: 10\n    col: 15\n```\n\n### Verification\n\nAdd `expect` fields to assert responses match expected values. Run with `--verify` to turn benchmarks into regression tests:\n\n```yaml\nmethods:\n  textDocument/definition:\n    expect:\n      file: SafeCast.sol\n      line: 39\n    didChange:\n      - file: Pool.sol.dirty\n        line: 216\n        col: 22\n```\n\n```\n$ lsp-bench --config goto.yaml --verify\n  ✓ [1] Pool.sol.dirty\n  verify 1/1 expectations passed\n```\n\nExits non-zero on any mismatch. Per-snapshot `expect` overrides the method-level one.\n\nFor completion benchmarks, `expect` also supports item-level predicates:\n\n- `containsItems`: items that must exist\n- `absentItems`: items that must not exist\n\nEach predicate can match on:\n\n- `label`\n- `detailContains`\n- `sortTextPrefix`\n- `hasAdditionalTextEdits`\n- `additionalTextEditsContain`\n\n### Config Fields\n\n| Field | Default | Description |\n|-------|---------|-------------|\n| `project` | -- | Path to project root |\n| `file` | -- | Target file to benchmark (relative to project) |\n| `line` | 102 | Target line for position-based benchmarks (0-based) |\n| `col` | 15 | Target column (0-based) |\n| `iterations` | 10 | Measured iterations per benchmark |\n| `warmup` | 2 | Warmup iterations (discarded) |\n| `timeout` | 10 | Seconds per LSP request |\n| `index_timeout` | 15 | Seconds for server to index |\n| `output` | `benchmarks` | Directory for JSON results |\n| `benchmarks` | all | List of benchmarks to run |\n| `methods` | -- | Per-method `line`, `col`, and `trigger` overrides |\n| `response` | 80 | `full` (no truncation) or a number (truncate to N chars) |\n| `report` | -- | Output path for generated report |\n| `report_style` | `delta` | Report format: `delta`, `readme`, or `analysis` |\n\nSee [DOCS.md](DOCS.md) for the full configuration reference, example configs, methodology, and report generation details.\n\n## CLI\n\n```sh\nlsp-bench                            # uses benchmark.yaml\nlsp-bench -c my-config.yaml          # custom config\nlsp-bench --verify                   # check responses against expect fields\nlsp-bench init                       # generate a benchmark.yaml template\nlsp-bench --version                  # show version with commit hash\n```\n\n| Flag | Description |\n|------|-------------|\n| `-c, --config \u003cPATH\u003e` | Config file (default: `benchmark.yaml`) |\n| `--verify` | Check responses against `expect` fields in config. Exits non-zero on mismatch. |\n| `-V, --version` | Show version (includes commit hash, OS, arch) |\n| `-h, --help` | Show help |\n\n## Binaries\n\n| Binary | Purpose |\n|--------|---------|\n| `lsp-bench` | Run benchmarks, produce JSON snapshots |\n| `gen-readme` | Generate README with medals and feature matrix |\n| `gen-analysis` | Generate per-feature analysis report |\n| `gen-delta` | Generate compact comparison table |\n\n## Output\n\nJSON snapshots with per-iteration latency, response data, and memory:\n\n```json\n{\n  \"server\": \"my-server\",\n  \"status\": \"ok\",\n  \"mean_ms\": 8.8,\n  \"p50_ms\": 8.8,\n  \"p95_ms\": 10.1,\n  \"rss_kb\": 40944,\n  \"response\": { \"uri\": \"file:///...Main.sol\", \"range\": { \"start\": { \"line\": 9, \"character\": 4 }, \"end\": { \"line\": 9, \"character\": 10 } } },\n  \"iterations\": [\n    { \"ms\": 8.80 },\n    { \"ms\": 8.45 },\n    { \"ms\": 9.12, \"response\": { \"uri\": \"file:///...Other.sol\", \"range\": { \"...\" : \"...\" } } }\n  ]\n}\n```\n\nThe top-level `response` is the canonical result. Per-iteration `response` is only included when it differs from the canonical one. Responses are stored as native JSON values (objects, arrays, strings, or null).\n\n### Report Generation\n\nSet `report` in your config to auto-generate a report after benchmarks:\n\n```yaml\nreport: REPORT.md\nreport_style: delta    # delta (default), readme, or analysis\n```\n\nExample delta output:\n\n```\n| Benchmark                | baseline | my-branch |       Delta |\n|--------------------------|----------|-----------|-------------|\n| initialize               |   4.05ms |    3.05ms | 1.3x faster |\n| textDocument/diagnostic  | 123.80ms |  124.10ms | 1.0x (tied) |\n| textDocument/hover       |   2.30ms |    2.21ms | 1.0x (tied) |\n| textDocument/definition  |   8.95ms |    8.90ms | 1.0x (tied) |\n```\n\n## Methodology\n\n- Real LSP requests over JSON-RPC stdio\n- Sequential iterations (next starts after previous completes)\n- Fresh server process for `initialize` and `textDocument/diagnostic`\n- Persistent server for method benchmarks (definition, hover, etc.)\n- RSS memory sampled after indexing\n- Warmup iterations discarded from measurements\n\nSee [DOCS.md](DOCS.md) for full methodology details.\n\n## Attribution\n\nInspired by [@libkakashi](https://github.com/libkakashi) and his [lsp-bench](https://github.com/libkakashi/lsp-bench) — thanks for the inspiration to build this benchmark framework.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasyncswap%2Flsp-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fasyncswap%2Flsp-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasyncswap%2Flsp-bench/lists"}