{"id":18592932,"url":"https://github.com/litestar-org/api-performance-tests","last_synced_at":"2025-04-10T16:30:37.881Z","repository":{"id":44747880,"uuid":"450592702","full_name":"litestar-org/api-performance-tests","owner":"litestar-org","description":"Benchmarking Litestar vs other ASGI API framework","archived":false,"fork":false,"pushed_at":"2024-07-15T18:29:04.000Z","size":17438,"stargazers_count":40,"open_issues_count":13,"forks_count":7,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-03-24T23:51:26.014Z","etag":null,"topics":["api","asgi","benchmark","benchmarking","http","litestar","litestar-api","litestar-framework","litestar-meta","open-source","openapi","pthon","python","rest","starlite"],"latest_commit_sha":null,"homepage":"https://litestar.dev/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/litestar-org.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["litestar-org"],"open_collective":"litestar","polar":"litestar-org"}},"created_at":"2022-01-21T18:05:50.000Z","updated_at":"2024-11-30T06:20:23.000Z","dependencies_parsed_at":"2023-10-03T08:55:08.153Z","dependency_job_id":"db8bae41-fd7f-4783-926f-8a2e61b8d064","html_url":"https://github.com/litestar-org/api-performance-tests","commit_stats":null,"previous_names":["starlite-api/api-performance-tests"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/litestar-org%2Fapi-performance-tests","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/litestar-org%2Fapi-performance-tests/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/litestar-org%2Fapi-performance-tests/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/litestar-org%2Fapi-performance-tests/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/litestar-org","download_url":"https://codeload.github.com/litestar-org/api-performance-tests/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248252691,"owners_count":21072699,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","asgi","benchmark","benchmarking","http","litestar","litestar-api","litestar-framework","litestar-meta","open-source","openapi","pthon","python","rest","starlite"],"created_at":"2024-11-07T01:10:40.188Z","updated_at":"2025-04-10T16:30:35.512Z","avatar_url":"https://github.com/litestar-org.png","language":"Python","funding_links":["https://github.com/sponsors/litestar-org","https://opencollective.com/litestar","https://polar.sh/litestar-org"],"categories":[],"sub_categories":[],"readme":"\u003c!-- markdownlint-disable --\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/litestar-org/branding/blob/473f54621e55cde9acbb6fcab7fc03036173eb3d/assets/Branding%20-%20SVG%20-%20Transparent/Logo%20-%20Banner%20-%20Inline%20-%20Light.svg#gh-light-mode-only\" alt=\"Litestar Logo - Light\" width=\"100%\" height=\"auto\" /\u003e\n  \u003cimg src=\"https://github.com/litestar-org/branding/blob/473f54621e55cde9acbb6fcab7fc03036173eb3d/assets/Branding%20-%20SVG%20-%20Transparent/Logo%20-%20Banner%20-%20Inline%20-%20Dark.svg#gh-dark-mode-only\" alt=\"Litestar Logo - Dark\" width=\"100%\" height=\"auto\" /\u003e\n\u003c/p\u003e\n\u003c!-- markdownlint-restore --\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n\u003c!-- prettier-ignore-start --\u003e\n\n\n| Project   |     | Status                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |\n|-----------|:----|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| CI/CD     |     | [![CI](https://github.com/litestar-org/api-performance-tests/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/litestar-org/api-performance-tests/actions/workflows/ci.yml)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |\n| Community |     | [![Reddit](https://img.shields.io/reddit/subreddit-subscribers/litestarapi?label=r%2FLitestar\u0026logo=reddit\u0026labelColor=202235\u0026color=edb641\u0026logoColor=edb641)](https://reddit.com/r/litestarapi) [![Discord](https://img.shields.io/discord/919193495116337154?labelColor=202235\u0026color=edb641\u0026label=chat%20on%20discord\u0026logo=discord\u0026logoColor=edb641)](https://discord.gg/X3FJqy8d2j) [![Matrix](https://img.shields.io/badge/chat%20on%20Matrix-bridged-202235?labelColor=202235\u0026color=edb641\u0026logo=matrix\u0026logoColor=edb641)](https://matrix.to/#/#litestar:matrix.org) [![Medium](https://img.shields.io/badge/Medium-202235?labelColor=202235\u0026color=edb641\u0026logo=medium\u0026logoColor=edb641)](https://blog.litestar.dev) [![Twitter](https://img.shields.io/twitter/follow/LitestarAPI?labelColor=202235\u0026color=edb641\u0026logo=twitter\u0026logoColor=edb641\u0026style=flat)](https://twitter.com/LitestarAPI) [![Blog](https://img.shields.io/badge/Blog-litestar.dev-202235?logo=blogger\u0026labelColor=202235\u0026color=edb641\u0026logoColor=edb641)](https://blog.litestar.dev) |\n| Meta      |     | [![Litestar Project](https://img.shields.io/badge/Litestar%20Org-%E2%AD%90%20API%20Perf%20Tests-202235.svg?logo=python\u0026labelColor=202235\u0026color=edb641\u0026logoColor=edb641)](https://github.com/litestar-org/litestar) [![License - MIT](https://img.shields.io/badge/license-MIT-202235.svg?logo=python\u0026labelColor=202235\u0026color=edb641\u0026logoColor=edb641)](https://spdx.org/licenses/) [![Litestar Sponsors](https://img.shields.io/badge/Sponsor-%E2%9D%A4-%23edb641.svg?\u0026logo=github\u0026logoColor=edb641\u0026labelColor=202235)](https://github.com/sponsors/litestar-org) [![linting - Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v2.json\u0026labelColor=202235)](https://github.com/astral-sh/ruff) [![code style - Black](https://img.shields.io/badge/code%20style-black-000000.svg?logo=python\u0026labelColor=202235\u0026logoColor=edb641)](https://github.com/psf/black)                                                                                                                        |\n\n\u003c!-- prettier-ignore-end --\u003e\n\u003c/div\u003e\n\n# api-performance-tests\n\n\u003e [!IMPORTANT]\\\n\u003e [**_Starlite has been renamed to Litestar_**](https://litestar.dev/about/organization.html#litestar-and-starlite)\n\nThis is an API performance test comparing:\n\n1. [Litestar](https://github.com/litestar-org/litestar)\n2. [Starlite v1.5x](https://github.com/litestar-org/litestar/tree/v1.51)\n3. [Starlette](https://github.com/encode/starlette)\n4. [FastAPI](https://github.com/tiangolo/fastapi)\n5. [Sanic](https://github.com/sanic-org/sanic)\n6. [BlackSheep](https://github.com/Neoteroi/BlackSheep)\n\nUsing the [bombardier](https://github.com/codesenberg/bombardier) HTTP benchmarking tool.\n\n## Test Setup\n\nSetup is identical for all frameworks.\n\n- Applications reside in the `frameworks` folder and consist of a single file named `\u003cframework_name\u003e_app.py`\n\n### Tests\n\nAll tests are run sync and async\n\n#### Serialization and data sending\n\n##### Plaintext\n\n- Sending 100 bytes plaintext\n- Sending 1kB plaintext\n- Sending 10kB plaintext\n- Sending 100kB plaintext\n- Sending 500kB plaintext\n- Sending 1MB plaintext\n- Sending 5MB plaintext\n\n##### JSON\n\nSerializing a dictionary into JSON\n\n- Serializing and sending 1kB JSON\n- Serializing and sending 10kB JSON\n- Serializing and sending 100kB JSON\n- Serializing and sending 500kB JSON\n- Serializing and sending 1MB JSON\n\n##### Serialization\n\n(only supported by `Litestar`, `Starlite`, and `FastAPI`)\n\n- Serializing 50 dataclass objects each referencing 2 more dataclass objects\n- Serializing 100 dataclass objects each referencing 5 more dataclass objects\n- Serializing 500 dataclass objects each referencing 3 more dataclass objects\n- Serializing 50 pydantic objects each referencing 2 more pydantic objects\n- Serializing 100 pydantic objects each referencing 5 more pydantic objects\n- Serializing 500 pydantic objects each referencing 3 more pydantic objects\n\n##### Files\n\n- Sending a 100 bytes binary file\n- Sending a 1kB bytes binary file\n- Sending a 50kB binary file\n- Sending a 1MB bytes binary file\n\n#### Path amd query parameter handling\n\nAll responses return \"No Content\"\n\n- No path parameters\n- Single path parameter, coerced into an integer\n- Single query parameter, coerced into an integer\n- A path and a query parameters, coerced into integers\n\n#### Dependency injection\n\n(not supported by `Starlette`)\n\n- Resolving 3 nested synchronous dependencies\n- Resolving 3 nested asynchronous dependencies (only supported by `Litestar`, `Starlite`, and `FastAPI`)\n- Resolving 3 nested synchronous, and 3 nested asynchronous dependencies (only supported by `Litestar`, `Starlite`, and `FastAPI`)\n\n#### Modifying responses\n\nAll responses return \"No Content\"\n\n- Setting response headers\n- Setting response cookies\n\n## Running the tests\n\n### Prerequisites\n\n- [docker](https://docs.docker.com/get-docker/)\n- [poetry](https://python-poetry.org/docs/#installation)\n- Python 3.11\n\n### Running tests\n\n1. Clone this repo\n2. Run `poetry install`\n3. Run tests with `poetry run bench run --rps --latency`\n\nAfter the run, the results will be stored in `results/run_\u003crun_mumber\u003e.json`\n\n#### Selecting which frameworks to test\n\nTo select a framework, simply pass its name to the `run command`:\n\n`bench run --rps litestar starlite starlette fastapi`\n\n##### Selecting a framework version\n\n- A version available on PyPi: `bench run --rps litestar@v2.0.0`\n- A version from git: `bench run --rps litestar@git+branch_or_tag_name`\n- A version from a specific git repository: `bench run --rps litestar@git+https://github.com/litestar-org/litestar.git@branch_or_tag_name`\n- A local file: `bench run --rps litestar@file+/path/to/litestar`\n\n#### Running a specific test\n\nYou can run a single test by specifying its full name and category:\n\n`bench run --rps litestar -t json:json-1K`\n\n#### Test Settings\n\n|                                                                                                                                                                       |                                                                 |\n| --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |\n| -r, --rebuild                                                                                                                                                         | rebuild docker images                                           |\n| -L, --latency                                                                                                                                                         | run latency tests                                               |\n| -R, --rps                                                                                                                                                             | run RPS tests                                                   |\n| -w, --warmup                                                                                                                                                          | duration of the warmup period (default: 5s)                     |\n| -e, --endpoint mode [sync\u0026#124;async]                                                                                                                                 | endpoint types to select (default: sync, async)                 |\n| -c, --endpoint-category [plaintext\u0026#124;json\u0026#124;files\u0026#124;params\u0026#124;dynamic-response\u0026#124;dependency-injection\u0026#124;serialization\u0026#124;post-json\u0026#124;post-body] | test types to select (default: all)                             |\n| -d, --duration                                                                                                                                                        | duration of the rps benchmarks (default: 15s)                   |\n| -l, --limit                                                                                                                                                           | max requests per second for latency benchmarks (default: 20)    |\n| -r, --requests                                                                                                                                                        | total number of requests for latency benchmarks (default: 1000) |\n\n### Analyzing the results\n\n- Run `bench results` to generate plots from the latest test results\n- Run `bench results -s` to generate plots from the latest test results and split them into separate files for each category\n\n## Contributing\n\nPRs are welcome.\n\nPlease make sure to install [pre-commit](https://pre-commit.com/) on your system, and then execute `pre-commit install`\nin the repository root - this will ensure the pre-commit hooks are in place.\n\nAfter doing this, add a PR with your changes and a clear description of the changes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flitestar-org%2Fapi-performance-tests","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flitestar-org%2Fapi-performance-tests","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flitestar-org%2Fapi-performance-tests/lists"}