{"id":20406763,"url":"https://github.com/defnull/multipart_bench","last_synced_at":"2025-03-05T02:22:16.606Z","repository":{"id":259792218,"uuid":"864229180","full_name":"defnull/multipart_bench","owner":"defnull","description":"Benchmarks for Python based multipart/form-data parsers","archived":false,"fork":false,"pushed_at":"2024-12-28T16:50:01.000Z","size":63,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-15T12:16:52.206Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/defnull.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-27T18:28:42.000Z","updated_at":"2025-01-09T20:27:10.000Z","dependencies_parsed_at":"2025-01-15T11:52:35.896Z","dependency_job_id":"b081db71-ffcd-4c86-a271-5090639464bc","html_url":"https://github.com/defnull/multipart_bench","commit_stats":null,"previous_names":["defnull/multipart_bench"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defnull%2Fmultipart_bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defnull%2Fmultipart_bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defnull%2Fmultipart_bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/defnull%2Fmultipart_bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/defnull","download_url":"https://codeload.github.com/defnull/multipart_bench/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241950934,"owners_count":20047727,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T05:19:10.485Z","updated_at":"2025-03-05T02:22:16.588Z","avatar_url":"https://github.com/defnull.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Benchmark for Python multipart/form-data parsers\n\nThis repository contains scenarios and parser tests for different Python based\n`multipart/form-data` parsers, comparing both blocking and non-blocking APIs (if\navailable). The [multipart](https://pypi.org/project/multipart/) library is used\nas a baseline, because it is currently the fastest pure-python parser tested and\nalso the reason I'm benchmarking parsers in the first place.\n\n## Contestants\n\n* [multipart](https://pypi.org/project/multipart/) v1.2.1\n  * Used in [Bottle](https://pypi.org/project/bottle/), [LiteStar](https://litestar.dev/), [Zope](https://zope.readthedocs.io/) and others.\n  * [CPython docs](https://docs.python.org/3.12/library/cgi.html) recommend it as a `cgi.FieldStorage` replacement.\n  * Disclaimer: I am the author and maintainer of this library.\n* [werkzeug](https://pypi.org/project/Werkzeug/) v3.1.3\n  * Used in [Flask](https://pypi.org/project/Flask/) and others.\n  * Does a lot more than *just* multipart parsing.\n* [django](https://pypi.org/project/Django/) v5.1.4\n  * Full featured web framework, not just a parser.\n* [python-multipart](https://pypi.org/project/python-multipart/) v0.0.20\n  * Used in [Starlette](https://pypi.org/project/starlette/) and thus [FastAPI](https://pypi.org/project/fastapi/).\n* [streaming-form-data](https://pypi.org/project/streaming-form-data/) v1.19.0 \n  * Partly written in Cython.\n* [emmett-core](https://pypi.org/project/emmett-core/) 1.0.5\n  * Mostly written in Rust.\n  * Similar to Django or werkzeug, this library does a lot more than *just* multipart parsing. It is not a stand-alone parser, but a support library for the [emmett](https://emmett.sh/) framework and rarely used outside of this context. \n* [cgi.FieldStorage](https://docs.python.org/3.12/library/cgi.html) CPython 3.12.3\n  * Deprecated in Python 3.11 and removed in Python 3.13\n* [email.parser.BytesFeedParser](https://docs.python.org/3.12/library/email.parser.html#email.parser.BytesFeedParser) CPython 3.12.3\n  * Designed as a parser for emails, not `multipart/form-data`.\n  * Buffers everything in memory, including large file uploads.\n\n**Not included:** Some parsers *cheat* by loading the entire request body into memory\n(e.g. sanic or litestar before they switched to multipart). Those are obviously\nvery fast in benchmarks but also very unpractical when dealing with large file\nuploads.\n\n\n\n## Updates\n\n* **30.09.2024** `python-multipart` v0.0.11 fixed a bug that caused extreme\n  slowdowns (as low as 0.75MB/s) in all three worst-case scenarios.\n* **30.09.2024** There was an issue with the `email` parser that caused it to\n  skip over the actual parsing and also not do any IO in the blocking test.\n  Throughput was way higher than expected. This is fixed now.\n* **30.09.2024** Default size for in-memory buffers is different for each parser,\n  resulting in an unfair comparison. The tests now configure a limit of 500K for\n  each parser, which is the hard-coded value in `werkzeug` and also a sensible\n  default.\n* **03.10.2024** New version of `multipart` with slightly better results in some tests.\n* **05.10.2024** Added results for `streaming-form-data` parser.\n* **25.10.2024** Added results for `django` parser.\n* **06.11.2024** Added results for `emett-core` parser.\n* **24.12.2024** New versions for many libraries and an additional \"worstcase_junk\"\n  scenario. The results were so bad for some of the libraries that I reported it\n  as a potential security issue (DoS vulnerability) to the most affected libraries\n  and waited for a fix to be available before publishing results.\n\n\n## Method\n\nAll tests were performed on a pretty old \"AMD Ryzen 5 3600\" running Linux 6.8.0\nand Python 3.12.3 with highest possible priority and pinned to a single core.\n\nFor each test, the parser is created with default¹ settings and the results are\nthrown away. Some parsers buffer to disk, but `TEMP` points to a ram-disk to\nreduce disk IO from the equation. Each test is repeated until there is no\nimprovement for at least 100 runs in a row, then the best run is used to compute\nthe theoretical maximum throughput per core.\n\nThe fastest pure-python parser (currently `multipart`) is used as the 100% baseline\nfor each test. This ensures that pure python parsers are always easy to compare\nagainst each other, and compiled parsers can be included without screwing with\nthe results too much.\n\n¹) There is one exception: The limit for in-memory buffered files is set to\n500KB (hard-coded in `werkzeug`) to ensure a fair comparison.\n\n\n## Results\n\nParser throughput is measured in MB/s (input size / time). Higher throughput is\nbetter.\n\n\n### Scenario: simple\n\nA simple form with just two small text fields.\n\n| Parser              | Blocking (MB/s)   | Non-Blocking (MB/s)   |\n|---------------------|-------------------|-----------------------|\n| multipart           | 15.57 MB/s (100%) | 23.24 MB/s (100%)     |\n| werkzeug            | 5.55 MB/s (36%)   | 7.11 MB/s (31%)       |\n| django              | 3.08 MB/s (20%)   | -                     |\n| python-multipart    | 3.66 MB/s (23%)   | 6.14 MB/s (26%)       |\n| streaming-form-data | 0.80 MB/s (5%)    | 0.84 MB/s (4%)        |\n| emmett-core         | 71.14 MB/s (457%) | -                     |\n| cgi                 | 4.79 MB/s (31%)   | -                     |\n| email               | 3.95 MB/s (25%)   | 4.36 MB/s (19%)       |\n\nThis scenario is so small that it shows initialization and interpreter overhead\nmore than actual parsing performance, which benefits `emmett-core` the most\nbecause everything happens in Rust and outside of the python runtime. The results\nfor `streaming-form-data` are a bit surprising though, given that it is partly\nwritten in Cython and compiled to native code. My guess is that there is some\nsignificant overhead when calling Python callbacks from Cython, which happens a\nlot in this test. When comparing the pure-python parsers, `multipart` is the\nclear winner.\n\n**Note:** Small forms like these should better be transmitted as\n`application/x-www-form-urlencoded`, which has a lot less overhead compared to\n`multipart/form-data` and should be a lot faster to parse, so take this benchmark\nwith a large grain of salt. This is an uncommon and artificial scenario.\n\n\n### Scenario: large\n\nA large form with 100 small text fields.\n\n| Parser              | Blocking (MB/s)    | Non-Blocking (MB/s)   |\n|---------------------|--------------------|-----------------------|\n| multipart           | 28.09 MB/s (100%)  | 36.37 MB/s (100%)     |\n| werkzeug            | 9.65 MB/s (34%)    | 12.49 MB/s (34%)      |\n| django              | 5.53 MB/s (20%)    | -                     |\n| python-multipart    | 5.11 MB/s (18%)    | 9.25 MB/s (25%)       |\n| streaming-form-data | 1.13 MB/s (4%)     | 1.17 MB/s (3%)        |\n| emmett-core         | 131.14 MB/s (467%) | -                     |\n| cgi                 | 6.43 MB/s (23%)    | -                     |\n| email               | 11.18 MB/s (40%)   | 12.95 MB/s (36%)      |\n\nThis scenario benefits parsers with low per-field overhead or a line-based\nparser design (like `cgi` and `email`) because each field is just a single line,\nand there are a lot of them. Initialization overhead is less important here\ncompared to the 'simple' scenario above.\n\nNo surprise that `emmett-core` performs well here, because the payload still\nfits in a small number of chunks and other than `streaming-form-data` the parser\ndoes not have to call into Python code for each field. The Rust parser thus\ncompletely bypasses the python interpreter overhead. `email` also performs\nreasonably well, as it is designed for this type of line-based text input and\neven surpasses many of the other pure-python parsers, but `multipart` is still\nmore than twice as fast.\n\n\n### Scenario: upload\n\nA file upload with a single large (32MB) file.\n\n| Parser              | Blocking (MB/s)     | Non-Blocking (MB/s)   |\n|---------------------|---------------------|-----------------------|\n| multipart           | 1202.36 MB/s (100%) | 6193.85 MB/s (100%)   |\n| werkzeug            | 758.72 MB/s (63%)   | 2654.56 MB/s (43%)    |\n| django              | 788.98 MB/s (66%)   | -                     |\n| python-multipart    | 1119.54 MB/s (93%)  | 4537.55 MB/s (73%)    |\n| streaming-form-data | 1048.11 MB/s (87%)  | 4895.12 MB/s (79%)    |\n| emmett-core         | 292.10 MB/s (24%)   | -                     |\n| cgi                 | 107.48 MB/s (9%)    | -                     |\n| email               | 55.58 MB/s (5%)     | 64.37 MB/s (1%)       |\n\nNow it gets interesting! When dealing with actual file uploads, both\n`python-multipart` and `streaming-form-data` catch up and are now faster than \n`werkzeug` or `django`. All four are slower than `multipart`, but the results\nare still impressive. The line-based `cgi` and `email` parsers on the other hand\nstruggle a lot, probably because there are some line-breaks in the test file\ninput. This flaw shows even more in some of the tests below.\n\nWhat really surprised me here was the poor performance of `emmett-core`. It\nshould be the fastest parser in all scenarios (because \"Rust\") but in the first\ntest that actually moves some bytes, it falls back significantly. My best guess\nis that the context translation overhead between Python and the native Rust code\nis to blame. The parser is fed chunks of bytes and each round involves call-overhead\nand expensive copy operations. Pure python code can work directly with the\nprovided byte string and can avoid a copy in most cases. But that's just a guess.\n\n\n### Scenario: mixed\n\nA form with two text fields and two small file uploads (1MB and 2MB).\n\n| Parser              | Blocking (MB/s)     | Non-Blocking (MB/s)   |\n|---------------------|---------------------|-----------------------|\n| multipart           | 1222.65 MB/s (100%) | 7096.61 MB/s (100%)   |\n| werkzeug            | 785.09 MB/s (64%)   | 2668.64 MB/s (38%)    |\n| django              | 753.48 MB/s (62%)   | -                     |\n| python-multipart    | 961.94 MB/s (79%)   | 4593.43 MB/s (65%)    |\n| streaming-form-data | 783.71 MB/s (64%)   | 2583.91 MB/s (36%)    |\n| emmett-core         | 294.25 MB/s (24%)   | -                     |\n| cgi                 | 107.25 MB/s (9%)    | -                     |\n| email               | 68.35 MB/s (6%)     | 72.71 MB/s (1%)       |\n\nThis is the most realistic test and shows similar results to the upload\ntest above, with two notable exceptions: `python-multipart` and\n`streaming-form-data` fall back a bit and are now more close to `werkzeug` and\n`django`. `emett-core` is unexpectedly slow again, slower than most modern\npure-python parsers, but still way faster than the line-based `cgi` and `email`\nparsers. `multipart` outperforms all of them by a significant margin.\n\n\n### Scenario: worstcase_crlf\n\nA 1MB upload that contains nothing but windows line-breaks.\n\n| Parser              | Blocking (MB/s)     | Non-Blocking (MB/s)   |\n|---------------------|---------------------|-----------------------|\n| multipart           | 1277.10 MB/s (100%) | 6776.64 MB/s (100%)   |\n| werkzeug            | 862.28 MB/s (68%)   | 3930.37 MB/s (58%)    |\n| django              | 791.83 MB/s (62%)   | -                     |\n| python-multipart    | 632.24 MB/s (50%)   | 1371.32 MB/s (20%)    |\n| streaming-form-data | 48.49 MB/s (4%)     | 50.76 MB/s (1%)       |\n| emmett-core         | 295.85 MB/s (23%)   | -                     |\n| cgi                 | 3.78 MB/s (0%)      | -                     |\n| email               | 4.27 MB/s (0%)      | 4.31 MB/s (0%)        |\n\nThis is the first scenario that should not happen under normal circumstances\nbut is still an important factor if you want to prevent malicious uploads from\nslowing down your web service. `multipart`, `werkzeug`, `django` and `emett-core`\nare mostly unaffected and produce consistent results. `python-multipart` slows\ndown compared to the non-malicious tests, but still performs reasonably well.\n`streaming-form-data` seem to struggle a lot here, but not as much as the\nline-based parsers. Those choke on the high number of line-endings and are\npractically unusable.\n\n\n### Scenario: worstcase_lf\n\nA 1MB upload that contains nothing but linux line-breaks.\n\n| Parser              | Blocking (MB/s)     | Non-Blocking (MB/s)   |\n|---------------------|---------------------|-----------------------|\n| multipart           | 1269.47 MB/s (100%) | 6747.33 MB/s (100%)   |\n| werkzeug            | 844.68 MB/s (67%)   | 3675.44 MB/s (54%)    |\n| django              | 914.33 MB/s (72%)   | -                     |\n| python-multipart    | 1053.97 MB/s (83%)  | 4600.36 MB/s (68%)    |\n| streaming-form-data | 771.28 MB/s (61%)   | 2353.16 MB/s (35%)    |\n| emmett-core         | 294.77 MB/s (23%)   | -                     |\n| cgi                 | 1.71 MB/s (0%)      | -                     |\n| email               | 2.58 MB/s (0%)      | 2.61 MB/s (0%)        |\n\nLinux line breaks are not valid in segment headers or boundaries, which benefits\nparsers that do not try to be nice and parse invalid input for compatibility\nreasons. `streaming-form-data` is less affected this time and performs well. The\ntwo line-based parsers on the other hand are even worse than before. Throughput\nis roughly halved, probably because there are twice as many line-breaks (and thus\nlines) in this scenario. \n\n\n### Scenario: worstcase_bchar\n\nA 1MB upload that contains parts of the boundary.\n\n| Parser              | Blocking (MB/s)     | Non-Blocking (MB/s)   |\n|---------------------|---------------------|-----------------------|\n| multipart           | 1239.48 MB/s (100%) | 5849.90 MB/s (100%)   |\n| werkzeug            | 836.65 MB/s (68%)   | 3502.45 MB/s (60%)    |\n| django              | 791.79 MB/s (64%)   | -                     |\n| python-multipart    | 1024.75 MB/s (83%)  | 4183.56 MB/s (72%)    |\n| streaming-form-data | 767.22 MB/s (62%)   | 2346.70 MB/s (40%)    |\n| emmett-core         | 294.68 MB/s (24%)   | -                     |\n| cgi                 | 1155.09 MB/s (93%)  | -                     |\n| email               | 168.06 MB/s (14%)   | 194.91 MB/s (3%)      |\n\nThis test was originally added to show an issue with the `python-multipart`\nparser, but that was fixed quickly after reporting. There is another interesting\nanomaly, though: Since the file does not contain any newlines, `cgi` is suddenly\ncompetitive again. Its internal `file.readline(1\u003c\u003c16)` call can read large chunks\nvery quickly and the slow parser logic is triggered less often.\n\n\n### Scenario: worstcase_junk\nJunk before the first and after the last boundary (1MB each)\n\n| Parser              | Blocking (MB/s)     | Non-Blocking (MB/s)   |\n|---------------------|---------------------|-----------------------|\n| multipart           | 6434.06 MB/s (100%) | 6746.04 MB/s (100%)   |\n| werkzeug            | 23.45 MB/s (0%)     | 23.45 MB/s (0%)       |\n| django              | 993.23 MB/s (15%)   | -                     |\n| python-multipart    | 10.82 MB/s (0%)     | 10.77 MB/s (0%)       |\n| streaming-form-data | 47.15 MB/s (1%)     | 49.34 MB/s (1%)       |\n| emmett-core         | *(fails)*           | -                     |\n| cgi                 | 12.74 MB/s (0%)     | -                     |\n| email               | 3.03 MB/s (0%)      | 3.00 MB/s (0%)        |\n\nThe multipart protocol allows arbitrary junk before the first and after the last\nboundary, and requires parsers to ignore it. This protocol 'feature' has no\npractical use and no browser or HTTP client would ever do that, but parsers\nstill have to deal with it, one way or the other.\n\nWhen this was first discovered, `multipart` was the only implementation not\nshowing a drastic slowdown in this test. All the other parsers spent way too\nmuch time parsing and the discarding junk. Some were so slow that I waited for\nthe most affected libraries to release fixes before I published any results, as\nthis may be abused for denial of service attacks and qualify as a security issue.\nThe results are still really bad for most of the parsers, but not as catastrophic\nas a couple of weeks ago. Update your dependencies!\n\nYou may have noticed that the blocking `multipart` parser is almost as fast as the\nnon-blocking parser in this scenario, while the other scenarios show a way bigger\ndifference between blocking and non-blocking variants. This is because 'junk'\ndoes not emit any parser events and the blocking parts of the parser do not have\nto do much.\n\n**Note:** `emmett-core` fails here, which is good! Malicious input can and should\nbe rejected. `multipart` will also bail out very quickly in *strict* mode, but\nthese tests are run in default mode which accepts some amounts of unusual input\nfor compatibility reasons. It's still unaffected, even in non-strict mode, as it\nmanages to skip junk fast enough.\n\n\n## Conclusion\n\nAll modern pure-python parsers (`multipart`, `werkzeug`, `python-multipart`) are\nfast and behave correctly. All three offer non-blocking APIs for asnycio/ASGI\nenvironments with very little overhead and a high level of control. There are\ndifferences in API design, code quality, maturity, support and documentation,\nbut that's not the focus of this benchmark. The `django` parser is also pretty\nsolid, but hard to use outside of Django applications. \n\nFor me, both `streaming-form-data` and `emmett-core` were a bit of a surprise.\nBoth are reasonably fast for large file uploads, but not as fast as you might\nexpect from parsers written in Cython or Rust. I would have never guessed that a\npure python parser can outperform both in the upload tests. The overhead\nintroduced by those Python/native compatibility layers seems to be significant.\nThe results for those two parsers were also very different. Lessons learned:\nAlways measure. Just because something is implemented in a faster language does\nnot mean it's actually faster.\n\nI probably do not need to talk much about `email` or `cgi`. Both show mixed\nperformance and are vulnerable to malicious inputs. `cgi` is deprecated (for\ngood reasons) and `email` is not designed for form data or large uploads at all.\nBoth are unsuitable or even dangerous to use in modern web applications.\n\nAll in all, `multipart` seems to be a good choice for new projects. It's fast,\nsmall, well tested, has no dependencies and behaves correctly when presented with\nmalicious inputs. But don't just take my word for it, I'm obviously biased as the\nauthor of that library. Look at the results, look at the test cases, check out\nthe projects, try them out and make up your own mind.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefnull%2Fmultipart_bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdefnull%2Fmultipart_bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdefnull%2Fmultipart_bench/lists"}