{"id":13499992,"url":"https://github.com/unum-cloud/ucall","last_synced_at":"2025-04-10T20:11:58.581Z","repository":{"id":102791809,"uuid":"584714008","full_name":"unum-cloud/ucall","owner":"unum-cloud","description":"Web Serving and Remote Procedure Calls at 50x lower latency and 70x higher bandwidth than FastAPI, implementing JSON-RPC \u0026 REST over io_uring ☎️","archived":false,"fork":false,"pushed_at":"2025-01-13T15:18:42.000Z","size":4169,"stargazers_count":1201,"open_issues_count":13,"forks_count":46,"subscribers_count":20,"default_branch":"main","last_synced_at":"2025-04-02T13:18:12.371Z","etag":null,"topics":["backend","cpython","dpdk","epoll","fast-api","flask","http","http-server","io-uring","json","json-rpc","liburing","linux-kernel","python","rest-api","rpc","rpc-framework","simd","tcp","tcp-ip"],"latest_commit_sha":null,"homepage":"https://unum-cloud.github.io/ucall/","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/unum-cloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-03T10:23:50.000Z","updated_at":"2025-04-01T07:29:07.000Z","dependencies_parsed_at":"2023-11-29T17:29:40.669Z","dependency_job_id":"23aead5b-1dc9-4519-9b2b-62fb7375bdce","html_url":"https://github.com/unum-cloud/ucall","commit_stats":{"total_commits":432,"total_committers":12,"mean_commits":36.0,"dds":0.5277777777777778,"last_synced_commit":"5c1ab1ff9b9a27cf5ee4a70fb3d4dde7a4ed0c7f"},"previous_names":["unum-cloud/ujrpc"],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unum-cloud%2Fucall","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unum-cloud%2Fucall/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unum-cloud%2Fucall/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/unum-cloud%2Fucall/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/unum-cloud","download_url":"https://codeload.github.com/unum-cloud/ucall/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248288361,"owners_count":21078903,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend","cpython","dpdk","epoll","fast-api","flask","http","http-server","io-uring","json","json-rpc","liburing","linux-kernel","python","rest-api","rpc","rpc-framework","simd","tcp","tcp-ip"],"created_at":"2024-07-31T22:00:49.407Z","updated_at":"2025-04-10T20:11:58.557Z","avatar_url":"https://github.com/unum-cloud.png","language":"C","readme":"\u003ch1 align=\"center\"\u003eUCall\u003c/h1\u003e\n\u003ch3 align=\"center\"\u003e\nJSON Remote Procedure Calls Library\u003cbr/\u003e\nUp to 100x Faster than FastAPI\u003cbr/\u003e\n\u003c/h3\u003e\n\u003cbr/\u003e\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://discord.gg/xuDmpbEDnQ\"\u003e\u003cimg height=\"25\" src=\"https://github.com/unum-cloud/ukv/raw/main/assets/icons/discord.svg\" alt=\"Discord\"\u003e\u003c/a\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://www.linkedin.com/company/unum-cloud/\"\u003e\u003cimg height=\"25\" src=\"https://github.com/unum-cloud/ukv/raw/main/assets/icons/linkedin.svg\" alt=\"LinkedIn\"\u003e\u003c/a\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://twitter.com/unum_cloud\"\u003e\u003cimg height=\"25\" src=\"https://github.com/unum-cloud/ukv/raw/main/assets/icons/twitter.svg\" alt=\"Twitter\"\u003e\u003c/a\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://unum.cloud/post\"\u003e\u003cimg height=\"25\" src=\"https://github.com/unum-cloud/ukv/raw/main/assets/icons/blog.svg\" alt=\"Blog\"\u003e\u003c/a\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\n\u003ca href=\"https://github.com/unum-cloud/ucall\"\u003e\u003cimg height=\"25\" src=\"https://github.com/unum-cloud/ukv/raw/main/assets/icons/github.svg\" alt=\"GitHub\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\nMost modern networking is built either on slow and ambiguous REST APIs or unnecessarily complex gRPC.\nFastAPI, for example, looks very approachable.\nWe aim to be equally or even simpler to use.\n\n\u003ctable width=\"100%\"\u003e\n\u003ctr\u003e\n\u003cth width=\"50%\"\u003eFastAPI\u003c/th\u003e\u003cth width=\"50%\"\u003eUCall\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```sh\npip install fastapi uvicorn\n```\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n```sh\npip install ucall\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```python\nfrom fastapi import FastAPI\nimport uvicorn\n\nserver = FastAPI()\n\n@server.get('/sum')\ndef sum(a: int, b: int):\n    return a + b\n\nuvicorn.run(...)    \n```\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n```python\nfrom ucall.posix import Server\n# from ucall.uring import Server on 5.19+\n\nserver = Server()\n\n@server\ndef sum(a: int, b: int):\n    return a + b\n\nserver.run()    \n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\nIt takes over a millisecond to handle a trivial FastAPI call on a recent 8-core CPU.\nIn that time, light could have traveled 300 km through optics to the neighboring city or country, in my case.\nHow does UCall compare to FastAPI and gRPC?\n\n| Setup                   |   🔁   | Server | Latency w 1 client | Throughput w 32 clients |\n| :---------------------- | :---: | :----: | -----------------: | ----------------------: |\n| Fast API over REST      |   ❌   |   🐍    |           1'203 μs |               3'184 rps |\n| Fast API over WebSocket |   ✅   |   🐍    |              86 μs |            11'356 rps ¹ |\n| gRPC ²                  |   ✅   |   🐍    |             164 μs |               9'849 rps |\n|                         |       |        |                    |                         |\n| UCall with POSIX        |   ❌   |   C    |              62 μs |              79'000 rps |\n| UCall with io_uring     |   ✅   |   🐍    |              40 μs |             210'000 rps |\n| UCall with io_uring     |   ✅   |   C    |              22 μs |             231'000 rps |\n\n\u003cdetails\u003e\n  \u003csummary\u003eTable legend\u003c/summary\u003e\n\nAll benchmarks were conducted on AWS on general purpose instances with **Ubuntu 22.10 AMI**.\nIt is the first major AMI to come with **Linux Kernel 5.19**, featuring much wider `io_uring` support for networking operations.\nThese specific numbers were obtained on `c7g.metal` beefy instances with Graviton 3 chips.\n\n- The 🔁 column marks, if the TCP/IP connection is being reused during subsequent requests.\n- The \"server\" column defines the programming language, in which the server was implemented.\n- The \"latency\" column report the amount of time between sending a request and receiving a response. μ stands for micro, μs subsequently means microseconds.\n- The \"throughput\" column reports the number of Requests Per Second when querying the same server application from multiple client processes running on the same machine.\n\n\u003e ¹ FastAPI couldn't process concurrent requests with WebSockets.\n\n\u003e ² We tried generating C++ backends with gRPC, but its numbers, suspiciously, weren't better. There is also an async gRPC option, that wasn't tried.\n\n\u003c/details\u003e\n\n## How is that possible?!\n\nHow can a tiny pet-project with just a couple thousand lines of code compete with two of the most established networking libraries?\n**UCall stands on the shoulders of Giants**:\n\n- `io_uring` for interrupt-less IO.\n  - `io_uring_prep_read_fixed` on 5.1+.\n  - `io_uring_prep_accept_direct` on 5.19+.\n  - `io_uring_register_files_sparse` on 5.19+.\n  - `IORING_SETUP_COOP_TASKRUN` optional on 5.19+.\n  - `IORING_SETUP_SINGLE_ISSUER` optional on 6.0+.\n\n- SIMD-accelerated parsers with manual memory control.\n  - [`simdjson`][simdjson] to parse JSON faster than gRPC can unpack `ProtoBuf`.\n  - [`Turbo-Base64`][base64] to decode binary values from a `Base64` form.\n  - [`picohttpparser`][picohttpparser] to navigate HTTP headers.\n\nYou have already seen the latency of the round trip..., the throughput in requests per second..., want to see the bandwidth?\nTry yourself!\n\n```python\n@server\ndef echo(data: bytes):\n    return data\n```\n\n## More Functionality than FastAPI\n\nFastAPI supports native type, while UCall supports `numpy.ndarray`, `PIL.Image` and other custom types.\nThis comes handy when you build real applications or want to deploy Multi-Modal AI, like we do with [UForm](https://github.com/unum-cloud/uform).\n\n```python\nfrom ucall.rich_posix import Server\nimport uform\n\nserver = Server()\nmodel = uform.get_model('unum-cloud/uform-vl-multilingual')\n\n@server\ndef vectorize(description: str, photo: PIL.Image.Image) -\u003e numpy.ndarray:\n    image = model.preprocess_image(photo)\n    tokens = model.preprocess_text(description)\n    joint_embedding = model.encode_multimodal(image=image, text=tokens)\n\n    return joint_embedding.cpu().detach().numpy()\n```\n\nWe also have our own optional `Client` class that helps with those custom types.\n\n```python\nfrom ucall.client import Client\n\nclient = Client()\n# Explicit JSON-RPC call:\nresponse = client({\n    'method': 'vectorize',\n    'params': {\n        'description': description,\n        'image': image,\n    },\n    'jsonrpc': '2.0',\n    'id': 100,\n})\n# Or the same with syntactic sugar:\nresponse = client.vectorize(description=description, image=image) \n```\n\n## CLI like [cURL](https://curl.se/docs/manpage.html)\n\nAside from the Python `Client`, we provide an easy-to-use Command Line Interface, which comes with `pip install ucall`.\nIt allow you to call a remote server, upload files, with direct support for images and NumPy arrays.\nTranslating previous example into a Bash script, to call the server on the same machine:\n\n```sh\nucall vectorize description='Product description' -i image=./local/path.png\n```\n\nTo address a remote server:\n\n```sh\nucall vectorize description='Product description' -i image=./local/path.png --uri 0.0.0.0 -p 8545\n```\n\nTo print the docs, use `ucall -h`:\n\n```txt\nusage: ucall [-h] [--uri URI] [--port PORT] [-f [FILE ...]] [-i [IMAGE ...]] [--positional [POSITIONAL ...]] method [kwargs ...]\n\nUCall Client CLI\n\npositional arguments:\n  method                method name\n  kwargs                method arguments\n\noptions:\n  -h, --help            show this help message and exit\n  --uri URI             server uri\n  --port PORT           server port\n  -f [FILE ...], --file [FILE ...]\n                        method positional arguments\n  -i [IMAGE ...], --image [IMAGE ...]\n                        method positional arguments\n  --positional [POSITIONAL ...]\n                        method positional arguments\n```\n\nYou can also explicitly annotate types, to distinguish integers, floats, and strings, to avoid ambiguity.\n\n```\nucall auth id=256\nucall auth id:int=256\nucall auth id:str=256\n```\n\n## Free Tier Throughput\n\nWe will leave bandwidth measurements to enthusiasts, but will share some more numbers.\nThe general logic is that you can't squeeze high performance from Free-Tier machines.\nCurrently AWS provides following options: `t2.micro` and `t4g.small`, on older Intel and newer Graviton 2 chips.\nThis library is so fast, that it doesn't need more than 1 core, so you can run a fast server even on a tiny Free-Tier server!\n\n| Setup                   |   🔁   | Server | Clients | `t2.micro` | `t4g.small` |\n| :---------------------- | :---: | :----: | :-----: | ---------: | ----------: |\n| Fast API over REST      |   ❌   |   🐍    |    1    |    328 rps |     424 rps |\n| Fast API over WebSocket |   ✅   |   🐍    |    1    |  1'504 rps |   3'051 rps |\n| gRPC                    |   ✅   |   🐍    |    1    |  1'169 rps |   1'974 rps |\n|                         |       |        |         |            |             |\n| UCall with POSIX        |   ❌   |   C    |    1    |  1'082 rps |   2'438 rps |\n| UCall with io_uring     |   ✅   |   C    |    1    |          - |   5'864 rps |\n| UCall with POSIX        |   ❌   |   C    |   32    |  3'399 rps |  39'877 rps |\n| UCall with io_uring     |   ✅   |   C    |   32    |          - |  88'455 rps |\n\nIn this case, every server was bombarded by requests from 1 or a fleet of 32 other instances in the same availability zone.\nIf you want to reproduce those benchmarks, check out the [`sum` examples on GitHub][sum-examples].\n\n## Quick Start\n\nFor Python:\n\n```sh\npip install ucall\n```\n\nFor CMake projects:\n\n```cmake\ninclude(FetchContent)\nFetchContent_Declare(\n    ucall\n    GIT_REPOSITORY https://github.com/unum-cloud/ucall\n    GIT_SHALLOW TRUE\n)\nFetchContent_MakeAvailable(ucall)\ninclude_directories(${ucall_SOURCE_DIR}/include)\n```\n\nThe C usage example is mouthful compared to Python.\nWe wanted to make it as lightweight as possible and to allow optional arguments without dynamic allocations and named lookups.\nSo unlike the Python layer, we expect the user to manually extract the arguments from the call context with `ucall_param_named_i64()`, and its siblings.\n\n```c\n#include \u003ccstdio.h\u003e\n#include \u003cucall/ucall.h\u003e\n\nstatic void sum(ucall_call_t call, ucall_callback_tag_t) {\n    int64_t a{}, b{};\n    char printed_sum[256]{};\n    bool got_a = ucall_param_named_i64(call, \"a\", 0, \u0026a);\n    bool got_b = ucall_param_named_i64(call, \"b\", 0, \u0026b);\n    if (!got_a || !got_b)\n        return ucall_call_reply_error_invalid_params(call);\n\n    int len = snprintf(printed_sum, 256, \"%ll\", a + b);\n    ucall_call_reply_content(call, printed_sum, len);\n}\n\nint main(int argc, char** argv) {\n\n    ucall_server_t server{};\n    ucall_config_t config{};\n\n    ucall_init(\u0026config, \u0026server);\n    ucall_add_procedure(server, \"sum\", \u0026sum, NULL);\n    ucall_take_calls(server, 0);\n    ucall_free(server);\n    return 0;\n}\n```\n\n## Roadmap\n\n- [x] Batch Requests\n- [x] JSON-RPC over raw TCP sockets\n- [x] JSON-RPC over TCP with HTTP\n- [x] Concurrent sessions\n- [x] NumPy `array` and Pillow serialization\n- [ ] HTTP**S** support\n- [ ] Batch-capable endpoints for ML\n- [ ] Zero-ETL relay calls\n- [ ] Integrating with [UKV][ukv]\n- [ ] WebSockets for web interfaces\n- [ ] AF_XDP and UDP-based analogs on Linux\n\n\u003e Want to affect the roadmap and request a feature? Join the discussions on Discord.\n\n## Why JSON-RPC?\n\n- Transport independent: UDP, TCP, bring what you want.\n- Application layer is optional: use HTTP or not.\n- Unlike REST APIs, there is just one way to pass arguments.\n\n[simdjson]: https://github.com/simdjson/simdjson\n[base64]: https://github.com/powturbo/Turbo-Base64\n[picohttpparser]: https://github.com/h2o/picohttpparser\n[sum-examples]: https://github.com/unum-cloud/ucall/tree/dev/examples/sum\n[ukv]: https://github.com/unum-cloud/ukv\n","funding_links":[],"categories":["Networking","C","RPC Utilities"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funum-cloud%2Fucall","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Funum-cloud%2Fucall","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funum-cloud%2Fucall/lists"}