{"id":20371314,"url":"https://github.com/arrmansa/auto-function-serving","last_synced_at":"2026-04-11T06:44:26.799Z","repository":{"id":47425675,"uuid":"515265866","full_name":"arrmansa/auto-function-serving","owner":"arrmansa","description":"A python package to offload a function call to an http server running on localhost automatically using a decorator. Compatible with multiprocessing, pickle, flask, fastapi, async etc..","archived":false,"fork":false,"pushed_at":"2022-08-23T21:55:14.000Z","size":309,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-23T20:55:41.643Z","etag":null,"topics":["fastapi","flask","http-server","multiprocessing","pypi-package","python","python-3","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arrmansa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-18T16:44:12.000Z","updated_at":"2022-07-31T15:18:10.000Z","dependencies_parsed_at":"2022-09-03T03:36:00.771Z","dependency_job_id":null,"html_url":"https://github.com/arrmansa/auto-function-serving","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/arrmansa/auto-function-serving","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arrmansa%2Fauto-function-serving","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arrmansa%2Fauto-function-serving/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arrmansa%2Fauto-function-serving/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arrmansa%2Fauto-function-serving/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arrmansa","download_url":"https://codeload.github.com/arrmansa/auto-function-serving/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arrmansa%2Fauto-function-serving/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31671629,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-10T17:19:37.612Z","status":"online","status_checked_at":"2026-04-11T02:00:05.776Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fastapi","flask","http-server","multiprocessing","pypi-package","python","python-3","python3"],"created_at":"2024-11-15T01:07:25.512Z","updated_at":"2026-04-11T06:44:26.773Z","avatar_url":"https://github.com/arrmansa.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# auto-function-serving\n\nA python package to offload a function call to an http server running on localhost automatically using a decorator. Compatible with multiprocessing, pickle, flask, fastapi, async etc..\n\n## Why\n\nImagine a case of a multi threaded or multiprocessing application where 1 or few functions are heavily resource (cpu or memory) intensive, but the other functions can run in parallel.\\\nExample - an api call followed by tokenization and classification using a large DL model followed by further API calls.\\\nIn such a case, it would make sense to create a server (generally using torchserve or tfserving) to serve requests, and replace the function call with a post request to the server.\\\nServerHandler creates a **synchronous** server and replaces any calls to the function automatically during runtime.\\\nRequests are made to 1 instance of a process running a [http.server.HTTPServer](https://docs.python.org/3/library/http.server.html) which runs the function within it.\\\nAsyncServerHandler is also available which makes the requests asynchronously.\\\nEven calls made from different processes, threads, multiprocessing, flask, FastApi and async event loops are made to the same server process.\n\n## Usage\n\nIn general : \n```\nsome code with a callable\n```\ncan be replaced with an instance of Either ServerHandler or AsyncserverHandler that accepts the code as a string in it's first argument and the name of the callable as the second argument.\n```python\nfrom auto_function_serving.ServerHandler import ServerHandler\ncallable_name = ServerHandler(\"\"\"\nsome independent code with a callable\n\"\"\", \"callable_name\")\n```\nExample :\n```python\nimport module1\nimport module2\ndef functionname(someinput):\n    a = module1.function1(someinput)\n    return module2.function2(a)\n```\ncan be replaced with\n```python\nfrom auto_function_serving.ServerHandler import AsyncserverHandler\nfunctionname = AsyncServerHandler(\"\"\"\nimport module1\nimport module2\ndef functionname(someinput):\n    a = module1.function1(someinput)\n    return module2.function2(a)\n\"\"\", \"functionname\", port=\"Any\")\n```\nDecorators (@AsyncserverHandler.decorator and @ServerHandler.decorator) and AsyncServerHandler details in more usage.\n\n## Arguments\n\n```python\nfrom auto_function_serving.ServerHandler import ServerHandler\ncallable_name = ServerHandler(\"\"\"\nsome independent code with a callable\n\"\"\", \"callable_name\", port=None, backend='Popen', wait=100, backlog = 1024))\n```\n1. port\n    * if None, then the input code is hashed and a port is chosen from 50000 to 60000 using the hash\n    * if int, then int is chosen\n    * otherwise, a random open port is chosen\n2. backend - either 'Popen' or 'multiprocessing'. Popen Should be used in general.\n3. wait - approx max number of seconds to wait for the server to run. No waiting done if set to 0, default 100\n4. backlog - max number of backlogged requests before returning errors, python default is 5, but default in ServerHandler is 1024.\n\n## Features\n\nruns [http.server.HTTPServer](https://docs.python.org/3/library/http.server.html).\\\nServerHandler and AsyncServerHandler objects can be loaded and unloaded with pickle.\\\nUses Popen or multiprocessing to run the server.\\\nUses only a single external dependency (aiohttp), and only for async.\\\nhttp, not https.\\\nchooses a port based on hash of input. (unless specified otherwise)\n\n### Advantages\n\nMinimal code changes.\\\nShould be compatible with almost all functions in almost all CPython envs. (Not sure where it could fail? Please add an issue if you find one.)\\\nMemory leaks or errors (from the server) are extremely unlikely since it is minimal, single threaded, single process and a default component of python stdlib.\\\nExceptions cause 5xx errors without closing the server.\\\nEven Separate Processes will make requests to 1 instance of the same server unless specified otherwise. (Because it's looking for a server on a specific port.).\\\nCan specify otherwise by set the port to any free port so that a new ServerHandler object starts a new server.\\\nhttp post requests : lightweight, few ms overhead, reliable.\\\nAsync is a good feature.\\\nnow with tests.\n\n### Disadvatages\n\nHaving a string of code as an argument to a class is not pythonic, unless the decorator is used.\\\nImporting inside functions is not ideal, even when the decorator is used.\\\nhttp post requests : insecure, few ms overhead.\\\nExceptions inside the server are not sent back.\\\nNo batching.\\\nNo inbuilt logging. (Could be added).\nInitialization delay of upto few seconds to start the server.\nAsync functions will not work on the server.\n\n#### Possible Edge cases\n\nNo auto server restart in case server closes.\\\nMay leave some resources locked for a while (\u003c1min) if not closed properly.\\\nProblems might occur if Popen or multiprocessing are not available.\\\nPossible nested async errors with jupyter or other? Please look into [nest-asyncio](https://pypi.org/project/nest-asyncio/) and the [iss](https://github.com/python/cpython/issues/93462)[ues](https://github.com/python/cpython/issues/66435).\\\nWarnings from somewhat hacky (but legit and completely functional) workarounds. \\\nClosing of server process in __del__ and atexit.redister(__del__) fail for some reason (tested and unlikely).\n\n\n## Installation\n\nUse the package manager pip to install [auto_function_serving](https://pypi.org/project/auto-function-serving/)\n```bash\npip install auto_function_serving\n```\n\n## How does this work?\n\nCode for the server is stored in [ServerHandler](https://github.com/arrmansa/auto-function-serving/blob/main/src/auto_function_serving/ServerHandler.py).base_code and some string formatting is used to fill in the blanks.\\\nThe server process is started with Popen (or multiprocessing if specified). The first thing it does is import socket and bind the port - if it's not available the code stops after an exception. Therefore only 1 instance of the server runs at a time on a machine.\\\nWe know the function is ready after we can receive a valid get request from the server.\\\nInputs and outputs are sent as bytes, converted to and from objects using pickle.\\\nIf port is None in while initializing (default), a port from 50000 to 60000 is chosen by hashing the input code to make it independent of the source of a function. Collisions of different functions are possible, but unlikely. The collision of the same function in multiple processes is used to make sure only 1 server process runs at a time. The port can be specified if needed.\n\n## Performance (On my machine)\n\noverhead for small input and output (few bytes) - \\\n~2ms for requests with urllib.request\\\n~4ms for async requests with aiohttp.ClientSession \\\noverhead for large input and output\\\n~10ms for 0.5 mb input and output (1mb total transfer).\\\n~60ms for 5 mb input and output (10 mb total transfer).\\\n~600ms for 50 mb input and output (100 mb total transfer).\n\n## More Usage\n\nIt can also be used with the provided decorator for functions with no dependencies outside the function.\n```python\nfrom auto_function_serving.ServerHandler import ServerHandler\n@ServerHandler.decorator\ndef someheavyfunction(args,**kwargs):\n    for i in range(big_number)\n        someexpensivecomputation\n```\nimports inside the function will work\n```python\nfrom auto_function_serving.ServerHandler import ServerHandler\n@ServerHandler.decorator\ndef someheavyfunction(args,**kwargs):\n    import numpy as np\n```\n```python\nfrom auto_function_serving.ServerHandler import ServerHandler\n@ServerHandler.decorator\ndef someheavyfunction(args,**kwargs):\n    if not hasattr(someheavyfunction,'RunOnce'):\n\t    global np\n        import numpy as np\n    setattr(someheavyfunction,'RunOnce',None)\n\t... etc\n```\n\nWhen the somemodule does not have any expensive global loading.\n```python\nfrom auto_function_serving.ServerHandler import ServerHandler\nfrom somemodule import someheavyfunction\nsomeheavyfunction = ServerHandler.decorator(someheavyfunction)\n```\nIp address can be changed by setting ServerHandler.ip_address (default \"127.0.0.1\") before creating a new instance.\n\n### AsyncServerHandler\n\nAsyncServerHandler is also available which uses [aiohttp](https://docs.aiohttp.org/) to make the requests asynchronously, for use with fastapi and other async use cases. \\\nAsyncServerHandler has the same usage as ServerHandler, except calls need to be awaited or used with asyncio.run() or with asyncio.get_event_loop().run_until_complete().\\\nNumber of async calls can be limited by setting AsyncServerHandler.TCPConnector_limit which controls the [TCPconnector](https://docs.aiohttp.org/en/stable/client_reference.html?highlight=connector#aiohttp.TCPConnector) limit (default 100). Using [Semaphore](https://docs.python.org/3/library/asyncio-sync.html#asyncio.Semaphore) is also something to consider.\n\n## Other things to look into\nLibraries : Celery, Tfserving, Torchserve, Flask\\\nSending globals and locals to exec\\\nast trees\n\n## Contributing\nPull requests are welcome.\n\n## License\n[Apache License 2.0](https://choosealicense.com/licenses/apache-2.0/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farrmansa%2Fauto-function-serving","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farrmansa%2Fauto-function-serving","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farrmansa%2Fauto-function-serving/lists"}