{"id":21704131,"url":"https://github.com/so1n/fast-tools","last_synced_at":"2025-04-12T15:22:54.382Z","repository":{"id":38961850,"uuid":"271854782","full_name":"so1n/fast-tools","owner":"so1n","description":"A collection of tools available for FastApifast-tools is a FastApi/Starlette toolset, Most of the tools can be used in FastApi/Starlette, a few tools only support FastApi which is divided into the lack of compatibility with FastApi","archived":false,"fork":false,"pushed_at":"2023-08-15T15:41:43.000Z","size":529,"stargazers_count":31,"open_issues_count":5,"forks_count":8,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-26T10:01:32.536Z","etag":null,"topics":["fastapi","starlette"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/so1n.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-12T17:30:18.000Z","updated_at":"2025-03-06T04:28:31.000Z","dependencies_parsed_at":"2024-11-15T12:16:32.548Z","dependency_job_id":null,"html_url":"https://github.com/so1n/fast-tools","commit_stats":{"total_commits":178,"total_committers":2,"mean_commits":89.0,"dds":0.050561797752809,"last_synced_commit":"d329e22319296e4e35ca890b34bb6d93f55a38e3"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/so1n%2Ffast-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/so1n%2Ffast-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/so1n%2Ffast-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/so1n%2Ffast-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/so1n","download_url":"https://codeload.github.com/so1n/fast-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248147045,"owners_count":21055465,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fastapi","starlette"],"created_at":"2024-11-25T21:43:53.503Z","updated_at":"2025-04-12T15:22:54.354Z","avatar_url":"https://github.com/so1n.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fast-tools\n`fast-tools` is a `FastApi/Starlette` toolset, Most of the tools can be used in FastApi/Starlette, a few tools only support `FastApi` which is divided into the lack of compatibility with `FastApi`\n\nNote:\n    - this is alpha quality code still, the API may change, and things may fall apart while you try it.\n    - The current branch is under development and the documentation may not be unified\n\n```python\n# origin of name\nproject_name = ('FastApi'[:2] + 'Starlette'[:2]).lower() + '-tools'\nprint(project_name)  # 'fast-tools'\n```\n[中文文档](https://github.com/so1n/fast-tools/blob/master/README_CH.md)\n# Usage\n## 0.base\n- explanation: Some tool dependencies of `fast-tools` and can also be used alone\n- applicable framework:`FastApi`,`Starlette`, more....\n### 0.1.redis_helper\n- explanation: It is used to encapsulate the conn pool of aioredis and encapsulate some common commands.\n```python\nimport aioredis\nfrom fastapi import FastAPI\n\nfrom fast_tools.base import RedisHelper\n\napp: 'FastApi' = FastAPI()\nredis_helper: 'RedisHelper' = RedisHelper()  # init object\n\n\n@app.on_event(\"startup\")\nasync def startup():\n    # create redis conn pool and connect\n    redis_helper.init(await aioredis.create_pool('redis://localhost', minsize=1, maxsize=10, encoding='utf-8'))\n\napp.on_event(\"shutdown\")\nasync def shutdown():\n    # close redis conn pool\n    await redis_helper.close()\n\n\n@app.get(\"/\")\nasync def root() -\u003e dict:\n    info = await redis_helper.client.info()\n    return {\"info\": info}\n\nif __name__ == '__main__':\n    import uvicorn\n    uvicorn.run(app)\n```\n### 0.2.route_trie\nMost of python's web framework routing lookups traverse the entire routing table. If the current url matches the registered url of the route, the lookup is successful. It can be found that the time complexity of the route lookup is O(n).\nI guess the reason why the python web framework uses the traversal routing table is to support `/api/user/{user_id}` while keeping it simple.\nIt can be found that the time complexity of each route lookup is O(n). When the number of routes reaches a certain level, the matching time will becomes slower, but when we use middleware, if we need to check whether the route is matched, then It needs to be matched again, and this piece of ours can be controlled, so we need to optimize the routing matching speed here.\n\nThe fastest route matching speed is dict, but it cannot support urls similar to `/api/user/{user_id}`. Fortunately, the url matches the data structure of the trie, so the trie is used to refactor the route search, which can be as fast as possible Match the approximate area of the route, and then perform regular matching to check whether the route is correct.\n```Python\nfrom typing import (\n    List,\n    Optional\n)\nfrom fastapi import FastAPI\nfrom starlette.routing import Route\nfrom fast_tools.base import RouteTrie\n\napp: 'FastAPI' = FastAPI()\n\n\n@app.get(\"/\")\nasync def root() -\u003e dict:\n    return {\"Hello\": \"World\"}\n\n\n@app.get(\"/api/users/login\")\nasync def user_login() -\u003e str:\n    return 'ok'\n\n\nroute_trie: RouteTrie = RouteTrie()  # init route trie\nroute_trie.insert_by_app(app)  # load route from app\n\n\ndef print_route(route_list: Optional[List[Route]]):\n    \"\"\"print route list\n    \"\"\"\n    if route_list:\n        for route in route_list:\n            print(f'route:{route} url:{route.path}')\n    else:\n        print(f'route:{route_list} url: not found')\n\n# Scope param is needed to match app routing, you can learn more from the exporter example\nprint_route(route_trie.search('/'))\nprint_route(route_trie.search('/api/users/login'))\n```\n[Simply compare the efficiency of the built-in route matching and trie matching](https://github.com/so1n/fast-tools/blob/master/example/route_trie_simple_benchmarks.py)\n## 1.exporter\n- explanation: A prometheus exporter middleware that can be used for `Starlette` and `FastAPI`, which can monitor the status of each URL, such as the number of connections, the number of responses, the number of requests, the number of errors, and the number of current requests.\n- applicable framework: `FastApi`,`Starlette`\n\n### 1.1 install\npip install prometheus_client\n### 1.2 Usage\n```python\nfrom typing import Optional\n\nfrom fastapi import FastAPI\nfrom fast_tools.exporter import PrometheusMiddleware, get_metrics\nfrom fast_tools.base.route_trie import RouteTrie\n\n\napp = FastAPI()\nroute_trie = RouteTrie()\n\napp.add_middleware(\n    PrometheusMiddleware,\n    route_trie=route_trie,      # use route trie, speed up routing query\n    block_url_set={\"/metrics\"}  # not monitor url: /metrics\n)\n\napp.add_route(\"/metrics\", get_metrics)\n```\n### 1.3 example\n[example](https://github.com/so1n/fastapi-tools/blob/master/example/exporter.py)\n## 2.cbv\n- explanation: At present, due to the changes of fastapi, fastapi does not yet support cbv mode, only [fastapi_utils](https://github.com/dmontagu/fastapi-utils/blob/master/fastapi_utils/cbv.py)\nProvides cbv support, but I feel that it is not very convenient to use, so I reused its core code and made some modifications.You can use cbv like Starlette, and provide cbv_decorator to support other functions of fastapi.\n- applicable framework: `FastApi`\n\n```python\n#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n__author__ = 'so1n'\n__date__ = '2020-08'\nfrom fastapi import FastAPI, Depends, Header, Query\nfrom fast_tools.cbv import cbv_decorator, Cbv\n\napp = FastAPI()\n\n\ndef get_user_agent(user_agent: str = Header(\"User-Agent\")) -\u003e str:\n    return user_agent\n\n\nclass TestCbv(object):\n    # Don't worry about the parent attribute.\n    # Every time the get or post method is called, a new object is actually created and passed in through self.\n    # Different requests will not share the same object.\n    host: str = Header('host')\n    user_agent: str = Depends(get_user_agent)\n\n    def __init__(self, test_default_id: int = Query(123)):\n        \"\"\"support __init__ method param\"\"\"\n        self.test_default_id = test_default_id\n\n    def _response(self):\n        return {\"message\": \"hello, world\", \"user_agent\": self.user_agent, \"host\": self.host, \"id\": self.test_default_id}\n\n    @cbv_decorator(status_code=203)   # only support fastapi.route.add_api_route keywords param\n    def get(self):\n        return self._response()\n\n    def post(self):\n        return self._response()\n\n\napp.include_router(Cbv(TestCbv).router)\n\nif __name__ == '__main__':\n    import uvicorn\n    uvicorn.run(app)\n```\n## 3.config\n- explanation: config is an object that provides configuration files to be converted into python objects. config is based on `Pydantic` and Type Hints, so it can quickly convert or verify parameters without using a large amount of code.\n- applicable framework: `FastApi`,`Starlette`\n```python\nfrom typing import List, Optional\n\nfrom pydantic.fields import Json\n\nfrom fast_tools.config import Config\n\n\nclass MyConfig(Config):\n    DEBUG: bool\n    HOST: str\n    PORT: int\n\n    REDIS_ADDRESS: str\n    REDIS_PASS: Optional[str] = None  # Set the default value, if the configuration file does not have this value and does not set the default value, an error will be reported\n\n    MYSQL_DB_HOST: str\n    MYSQL_DB_NAME: str\n    MYSQL_DB_PASS: str\n    MYSQL_DB_USER: str\n    ES_HOST: Json[List]\n    TEST_LIST_INT: Json[List]\n    YML_ES_HOST: Optional[List[str]] = None\n    YML_TEST_LIST_INT: Optional[List[int]] = None\n```\nconfig supports the following parameters:\n- config_file: config file,Support ini and yml config files, f the value is empty, data is pulled from environment variables (but only a global dictionary is pulled), see [example](https://github.com/so1n/fast-tools/tree/master/example /config)\n- group: group can specify a configuration group. When using ini and yml files, multiple group configurations are supported, such as dev configuration and test configuration. If you don't want to configure this option in the code, you can directly configure group=test in the environment variable.\n- global_key: Specify that group as the global configuration. When using ini and yml files, multiple group configurations are supported, and there is also a global configuration, which can be shared by multiple groups (if the group does not have a corresponding configuration, it will be referenced to the global_key Configuration, if there is no reference)\n\nsee [example](https://github.com/so1n/fastapi-tools/blob/master/example/config/__init__.py)\n## 4.context\n- explanation:Using the characteristics of `contextvars`, you can conveniently call what you need in the route, without the need to call like requests.app.state, and it can also support type hints to facilitate writing code.\n- applicable framework: `FastApi`,`Starlette`\n\n```python\nimport asyncio\nimport httpx\nimport uuid\nfrom contextvars import (\n    copy_context,\n    Context\n)\nfrom functools import partial\nfrom fastapi import (\n    FastAPI,\n    Request,\n    Response\n)\nfrom fast_tools.context import (\n    ContextBaseModel,\n    ContextMiddleware,\n    CustomHelper,\n    HeaderHelper,\n)\n\napp = FastAPI()\nclient = httpx.AsyncClient()\n\n\nclass ContextModel(ContextBaseModel):\n    # ContextBaseModel  save data to contextvars\n    request_id: str = HeaderHelper(\n        'X-Request-Id',\n        default_func=lambda request: str(uuid.uuid4())\n    )\n    ip: str = HeaderHelper(\n        'X-Real-IP',\n        default_func=lambda request: request.client.host\n    )\n    user_agent: str = HeaderHelper('User-Agent')\n\n    # CustomHelper is a encapsulation of Context calls, and data can be read in the current context (if you want to set data, you need to instantiate it first)\n    http_client: httpx.AsyncClient = CustomHelper('http_client')\n\n    async def before_request(self, request: Request):\n        \"\"\"The method that will be called before the request is executed\"\"\"\n        self.http_client = httpx.AsyncClient()\n\n    async def after_response(self, request: Request, response: Response):\n        \"\"\"The method that will be called after the request is executed\"\"\"\n        pass\n\n    async def before_reset_context(self, request: Request, response: Response):\n        \"\"\"The method that will be called before the context is destroyed\"\"\"\n        await self.http_client.aclose()\n\n# ContextMiddleware is used to store data to ContextBaseModel before requesting, and to reset contextvars data before responding to data\napp.add_middleware(ContextMiddleware, context_model=ContextModel())\n\n\nasync def test_ensure_future():\n    print(f'test_ensure_future {ContextModel.http_client}')\n\n\ndef test_run_in_executor():\n    print(f'test_run_in_executor {ContextModel.http_client}')\n\n\ndef test_call_soon():\n    print(f'test_call_soon {ContextModel.http_client}')\n\n\n@app.get(\"/\")\nasync def root():\n    # Python will automatically copy the context\n    asyncio.ensure_future(test_ensure_future())\n    loop: 'asyncio.get_event_loop()' = asyncio.get_event_loop()\n\n    # Python will automatically copy the context\n    loop.call_soon(test_call_soon)\n\n    # When opening another thread for processing, you need to copy the context yourself\n    ctx: Context = copy_context()\n    await loop.run_in_executor(None, partial(ctx.run, test_run_in_executor))\n\n    return {\n        \"message\": ContextModel.to_dict(is_safe_return=True),  # Only return data that can be converted to json\n        \"local_ip\": (await ContextModel.http_client.get('http://icanhazip.com')).text\n    }\n\n\nif __name__ == '__main__':\n    import uvicorn\n    uvicorn.run(app)\n```\n## 5.statsd_middleware\n- explanation: The method of use is similar to exporter, but there is an additional `url_replace_handle` to handle url\n- applicable framework: `FastApi`,`Starlette`\n### 5.1install\npip install aiostatsd\n```python\nfrom typing import Optional\n\nfrom fastapi import FastAPI\nfrom fast_tools.statsd_middleware import StatsdClient, StatsdMiddleware\nfrom fast_tools.base.route_trie import RouteTrie\n\n\napp = FastAPI()\nclient = StatsdClient()\nroute_trie = RouteTrie()\n\napp.add_middleware(\n    StatsdMiddleware,\n    client=client,\n    route_trie=route_trie,\n    url_replace_handle=lambda url: url.replace('/', '_'), # Metric naming does not support'/' symbol\n    block_url_set={\"/\"}\n)\napp.on_event(\"shutdown\")(client.close)\n\n\n@app.on_event(\"startup\")\nasync def startup_event():\n    await client.connect()\n    route_trie.insert_by_app(app)\n\n\n@app.get(\"/\")\nasync def root():\n    return {\"Hello\": \"World\"}\n\n\n@app.get(\"/api/users/{user_id}/items/{item_id}\")\nasync def read_user_item(\n    user_id: int, item_id: str, q: Optional[str] = None, short: bool = False\n):\n    \"\"\"\n    copy from:https://fastapi.tiangolo.com/tutorial/query-params/#multiple-path-and-query-parameters\n    \"\"\"\n    item = {\"item_id\": item_id, \"owner_id\": user_id}\n    if q:\n        item.update({\"q\": q})\n    if not short:\n        item.update(\n            {\"description\": \"This is an amazing item that has a long description\"}\n        )\n    return item\n\n\n@app.get(\"/api/users/login\")\nasync def user_login():\n    return 'ok'\n\n\nif __name__ == '__main__':\n    import uvicorn\n    uvicorn.run(app)\n```\n## 6.task\n- explanation:The ideal architecture does not need to use `task`, so `task` is not recommended, but it may be used in the evolution of the architecture\n- applicable framework: `FastApi`,`Starlette`\n```python\nimport time\nfrom fastapi import FastAPI\nfrom fast_tools.task import background_task\nfrom fast_tools.task import stop_task\n\napp = FastAPI()\n\n# call before start\n@app.on_event(\"startup\")\n# Execute every 10 seconds\n@background_task(seconds=10)\ndef test_task() -\u003e None:\n    print(f'test.....{int(time.time())}')\n\n# stop\napp.on_event(\"shutdown\")(stop_task)\n\n\nif __name__ == '__main__':\n    import uvicorn\n    uvicorn.run(app)\n```\n## 7.cache\n- explanation: Use the return type hint of the function to adaptively cache the corresponding response, and return the cached data when the next request and the cache time has not expired.\n- applicable framework: `FastApi`,`Starlette`\n- PS: The reason for the return type prompt judgment logic is to reduce the number of judgments. When there is an IDE to write code, the return response will be the same as the return type prompt\n```python\nimport time\n\nimport aioredis\nfrom fastapi import FastAPI\nfrom starlette.responses import JSONResponse\n\nfrom fast_tools.base import RedisHelper\nfrom fast_tools.cache import (\n    cache,\n    cache_control\n)\n\n\napp = FastAPI()\nredis_helper: 'RedisHelper' = RedisHelper()\n\n\n@app.on_event(\"startup\")\nasync def startup():\n    redis_helper.init(await aioredis.create_pool('redis://localhost', minsize=1, maxsize=10, encoding='utf-8'))\n\n\n@app.on_event(\"shutdown\")\nasync def shutdown():\n    if not redis_helper.closed:\n        await redis_helper.close()\n\n\n@app.get(\"/\")\n@cache(redis_helper, 60)\nasync def root() -\u003e dict:\n    \"\"\"Read the dict data and send the corresponding response data according to the response (the default is JSONResponse)\"\"\"\n    return {\"timestamp\": time.time()}\n\n\n# adter_cache_response_listSupport the incoming function and execute it before returning the cached response. For details, see the usage method of the example\n# cache_control Will add the cache time to the http header when returning the cached response\n@app.get(\"/api/users/login\")\n@cache(redis_helper, 60, after_cache_response_list=[cache_control])\nasync def user_login() -\u003e JSONResponse:\n    \"\"\"The response type cache does not cache the entire instance, but caches the main data in the instance, and re-splices it into a new respnose the next time it returns to the cache.\"\"\"\n    return JSONResponse({\"timestamp\": time.time()})\n\n\n@app.get(\"api/null\")\n@cache(redis_helper, 60)\nasync def test_not_return_annotation():\n    \"\"\"Functions without return annotation will not be cached\"\"\"\n    return JSONResponse({\"timestamp\": time.time()})\n\n\nif __name__ == '__main__':\n    import uvicorn\n    uvicorn.run(app)\n```\n## 8.limit\n- explanation: Use common current-limiting algorithms to limit the flow of requests, and support different user groups with different flow-limiting rules. Support decorators as a single function or use middleware to limit the flow of requests that meet the URL rules. Backend supports memory-based Token bucket and redis-based token bucket, cell module, and window limit\n- applicable framework: `FastApi`,`Starlette`\n```python\nfrom typing import Optional, Tuple\n\nimport aioredis\nfrom fastapi import FastAPI, Request\nfrom fast_tools.base import RedisHelper\nfrom fast_tools import limit\n\n\ndef limit_func(requests: Request) -\u003e Tuple[str, str]:\n    \"\"\"limit needs to determine the current request key and group according to the function\"\"\"\n    return requests.session['user'], requests.session['group']\n\n\napp = FastAPI()\nredis_helper: 'RedisHelper' = RedisHelper()\n\n\n@app.on_event(\"startup\")\nasync def startup():\n    redis_helper.init(await aioredis.create_pool('redis://localhost', minsize=1, maxsize=10, encoding='utf-8'))\n\n# For requests starting with /api, the admin group can request 10 times per second, while the user group can only request once per second\napp.add_middleware(\n    limit.LimitMiddleware,\n    func=limit_func,\n    rule_dict={\n        r\"^/api\": [limit.Rule(second=1, gen_token_num=10, group='admin'), limit.Rule(second=1, group='user')]\n    }\n)\n\n\n# Each ip can only be requested once every 10 seconds\n@app.get(\"/\")\n@limit.limit(\n    [limit.Rule(second=10, gen_token_num=1)],\n    limit.backend.RedisFixedWindowBackend(redis_helper),\n    limit_func=limit.func.client_ip\n)\nasync def root():\n    return {\"Hello\": \"World\"}\n\n\n@app.get(\"/api/users/{user_id}/items/{item_id}\")\nasync def read_user_item(\n    user_id: int, item_id: str, q: Optional[str] = None, short: bool = False\n):\n    \"\"\"\n    copy from:https://fastapi.tiangolo.com/tutorial/query-params/#multiple-path-and-query-parameters\n    \"\"\"\n    item = {\"item_id\": item_id, \"owner_id\": user_id}\n    if q:\n        item.update({\"q\": q})\n    if not short:\n        item.update(\n            {\"description\": \"This is an amazing item that has a long description\"}\n        )\n    return item\n\n\n@app.get(\"/api/users/login\")\nasync def user_login():\n    return 'ok'\n\n\nif __name__ == '__main__':\n    import uvicorn\n    uvicorn.run(app)\n```\n## 9.share\n- explanation: share is used to share the same time-consuming result in multiple coroutines in the same thread, see [example](https://github.com/so1n/fast-tools/blob/master/example/share.py)\n- applicable framework: `FastApi`,`Starlette`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fso1n%2Ffast-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fso1n%2Ffast-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fso1n%2Ffast-tools/lists"}