{"id":21501181,"url":"https://github.com/rapidrabbit76/fastapi-deep-learning-model-micro-batching-serving","last_synced_at":"2025-06-24T06:08:06.878Z","repository":{"id":139026363,"uuid":"531113452","full_name":"rapidrabbit76/FastAPI-Deep-Learning-Model-Micro-Batching-Serving","owner":"rapidrabbit76","description":"FastAPI pytorch model serving with micro batching ","archived":false,"fork":false,"pushed_at":"2022-08-31T14:13:26.000Z","size":57,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-23T18:47:43.287Z","etag":null,"topics":["fastapi","model-serving","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rapidrabbit76.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-08-31T14:10:23.000Z","updated_at":"2024-03-15T00:28:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"77d33849-9b8c-42c9-bf64-eb84585bd172","html_url":"https://github.com/rapidrabbit76/FastAPI-Deep-Learning-Model-Micro-Batching-Serving","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rapidrabbit76/FastAPI-Deep-Learning-Model-Micro-Batching-Serving","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapidrabbit76%2FFastAPI-Deep-Learning-Model-Micro-Batching-Serving","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapidrabbit76%2FFastAPI-Deep-Learning-Model-Micro-Batching-Serving/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapidrabbit76%2FFastAPI-Deep-Learning-Model-Micro-Batching-Serving/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapidrabbit76%2FFastAPI-Deep-Learning-Model-Micro-Batching-Serving/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rapidrabbit76","download_url":"https://codeload.github.com/rapidrabbit76/FastAPI-Deep-Learning-Model-Micro-Batching-Serving/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapidrabbit76%2FFastAPI-Deep-Learning-Model-Micro-Batching-Serving/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261618130,"owners_count":23185095,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fastapi","model-serving","pytorch"],"created_at":"2024-11-23T17:49:48.417Z","updated_at":"2025-06-24T06:08:06.867Z","avatar_url":"https://github.com/rapidrabbit76.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FastAPI Model Micro Batching Serving Sample using Service Streamer\n\nFastAPI DL Model Serving Sample\n\nmicro batch  depens on [Service Streamer](https://github.com/ShannonAI/service-streamer)\n\n\n\n![](./src/01.png)\n\n\n\n\n# Project Structur\n\n```\n.\n├── app_v1\n├── app_v2\n│   ├── manager # model micro batch manager\n│   │   ├── __init__.py\n│   │   └── resnet.py\n│   ├── routers\n│   │   ├── __init__.py\n│   │   └── resnet.py\n│   ├── schema.py\n│   ├── services\n│   │   ├── __init__.py\n│   │   └── resnet.py\n│   ├── dependencies.py\n│   ├── main.py\n│   └── settings.py\n├── imagenet_classes.txt\n├── model_store\n│   ├── classifier.zip\n│   └── embedding.zip\n├── README.md\n└── requirements.txt\n\n```\n\n# docker-compose \n\n```yaml\n\nversion: \"3.8\"\n\nservices:\n  api:\n    build: .\n    restart: unless-stopped\n    command: uvicorn app_v2.main:app --host 0.0.0.0  --workers 1 --port 8000 --log-config ./log.ini\n    ports:\n      - \"8000:8000\"\n    expose:\n      - 8000\n    volumes:\n      - ./log:/app/log:rw\n\n    environment:\n      DL_EMBEDDING_MODEL_PATH: model_store/embedding.zip\n      DL_CLASSIFIER_MODEL_PATH: model_store/classifier.zip\n      CLASSES_PATH: imagenet_classes.txt\n\n      MB_BATCH_SIZE: 256\n      MB_MAX_LATENCY: 1 # Sec\n      MB_WORKER_NUM: 1\n\n      CUDA_DEVICE: cuda\n      CUDA_DEVICES: \"[0]\"\n\n    deploy:\n      resources:\n        reservations:\n          devices:\n            - driver: nvidia\n              device_ids: [ '0' ]\n              capabilities: [ gpu ]\n\n    healthcheck:\n      test: curl --fail http://localhost:8000/health || exit 1\n      interval: 60s\n      retries: 5\n      start_period: 30s\n      timeout: 5s\n\n\n```\n\n\n# Requirements\n\n- torch==1.10.1\n- torchvision==0.11.2\n- fastapi[all]==0.80.0\n- fastapi-restful==0.4.3\n- fastapi-health==0.4.0\n- service-streamer==0.1.2\n- pydantic==1.9.2\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frapidrabbit76%2Ffastapi-deep-learning-model-micro-batching-serving","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frapidrabbit76%2Ffastapi-deep-learning-model-micro-batching-serving","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frapidrabbit76%2Ffastapi-deep-learning-model-micro-batching-serving/lists"}