{"id":13935614,"url":"https://github.com/eora-ai/inferoxy","last_synced_at":"2025-07-19T20:33:25.878Z","repository":{"id":48655870,"uuid":"371807350","full_name":"eora-ai/inferoxy","owner":"eora-ai","description":"Service for quick deploying and using dockerized Computer Vision models","archived":false,"fork":false,"pushed_at":"2021-07-15T15:11:16.000Z","size":9767,"stargazers_count":94,"open_issues_count":9,"forks_count":3,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-08-08T23:21:19.645Z","etag":null,"topics":["computer-vision","machine-learning","mlops","pipelines","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eora-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-28T20:04:41.000Z","updated_at":"2024-05-01T15:25:44.000Z","dependencies_parsed_at":"2022-09-11T03:00:38.325Z","dependency_job_id":null,"html_url":"https://github.com/eora-ai/inferoxy","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eora-ai%2Finferoxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eora-ai%2Finferoxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eora-ai%2Finferoxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eora-ai%2Finferoxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eora-ai","download_url":"https://codeload.github.com/eora-ai/inferoxy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226677155,"owners_count":17666013,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","machine-learning","mlops","pipelines","python"],"created_at":"2024-08-07T23:01:55.915Z","updated_at":"2024-11-27T03:30:54.215Z","avatar_url":"https://github.com/eora-ai.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n![Inferoxy](docs/inferoxy_dark.png)\n\n\u003c/div\u003e\n\n[![codecov](https://codecov.io/gh/eora-ai/inferoxy/branch/master/graph/badge.svg?token=HV6ZNPUFZE)](https://codecov.io/gh/eora-ai/inferoxy)\n\n## What is it?\n\nInferoxy is a service for quick deploying and using dockerized Computer Vision models. \nIt's a core of EORA's Computer Vision platform [Vision Hub](https://www.visionhub.ru/) that runs on top of AWS EKS.\n\n## Why use it?\n\nYou should use it if:\n- You want to simplify deploying Computer Vision models with an appropriate Data Science stack to production: \n  all you need to do is to build a Docker image \n  with your model including any pre- and post-processing steps and push it into an accessible registry\n- You have only one machine or cluster for inference (CPU/GPU)\n- You want automatic batching for multi-GPU/multi-node setup\n- Model versioning\n\n## Architecture\n![Overall architecture](docs/inferoxy-general.png)\n\nInferoxy is built using message broker pattern.\n- Roughly speaking, it accepts user requests through different interfaces\nwhich we call \"bridges\". Multiple bridges can run simultaneously. Current supported bridges are REST API, gRPC and \nZeroMQ\n- The requests are carefully split into batches and processed on a single multi-GPU machine or a multi-node cluster\n- The models to be deployed are managed through Model Manager that \ncommunicates with Redis to store/retrieve models information such as Docker image URL, maximum batch size value, etc.\n\n### Batching\n![Batching](docs/inferoxy-batching.png)\n\nOne of the core Inferoxy's features is the batching mechanism.\n- For batch processing it's taken into consideration \nthat different models can utilize different batch sizes and that some models can process a series of batches \nfrom a specific user, e.g. for video processing tasks. The latter models are called \"stateful\" models while models \nwhich don't depend on user state are called \"stateless\"\n- Multiple copies of the same model can run on different machines while only one copy can run on the same GPU device. \n  So, to increase models efficiency it's recommended to set batch size for models to be as high as possible\n- A user of the stateful model reserves the whole copy of the model and releases it when his task is finished.\n- Users of the stateless models can use the same copy of the model simultaneously\n- Numpy tensors of RGB images with metadata are all going through ZeroMQ to the models and the results are also read \n  from ZeroMQ socket\n  \n### Cluster management\n![Cluster](docs/inferoxy-cluster.png)\n\nThe cluster management consists of keeping track of the running copies of the models, load analysis, \nhealth checking and alerting.\n\n## Requirements\nYou can run Inferoxy locally on a single machine or [k8s](https://kubernetes.io/) cluster. \nTo run Inferoxy, you should have a minimum of **4GB RAM** and CPU or GPU device depending on your speed/cost trade-off. \n\n## Basic commands\n\n## Local run\nTo run locally you should use Inferoxy Docker image. The last version you can find \n[here](https://github.com/eora-ai/inferoxy/releases).\n```bash\ndocker pull public.registry.visionhub.ru/inferoxy:v1.0.4\n```\nAfter image is pulled we need to make basic configuration using `.env` file\n```env\n# .env\nCLOUD_CLIENT=docker\nTASK_MANAGER_DOCKER_CONFIG_NETWORK=inferoxy\nTASK_MANAGER_DOCKER_CONFIG_REGISTRY=\nTASK_MANAGER_DOCKER_CONFIG_LOGIN=\nTASK_MANAGER_DOCKER_CONFIG_PASSWORD=\nMODEL_STORAGE_DATABASE_HOST=redis\nMODEL_STORAGE_DATABASE_PORT=6379\nMODEL_STORAGE_DATABASE_NUMBER=0\nLOGGING_LEVEL=INFO\n```\nThe next step is to create `inferoxy` Docker network.\n```bash\ndocker network create inferoxy\n```\nNow we should run Redis in this network. Redis is needed to store information about your models.\n```bash\ndocker run --network inferoxy --name redis redis:latest \n```\nCreate `models.yaml` file with simple set of models. You can read about `models.yaml` in \n[documentation](https://github.com/eora-ai/inferoxy/wiki)\n```yaml\nstub:\n  address: public.registry.visionhub.ru/models/stub:v5\n  batch_size: 256\n  run_on_gpu: False\n  stateless: True\n```\n\nNow we can start Inferoxy:\n```bash\ndocker run --env-file .env \n\t-v /var/run/docker.sock:/var/run/docker.sock \\\n\t-p 7787:7787 -p 7788:7788 -p 8000:8000 -p 8698:8698\\\n\t--name inferoxy --rm \\\n\t--network inferoxy \\\n\t-v $(pwd)/models.yaml:/etc/inferoxy/models.yaml \\\n\tpublic.registry.visionhub.ru/inferoxy:${INFEROXY_VERSION}\n```\n\n## Documentation\n\nYou can find the full documentation [here](https://github.com/eora-ai/inferoxy/wiki)\n\n## Discord\nJoin our community in [Discord server](https://discord.gg/ASbWzUZuVp) to discuss stuff \nrelated to Inferoxy usage and development\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feora-ai%2Finferoxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feora-ai%2Finferoxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feora-ai%2Finferoxy/lists"}