{"id":15805258,"url":"https://github.com/ibaigorordo/sapiens-pytorch-inference","last_synced_at":"2025-04-04T09:10:01.909Z","repository":{"id":257806480,"uuid":"848222641","full_name":"ibaiGorordo/Sapiens-Pytorch-Inference","owner":"ibaiGorordo","description":"Minimal code and examnples for inferencing Sapiens foundation human models in Pytorch","archived":false,"fork":false,"pushed_at":"2024-10-26T06:56:35.000Z","size":49897,"stargazers_count":133,"open_issues_count":6,"forks_count":14,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-28T13:12:42.734Z","etag":null,"topics":["computer-vision","deep-learning","depth-estimation","inference","pose-estimation","python","pytorch","semantic-segmentation"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/sapiens-inferece/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ibaiGorordo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-27T11:14:44.000Z","updated_at":"2025-03-19T07:42:48.000Z","dependencies_parsed_at":"2024-12-28T20:17:55.248Z","dependency_job_id":null,"html_url":"https://github.com/ibaiGorordo/Sapiens-Pytorch-Inference","commit_stats":{"total_commits":34,"total_committers":2,"mean_commits":17.0,"dds":"0.11764705882352944","last_synced_commit":"a0bec2e524ef825fb269b691b32353426e93b5d0"},"previous_names":["ibaigorordo/sapiens-pytorch-inference"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibaiGorordo%2FSapiens-Pytorch-Inference","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibaiGorordo%2FSapiens-Pytorch-Inference/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibaiGorordo%2FSapiens-Pytorch-Inference/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibaiGorordo%2FSapiens-Pytorch-Inference/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ibaiGorordo","download_url":"https://codeload.github.com/ibaiGorordo/Sapiens-Pytorch-Inference/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247149505,"owners_count":20891954,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","depth-estimation","inference","pose-estimation","python","pytorch","semantic-segmentation"],"created_at":"2024-10-05T02:06:39.947Z","updated_at":"2025-04-04T09:10:01.892Z","avatar_url":"https://github.com/ibaiGorordo.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sapiens-Pytorch-Inference\n Minimal code and examples for inferencing Sapiens foundation human models in Pytorch\n\n![ONNX Sapiens_normal_segmentation](https://github.com/user-attachments/assets/a8f433f0-5f43-4797-89c6-5b33c58cbd01)\n\n# Why\n- Make it easy to run the models by creating a `SapiensPredictor` class that allows to run multiple tasks simultaneously\n- Add several examples to run the models on images, videos, and with a webcam in real-time.\n- Download models automatically from HuggigFace if not available locally.\n- Add a script for ONNX export. However, ONNX inference is not recommended due to the slow speed.\n- Added Object Detection to allow the model to be run for each detected person. However, this mode is disabled as it produces the worst results.\n\n\u003e [!CAUTION]\n\u003e - Use 1B models, since the accuracy of lower models is not good (especially for segmentation)\n\u003e - Exported ONNX models are too slow.\n\u003e - Input sizes other than 768x1024 don't produce good results.\n\u003e - Running Sapiens models on a cropped person produces worse results, even if you crop a wider rectangle around the person.\n\n## Installation [![PyPI](https://img.shields.io/pypi/v/sapiens-inferece?color=2BAF2B)](https://pypi.org/project/sapiens-inferece/)\n```bash\npip install sapiens-inferece\n```\nOr, clone this repository:\n```bash\ngit clone https://github.com/ibaiGorordo/Sapiens-Pytorch-Inference.git\ncd Sapiens-Pytorch-Inference\npip install -r requirements.txt\n```\n\n## Usage\n\n```python\nimport cv2\nfrom imread_from_url import imread_from_url\nfrom sapiens_inference import SapiensPredictor, SapiensConfig, SapiensDepthType, SapiensNormalType\n\n# Load the model\nconfig = SapiensConfig()\nconfig.depth_type = SapiensDepthType.DEPTH_03B  # Disabled by default\nconfig.normal_type = SapiensNormalType.NORMAL_1B  # Disabled by default\npredictor = SapiensPredictor(config)\n\n# Load the image\nimg = imread_from_url(\"https://github.com/ibaiGorordo/Sapiens-Pytorch-Inference/blob/assets/test2.png?raw=true\")\n\n# Estimate the maps\nresult = predictor(img)\n\ncv2.namedWindow(\"Combined\", cv2.WINDOW_NORMAL)\ncv2.imshow(\"Combined\", result)\ncv2.waitKey(0)\n```\n\n### SapiensPredictor\nThe `SapiensPredictor` class allows to run multiple tasks simultaneously. It has the following methods:\n- `SapiensPredictor(config: SapiensConfig)` - Load the model with the specified configuration.\n- `__call__(img: np.ndarray) -\u003e np.ndarray` - Estimate the maps for the input image.\n\n### SapiensConfig\nThe `SapiensConfig` class allows to configure the model. It has the following attributes:\n- `dtype: torch.dtype` - Data type to use. Default: `torch.float32`.\n- `device: torch.device` - Device to use. Default: `cuda` if available, otherwise `cpu`.\n- `depth_type: SapiensDepthType` - Depth model to use. Options: `OFF`, `DEPTH_03B`, `DEPTH_06B`, `DEPTH_1B`, `DEPTH_2B`. Default: `OFF`.\n- `normal_type: SapiensNormalType` - Normal model to use. Options: `OFF`, `NORMAL_03B`, `NORMAL_06B`, `NORMAL_1B`, `NORMAL_2B`. Default: `OFF`.\n- `segmentation_type: SapiensSegmentationType` - Segmentation model to use (Always enabled for the mask). Options: `SEGMENTATION_03B`, `SEGMENTATION_06B`, `SEGMENTATION_1B`. Default: `SEGMENTATION_1B`.\n- `detector_config: DetectorConfig` - Configuration for the object detector. Default: {`model_path: str = \"models/yolov8m.pt\"`, `person_id: int = 0`, `confidence: float = 0.25`}. Disabled as it produces worst results.\n- `minimum_person_height: float` - Minimum height ratio of the person to detect. Default: `0.5f` (50%). Not used if the object detector is disabled.\n\n## Examples\n\n* **Image Sapiens Predictor (Normal, Depth, Segmentation)**:\n```\npython image_predictor.py\n```\n\n![sapiens_human_model](https://github.com/user-attachments/assets/988c7551-061a-4b69-8b7c-4546cba336da)\n\n* **Video Sapiens Predictor (Normal, Depth, Segmentation)**: (https://youtu.be/hOyrnkQz1NE?si=jC76W7AY3zJnZhH4)\n```\npython video_predictor.py\n```\n\n* **Webcam Sapiens Predictor (Normal, Depth, Segmentation)**:\n```\npython webcam_predictor.py\n```\n\n\n* **Image Normal Estimation**:\n```\npython image_normal_estimation.py\n```\n\n* **Image Human Part Segmentation**:\n\n```\npython image_segmentation.py\n```\n\n* **Image Pose Estimation**\n\n```\npython image_pose_estimation.py\n```\n\n* **Video Normal Estimation**:\n\n```\npython video_normal_estimation.py\n```\n\n* **Video Human Part Segmentation**:\n```\npython video_segmentation.py\n```\n\n* **Webcam Normal Estimation**:\n```\npython webcam_normal_estimation.py\n```\n\n* **Webcam Human Part Segmentation**:\n```\npython webcam_segmentation.py\n```\n\n## Export to ONNX\nTo export the model to ONNX, run the following script:\n```bash\npython export_onnx.py seg03b\n```\nThe available models are `seg03b`, `seg06b`, `seg1b`, `depth03b`, `depth06b`, `depth1b`, `depth2b`, `normal03b`, `normal06b`, `normal1b`, `normal2b`.\n\n## Original Models\nThe original models are available at HuggingFace: https://huggingface.co/facebook/sapiens/tree/main/sapiens_lite_host\n- **License**: Creative Commons Attribution-NonCommercial 4.0 International (https://github.com/facebookresearch/sapiens/blob/main/LICENSE)\n\n## References\n- **Sapiens**: https://github.com/facebookresearch/sapiens\n- **Sapiens Lite**: https://github.com/facebookresearch/sapiens/tree/main/lite\n- **HuggingFace Model**: https://huggingface.co/facebook/sapiens\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibaigorordo%2Fsapiens-pytorch-inference","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fibaigorordo%2Fsapiens-pytorch-inference","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibaigorordo%2Fsapiens-pytorch-inference/lists"}