{"id":15011587,"url":"https://github.com/axodox/axodox-machinelearning","last_synced_at":"2025-04-04T15:10:24.240Z","repository":{"id":172530169,"uuid":"649402606","full_name":"axodox/axodox-machinelearning","owner":"axodox","description":"This repository contains a pure C++ ONNX implementation of multiple offline AI models, such as StableDiffusion (1.5 and XL), ControlNet, Midas, HED and OpenPose.","archived":false,"fork":false,"pushed_at":"2024-04-14T18:44:53.000Z","size":11212,"stargazers_count":613,"open_issues_count":10,"forks_count":35,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-03-28T14:08:18.808Z","etag":null,"topics":["controlnet","cpp","directml","holistically-nested-edge-detection","image-generation","midas","mit-license","native","nuget","onnx","openpose","stable-diffusion","stable-diffusion-xl"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/axodox.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-04T18:22:37.000Z","updated_at":"2025-02-24T15:14:37.000Z","dependencies_parsed_at":"2025-03-21T13:21:09.378Z","dependency_job_id":null,"html_url":"https://github.com/axodox/axodox-machinelearning","commit_stats":null,"previous_names":["axodox/axodox-machinelearning"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/axodox%2Faxodox-machinelearning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/axodox%2Faxodox-machinelearning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/axodox%2Faxodox-machinelearning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/axodox%2Faxodox-machinelearning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/axodox","download_url":"https://codeload.github.com/axodox/axodox-machinelearning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247198463,"owners_count":20900080,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["controlnet","cpp","directml","holistically-nested-edge-detection","image-generation","midas","mit-license","native","nuget","onnx","openpose","stable-diffusion","stable-diffusion-xl"],"created_at":"2024-09-24T19:41:18.407Z","updated_at":"2025-04-04T15:10:24.212Z","avatar_url":"https://github.com/axodox.png","language":"C++","readme":"## Introduction\r\n\r\nThis repository contains a **fully C++ implementation of Stable Diffusion**-based image synthesis, including the original txt2img, img2img and inpainting capabilities and the safety checker. This solution **does not depend on Python** and **runs the entire image generation process in a single process with competitive performance**, making deployments significantly simpler and smaller, essentially consisting a few executable and library files, and the model weights. Using the library it is possible to integrate Stable Diffusion into almost any application - as long as it can import C++ or C functions, but it is **most useful for the developers of realtime graphics applications and games**, which are often realized with C++.\r\n\r\n\u003ctable style=\"margin: 0px auto;\"\u003e\r\n  \u003ctr\u003e\r\n    \u003ctd\u003e\u003cimg src=\"https://github.com/axodox/unpaint/blob/main/Unpaint/Showcase/2023-06-03%2020-50-21.png?raw=true\" alt=\"a samurai drawing his sword to defend his land\" width=\"256\" height=\"256\"\u003e\u003c/td\u003e\r\n    \u003ctd\u003e\u003cimg src=\"https://github.com/axodox/unpaint/blob/main/Unpaint/Showcase/2023-06-03%2020-48-40.png?raw=true\" alt=\"a sailship crossing the high sea, 18st century, impressionist painting, closeup\" width=\"256\" height=\"256\"\u003e\u003c/td\u003e\r\n    \u003ctd\u003e\u003cimg src=\"https://github.com/axodox/unpaint/blob/main/Unpaint/Showcase/2023-06-03%2019-32-26.png?raw=true\" alt=\" close up portrait photo of woman in wastelander clothes, long haircut, pale skin, slim body, background is city ruins, (high detailed skin:1.2)\" width=\"256\" height=\"256\"\u003e\u003c/td\u003e\r\n  \u003c/tr\u003e\r\n\u003c/table\u003e\r\n\r\n## ControlNet support\r\n\r\nThe library also supports ControlNet, this allows using input images to guide the image generation process, for example:\r\n\r\n![OpenPose based ControlNet](Examples/controlnet.png)\r\nIn this first, example we use an OpenPose estimator and OpenPose conditioned ControlNet, we can guide the img2img generation by specifying the pose, so it produces better results.\r\n\r\n![HED based ControlNet](Examples/controlnet2.png)\r\nUsing HED edge detection and edge conditioned ControlNet, we change the style of the image to resemble a comic book illustration, but keep the layout intact.\r\n\r\n![Depth based ControlNet](Examples/controlnet3.png)\r\nUsing a depth estimator and depth map conditioned ControlNet, we generate a different character, but keep the original setup.\r\n\r\n## Feature extractors\r\n\r\nThe library also provides GPU accelerated implementations of the following feature extractors (showcased above):\r\n- Pose estimation: extracts the skeleton of a human from an image using [OpenPose](https://arxiv.org/abs/1812.08008)\r\n- Depth estimation: estimates the depth of each pixel from a single image using [MiDAS](https://arxiv.org/abs/1907.01341v3)\r\n- Edge Detection: extracts edges from an image, using [Holistically-Nested Edge Detection](https://arxiv.org/abs/1504.06375)\r\n\r\n## Code examples\r\n\r\nHere are some simple code examples:\r\n- [StableDiffusion](https://github.com/axodox/axodox-machinelearning/blob/main/Axodox.MachineLearning.Test/StableDiffusionTest.cpp)\r\n- [ControlNet](https://github.com/axodox/axodox-machinelearning/blob/main/Axodox.MachineLearning.Test/ControlNetTest.cpp)\r\n- [Feature extraction](https://github.com/axodox/axodox-machinelearning/blob/main/Axodox.MachineLearning.Test/FeatureExtractorTest.cpp)\r\n\r\n## Reference models\r\n\r\nThe AI models required for the library are stored in the ONNX format. All of the models have been run through Microsoft Olive and are optimized for DirectML. I have tested the library with the following models:\r\n\r\n- [Stable Diffusion 1.5 with ControlNet support](https://huggingface.co/axodoxian/stable_diffusion_onnx)\r\n- [Realistic Vision 1.4 with ControlNet support](https://huggingface.co/axodoxian/realistic_vision_onnx)\r\n- [ControlNet with feature extractors](https://huggingface.co/axodoxian/controlnet_onnx)\r\n\r\nYou may bring your own models, by converting them using [this guide](https://github.com/axodox/unpaint/wiki/Model-import).\r\n\r\n\u003e Please make sure to check the original license of the models if you plan to integrate them in your products.\r\n\r\n## Technical background\r\n\r\nThe implementation uses the [ONNX](https://onnx.ai/) to store the mathematical models involved in the image generation. These ONNX models are then executed using the [ONNX runtime](https://github.com/microsoft/onnxruntime), which support a variety of platforms (Windows, Linux, MacOS, Android, iOS, WebAssembly etc.), and execution providers (such as NVIDIA CUDA / TensorRT; AMD ROCm, Apple CoreML, Qualcomm QNN, Microsoft DirectML and many more). \r\n\r\nWe provide an example integration called [Unpaint](https://github.com/axodox/unpaint) which showcases how the libraries can be integrated in a simple WinUI based user interface. You may download the free app from the [Microsoft Store](https://www.microsoft.com/store/productId/9N25M770896D) to evaluate the performance characteristics of the solution.\r\n\r\nThe current codebase and the resulting [Nuget packages](https://www.nuget.org/packages/Axodox.MachineLearning) target Windows and use DirectML, however only small sections of the code utilize Windows specific APIs, and thus could be ported to other platforms with minimal effort.\r\n\r\n## Licensing\r\n\r\nThe source code of this library is provided under the MIT license.\r\n\r\n## Integrating the component\r\n\r\nPrebuilt versions of the project can be retrieved from Nuget under the name `Axodox.MachineLearning` and added to Visual Studio C++ projects (both desktop and UWP projects are supported) with the x64 platform.\r\n\r\nBasic integration:\r\n- Add the `Axodox.Common` and `Axodox.MachineLearning` packages to your project\r\n- Make sure to **only have x64 platform in your project** as this lib is x64 only for now\r\n- Ensure that your compiler is set to **C++20**, we also recommend enabling all warnings and conformance mode\r\n- Add the following include statement to your code file or precompiled header: `#include \"Include/Axodox.MachineLearning.h\"`\r\n- Follow this example code to integrate the pipeline: https://github.com/axodox/unpaint/blob/main/Unpaint/StableDiffusionModelExecutor.cpp\r\n\r\n\u003e We recommend adding appropriate safety mechanisms to your app to suppress inappropriate outputs of StableDiffusion, the performance overhead is insignificant.\r\n\r\nThe Stable Diffusion models we use have been generated using [Microsoft Olive](https://github.com/microsoft/Olive/tree/main/examples/directml/stable_diffusion), please follow the linked example to convert models from HuggingFace. By changing the script you can also convert models stored on your disk from various formats (e.g. `*.safetensors`). You can find some preconverted models [here](https://huggingface.co/models?other=unpaint,stable_diffusion_model) for testing.\r\n\r\n# Building the project\r\n\r\nBuilding the library is required to make and test changes. You will need to have the following installed to build the library:\r\n\r\n- [Visual Studio 2022](https://visualstudio.microsoft.com/downloads/)\r\n  - Select the following workloads:\r\n    - Desktop development with C++    \r\n    - Game development with C++\r\n  - To build [Unpaint](https://github.com/axodox/unpaint) as well also select these individual packages:\r\n    - Universal Windows Platform development\r\n    - C++ (v143) Universal Windows Platform tools\r\n\r\nYou can either run `build_nuget.ps1` or open `Axodox.MachineLearning.sln` and build from Visual Studio.\r\n\r\nOnce you have built the library, you override your existing nuget package install by setting the `AxodoxMachineLearning-Location` environment variable to point to your local build. \r\n\r\n\u003e For example `C:\\dev\\axodox-machinelearning\\Axodox.MachineLearning.Universal` for an UWP app and `C:\\dev\\axodox-machinelearning\\Axodox.MachineLearning.Desktop` for a desktop app.\r\n\r\nThen add the project pointed by the path to the solution of your own project. This allows to add all projects into the same solution and make changes on the library and your app seamlessly without copying files repeatedly.\r\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faxodox%2Faxodox-machinelearning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faxodox%2Faxodox-machinelearning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faxodox%2Faxodox-machinelearning/lists"}