{"id":15110157,"url":"https://github.com/nod-ai/AMD-SHARK-Studio","last_synced_at":"2026-01-09T22:56:02.930Z","repository":{"id":37185260,"uuid":"466309974","full_name":"nod-ai/SHARK-Studio","owner":"nod-ai","description":"SHARK Studio -- Web UI for SHARK+IREE High Performance Machine Learning Distribution","archived":false,"fork":false,"pushed_at":"2024-10-18T15:16:28.000Z","size":45794,"stargazers_count":1418,"open_issues_count":420,"forks_count":171,"subscribers_count":41,"default_branch":"main","last_synced_at":"2024-10-29T15:41:25.994Z","etag":null,"topics":["amd","apple-silicon","deep-learning","machine-learning","mlir","nvidia","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nod-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-05T00:37:21.000Z","updated_at":"2024-10-27T15:00:07.000Z","dependencies_parsed_at":"2022-07-11T03:47:31.374Z","dependency_job_id":"7350a58b-61fd-4ef8-bc06-f2e63de3ab4c","html_url":"https://github.com/nod-ai/SHARK-Studio","commit_stats":null,"previous_names":["nod-ai/shark-studio","nod-ai/shark"],"tags_count":814,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nod-ai%2FSHARK-Studio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nod-ai%2FSHARK-Studio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nod-ai%2FSHARK-Studio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nod-ai%2FSHARK-Studio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nod-ai","download_url":"https://codeload.github.com/nod-ai/SHARK-Studio/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248567998,"owners_count":21125942,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amd","apple-silicon","deep-learning","machine-learning","mlir","nvidia","pytorch"],"created_at":"2024-09-25T23:42:07.488Z","updated_at":"2026-01-09T22:56:02.924Z","avatar_url":"https://github.com/nod-ai.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# AMDSHARK\n\nHigh Performance Machine Learning Distribution\n\n\u003ch2\u003eNOTE: This project is not currently maintained.\u003c/h2\u003e\n\n*The latest versions of this project are developments towards a refactor on top of IREE-Turbine. Until further notice, make sure you use an .exe release or a checkout of the `AMDSHARK-1.0` branch, for a working AMDSHARK-Studio*\n\n[![Nightly Release](https://github.com/nod-ai/AMD-SHARK-Studio/actions/workflows/nightly.yml/badge.svg)](https://github.com/nod-ai/AMD-SHARK-Studio/actions/workflows/nightly.yml)\n\n\u003cdetails\u003e\n  \u003csummary\u003ePrerequisites - Drivers \u003c/summary\u003e\n\n#### Install your Windows hardware drivers\n* [AMD RDNA Users] Download the latest driver (23.2.1 is the oldest supported) [here](https://www.amd.com/en/support).\n* [macOS Users] Download and install the 1.3.216 Vulkan SDK from [here](https://sdk.lunarg.com/sdk/download/1.3.216.0/mac/vulkansdk-macos-1.3.216.0.dmg). Newer versions of the SDK will not work.\n* [Nvidia Users] Download and install the latest CUDA / Vulkan drivers from [here](https://developer.nvidia.com/cuda-downloads)\n\n#### Linux Drivers\n* MESA / RADV drivers wont work with FP16. Please use the latest AMGPU-PRO drivers (non-pro OSS drivers also wont work) or the latest NVidia Linux Drivers.\n\nOther users please ensure you have your latest vendor drivers and Vulkan SDK from [here](https://vulkan.lunarg.com/sdk/home) and if you are using vulkan check `vulkaninfo` works in a terminal window\n\n\u003c/details\u003e\n\n\n\n### Quick Start for AMDSHARK Stable Diffusion for Windows 10/11 Users\n\nInstall the Driver from [Prerequisites](https://github.com/nod-ai/AMD-SHARK-Studio#install-your-hardware-drivers) above\n\nDownload the [stable release](https://github.com/nod-ai/AMD-SHARK-Studio/releases/latest) or the most recent [AMDSHARK 1.0 pre-release](https://github.com/nod-ai/AMD-SHARK-Studio/releases).\n\nDouble click the .exe, or [run from the command line](#running) (recommended), and you should have the [UI](http://localhost:8080/) in the browser.\n\nIf you have custom models put them in a `models/` directory where the .exe is.\n\nEnjoy.\n\n\u003cdetails\u003e\n  \u003csummary\u003eMore installation notes\u003c/summary\u003e\n* We recommend that you download EXE in a new folder, whenever you download a new EXE version. If you download it in the same folder as a previous install, you must delete the old `*.vmfb` files with `rm *.vmfb`. You can also use `--clear_all` flag once to clean all the old files.\n* If you recently updated the driver or this binary (EXE file), we recommend you clear all the local artifacts with `--clear_all`\n\n## Running\n\n* Open a Command Prompt or Powershell terminal, change folder (`cd`) to the .exe folder. Then run the EXE from the command prompt. That way, if an error occurs, you'll be able to cut-and-paste it to ask for help. (if it always works for you without error, you may simply double-click the EXE)\n* The first run may take few minutes when the models are downloaded and compiled. Your patience is appreciated. The download could be about 5GB.\n* You will likely see a Windows Defender message asking you to give permission to open a web server port. Accept it.\n* Open a browser to access the Stable Diffusion web server. By default, the port is 8080, so you can go to http://localhost:8080/.\n* If you prefer to always run in the browser, use the `--ui=web` command argument when running the EXE.\n\n## Stopping\n\n* Select the command prompt that's running the EXE. Press CTRL-C and wait a moment or close the terminal.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eAdvanced Installation (Only for developers)\u003c/summary\u003e\n\n## Advanced Installation (Windows, Linux and macOS) for developers\n\n### Windows 10/11 Users\n\n* Install Git for Windows from [here](https://git-scm.com/download/win) if you don't already have it.\n\n## Check out the code\n\n```shell\ngit clone https://github.com/nod-ai/AMD-SHARK-Studio.git\ncd AMD-SHARK-Studio\n```\n\n## Switch to the Correct Branch (IMPORTANT!)\n\nCurrently AMDSHARK is being rebuilt for [Turbine](https://github.com/iree-org/iree-turbine) on the `main` branch. For now you are strongly discouraged from using `main` unless you are working on the rebuild effort, and should not expect the code there to produce a working application for Image Generation, So for now you'll need switch over to the `AMDSHARK-1.0` branch and use the stable code.\n\n```shell\ngit checkout AMDSHARK-1.0\n```\n\nThe following setup instructions assume you are on this branch.\n\n## Setup your Python VirtualEnvironment and Dependencies\n\n### Windows 10/11 Users\n\n* Install the latest Python 3.11.x version from [here](https://www.python.org/downloads/windows/)\n\n#### Allow the install script to run in Powershell\n```powershell\nset-executionpolicy remotesigned\n```\n\n#### Setup venv and install necessary packages (torch-mlir, nodLabs/AMDShark, ...)\n```powershell\n./setup_venv.ps1 #You can re-run this script to get the latest version\n```\n\n### Linux / macOS Users\n\n```shell\n./setup_venv.sh\nsource amdshark1.venv/bin/activate\n```\n\n### Run Stable Diffusion on your device - WebUI\n\n#### Windows 10/11 Users\n```powershell\n(amdshark1.venv) PS C:\\g\\amdshark\u003e cd .\\apps\\stable_diffusion\\web\\\n(amdshark1.venv) PS C:\\g\\amdshark\\apps\\stable_diffusion\\web\u003e python .\\index.py\n```\n#### Linux / macOS Users\n```shell\n(amdshark1.venv) \u003e cd apps/stable_diffusion/web\n(amdshark1.venv) \u003e python index.py\n```\n\n#### Access Stable Diffusion on http://localhost:8080/?__theme=dark\n\n\n\u003cimg width=\"1607\" alt=\"webui\" src=\"https://user-images.githubusercontent.com/74956/204939260-b8308bc2-8dc4-47f6-9ac0-f60b66edab99.png\"\u003e\n\n\n\n### Run Stable Diffusion on your device - Commandline\n\n#### Windows 10/11 Users\n```powershell\n(amdshark1.venv) PS C:\\g\\amdshark\u003e python .\\apps\\stable_diffusion\\scripts\\main.py --app=\"txt2img\" --precision=\"fp16\" --prompt=\"tajmahal, snow, sunflowers, oil on canvas\" --device=\"vulkan\"\n```\n\n#### Linux / macOS Users\n```shell\npython3.11 apps/stable_diffusion/scripts/main.py --app=txt2img --precision=fp16 --device=vulkan --prompt=\"tajmahal, oil on canvas, sunflowers, 4k, uhd\"\n```\n\nYou can replace `vulkan` with `cpu` to run on your CPU or with `cuda` to run on CUDA devices. If you have multiple vulkan devices you can address them with `--device=vulkan://1` etc\n\u003c/details\u003e\n\nThe output on a AMD 7900XTX would look something like:\n\n```shell\nAverage step time: 47.19188690185547ms/it\nClip Inference time (ms) = 109.531\nVAE Inference time (ms): 78.590\n\nTotal image generation time: 2.5788655281066895sec\n```\n\nHere are some samples generated:\n\n![tajmahal, snow, sunflowers, oil on canvas_0](https://user-images.githubusercontent.com/74956/204934186-141f7e43-6eb2-4e89-a99c-4704d20444b3.jpg)\n\n![a photo of a crab playing a trumpet](https://user-images.githubusercontent.com/74956/204933258-252e7240-8548-45f7-8253-97647d38313d.jpg)\n\n\nFind us on [AMDSHARK Discord server](https://discord.gg/RUqY2h2s9u) if you have any trouble with running it on your hardware.\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eBinary Installation\u003c/summary\u003e\n\n### Setup a new pip Virtual Environment\n\nThis step sets up a new VirtualEnv for Python\n\n```shell\npython --version #Check you have 3.11 on Linux, macOS or Windows Powershell\npython -m venv amdshark_venv\nsource amdshark_venv/bin/activate   # Use amdshark_venv/Scripts/activate on Windows\n\n# If you are using conda create and activate a new conda env\n\n# Some older pip installs may not be able to handle the recent PyTorch deps\npython -m pip install --upgrade pip\n```\n\n*macOS Metal* users please install https://sdk.lunarg.com/sdk/download/latest/mac/vulkan-sdk.dmg and enable \"System wide install\"\n\n### Install AMD-SHARK\n\nThis step pip installs AMD-SHARK and related packages on Linux Python 3.8, 3.10 and 3.11 and macOS / Windows Python 3.11\n\n```shell\npip install nodai-amdshark -f https://nod-ai.github.io/AMD-SHARK-Studio/package-index/ -f https://llvm.github.io/torch-mlir/package-index/ -f  https://nod-ai.github.io/SRT/pip-release-links.html --extra-index-url https://download.pytorch.org/whl/nightly/cpu\n```\n\n### Run amdshark tank model tests.\n```shell\npytest tank/test_models.py\n```\nSee tank/README.md for a more detailed walkthrough of our pytest suite and CLI.\n\n### Download and run Resnet50 sample\n\n```shell\ncurl -O https://raw.githubusercontent.com/nod-ai/AMD-SHARK-Studio/main/amdshark/examples/amdshark_inference/resnet50_script.py\n#Install deps for test script\npip install --pre torch torchvision torchaudio tqdm pillow gsutil --extra-index-url https://download.pytorch.org/whl/nightly/cpu\npython ./resnet50_script.py --device=\"cpu\"  #use cuda or vulkan or metal\n```\n\n### Download and run BERT (MiniLM) sample\n```shell\ncurl -O https://raw.githubusercontent.com/nod-ai/AMD-SHARK-Studio/main/amdshark/examples/amdshark_inference/minilm_jit.py\n#Install deps for test script\npip install transformers torch --extra-index-url https://download.pytorch.org/whl/nightly/cpu\npython ./minilm_jit.py --device=\"cpu\"  #use cuda or vulkan or metal\n```\n\u003c/details\u003e\n\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eDevelopment, Testing and Benchmarks\u003c/summary\u003e\n\nIf you want to use Python3.11 and with TF Import tools you can use the environment variables like:\nSet `USE_IREE=1` to use upstream IREE\n```\n# PYTHON=python3.11 VENV_DIR=0617_venv IMPORTER=1 ./setup_venv.sh\n```\n\n### Run any of the hundreds of AMDSHARK tank models via the test framework\n```shell\npython -m  amdshark.examples.amdshark_inference.resnet50_script --device=\"cpu\" # Use gpu | vulkan\n# Or a pytest\npytest tank/test_models.py -k \"MiniLM\"\n```\n\n### How to use your locally built IREE / Torch-MLIR with AMDSHARK\nIf you are a *Torch-mlir developer or an IREE developer* and want to test local changes you can uninstall\nthe provided packages with `pip uninstall torch-mlir` and / or `pip uninstall iree-compiler iree-runtime` and build locally\nwith Python bindings and set your PYTHONPATH as mentioned [here](https://github.com/iree-org/iree/tree/main/docs/api_docs/python#install-iree-binaries)\nfor IREE and [here](https://github.com/llvm/torch-mlir/blob/main/development.md#setup-python-environment-to-export-the-built-python-packages)\nfor Torch-MLIR.\n\nHow to use your locally built Torch-MLIR with AMDSHARK:\n```shell\n1.) Run `./setup_venv.sh in AMDSHARK` and activate `amdshark.venv` virtual env.\n2.) Run `pip uninstall torch-mlir`.\n3.) Go to your local Torch-MLIR directory.\n4.) Activate mlir_venv virtual envirnoment.\n5.) Run `pip uninstall -r requirements.txt`.\n6.) Run `pip install -r requirements.txt`.\n7.) Build Torch-MLIR.\n8.) Activate amdshark.venv virtual environment from the Torch-MLIR directory.\n8.) Run `export PYTHONPATH=`pwd`/build/tools/torch-mlir/python_packages/torch_mlir:`pwd`/examples` in the Torch-MLIR directory.\n9.) Go to the AMDSHARK directory.\n```\nNow the AMDSHARK will use your locally build Torch-MLIR repo.\n\n\n## Benchmarking Dispatches\n\nTo produce benchmarks of individual dispatches, you can add `--dispatch_benchmarks=All --dispatch_benchmarks_dir=\u003coutput_dir\u003e` to your pytest command line argument.\nIf you only want to compile specific dispatches, you can specify them with a space seperated string instead of `\"All\"`.  E.G. `--dispatch_benchmarks=\"0 1 2 10\"`\n\nFor example, to generate and run dispatch benchmarks for MiniLM on CUDA:\n```\npytest -k \"MiniLM and torch and static and cuda\" --benchmark_dispatches=All -s --dispatch_benchmarks_dir=./my_dispatch_benchmarks\n```\nThe given command will populate `\u003cdispatch_benchmarks_dir\u003e/\u003cmodel_name\u003e/` with an `ordered_dispatches.txt` that lists and orders the dispatches and their latencies, as well as folders for each dispatch that contain .mlir, .vmfb, and results of the benchmark for that dispatch.\n\nif you want to instead incorporate this into a python script, you can pass the `dispatch_benchmarks` and `dispatch_benchmarks_dir` commands when initializing `AMDSharkInference`, and the benchmarks will be generated when compiled.  E.G:\n\n```\namdshark_module = AMDSharkInference(\n        mlir_model,\n        device=args.device,\n        mlir_dialect=\"tm_tensor\",\n        dispatch_benchmarks=\"all\",\n        dispatch_benchmarks_dir=\"results\"\n    )\n```\n\nOutput will include:\n- An ordered list ordered-dispatches.txt of all the dispatches with their runtime\n- Inside the specified directory, there will be a directory for each dispatch (there will be mlir files for all dispatches, but only compiled binaries and benchmark data for the specified dispatches)\n- An .mlir file containing the dispatch benchmark\n- A compiled .vmfb file containing the dispatch benchmark\n- An .mlir file containing just the hal executable\n- A compiled .vmfb file of the hal executable\n- A .txt file containing benchmark output\n\n\nSee tank/README.md for further instructions on how to run model tests and benchmarks from the AMDSHARK tank.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eAPI Reference\u003c/summary\u003e\n\n### AMDShark Inference API\n\n```\n\nfrom amdshark.amdshark_importer import AMDSharkImporter\n\n# AMDSharkImporter imports mlir file from the torch, tensorflow or tf-lite module.\n\nmlir_importer = AMDSharkImporter(\n    torch_module,\n    (input),\n    frontend=\"torch\",  #tf, #tf-lite\n)\ntorch_mlir, func_name = mlir_importer.import_mlir(tracing_required=True)\n\n# AMDSharkInference accepts mlir in linalg, mhlo, and tosa dialect.\n\nfrom amdshark.amdshark_inference import AMDSharkInference\namdshark_module = AMDSharkInference(torch_mlir, device=\"cpu\", mlir_dialect=\"linalg\")\namdshark_module.compile()\nresult = amdshark_module.forward((input))\n\n```\n\n\n### Example demonstrating running MHLO IR.\n\n```\nfrom amdshark.amdshark_inference import AMDSharkInference\nimport numpy as np\n\nmhlo_ir = r\"\"\"builtin.module  {\n      func.func @forward(%arg0: tensor\u003c1x4xf32\u003e, %arg1: tensor\u003c4x1xf32\u003e) -\u003e tensor\u003c4x4xf32\u003e {\n        %0 = chlo.broadcast_add %arg0, %arg1 : (tensor\u003c1x4xf32\u003e, tensor\u003c4x1xf32\u003e) -\u003e tensor\u003c4x4xf32\u003e\n        %1 = \"mhlo.abs\"(%0) : (tensor\u003c4x4xf32\u003e) -\u003e tensor\u003c4x4xf32\u003e\n        return %1 : tensor\u003c4x4xf32\u003e\n      }\n}\"\"\"\n\narg0 = np.ones((1, 4)).astype(np.float32)\narg1 = np.ones((4, 1)).astype(np.float32)\namdshark_module = AMDSharkInference(mhlo_ir, device=\"cpu\", mlir_dialect=\"mhlo\")\namdshark_module.compile()\nresult = amdshark_module.forward((arg0, arg1))\n```\n\u003c/details\u003e\n\n## Examples Using the REST API\n\n* [Setting up AMDSHARK for use with Blender](./docs/amdshark_sd_blender.md)\n* [Setting up AMDSHARK for use with Koboldcpp](./docs/amdshark_sd_koboldcpp.md)\n\n## Supported and Validated Models\n\nAMDSHARK is maintained to support the latest innovations in ML Models:\n\n| TF HuggingFace Models | AMDSHARK-CPU | AMDSHARK-CUDA | AMDSHARK-METAL |\n|---------------------|----------|----------|-------------|\n| BERT                | :green_heart:         | :green_heart:         | :green_heart:            |\n| DistilBERT         | :green_heart:         | :green_heart:         | :green_heart:            |\n| GPT2         | :green_heart:         | :green_heart:         | :green_heart:            |\n| BLOOM         | :green_heart:         | :green_heart:         | :green_heart:            |\n| Stable Diffusion         | :green_heart:         | :green_heart:         | :green_heart:            |\n| Vision Transformer       | :green_heart:         | :green_heart:         | :green_heart:            |\n| ResNet50         | :green_heart:         | :green_heart:         | :green_heart:            |\n\nFor a complete list of the models supported in AMDSHARK, please refer to [tank/README.md](https://github.com/nod-ai/AMD-SHARK-Studio/blob/main/tank/README.md).\n\n## Communication Channels\n\n*   [AMDSHARK Discord server](https://discord.gg/RUqY2h2s9u): Real time discussions with the AMDSHARK team and other users\n*   [GitHub issues](https://github.com/nod-ai/AMD-SHARK-Studio/issues): Feature requests, bugs etc\n\n## Related Projects\n\n\u003cdetails\u003e\n  \u003csummary\u003eIREE Project Channels\u003c/summary\u003e\n\n*   [Upstream IREE issues](https://github.com/google/iree/issues): Feature requests,\n    bugs, and other work tracking\n*   [Upstream IREE Discord server](https://discord.gg/wEWh6Z9nMU): Daily development\n    discussions with the core team and collaborators\n*   [iree-discuss email list](https://groups.google.com/forum/#!forum/iree-discuss):\n    Announcements, general and low-priority discussion\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eMLIR and Torch-MLIR Project Channels\u003c/summary\u003e\n\n* `#torch-mlir` channel on the LLVM [Discord](https://discord.gg/xS7Z362) - this is the most active communication channel\n* Torch-MLIR Github issues [here](https://github.com/llvm/torch-mlir/issues)\n* [`torch-mlir` section](https://llvm.discourse.group/c/projects-that-want-to-become-official-llvm-projects/torch-mlir/41) of LLVM Discourse\n*  Weekly meetings on Mondays 9AM PST. See [here](https://discourse.llvm.org/t/community-meeting-developer-hour-refactoring-recurring-meetings/62575) for more information.\n* [MLIR topic within LLVM Discourse](https://llvm.discourse.group/c/llvm-project/mlir/31) AMDSHARK and IREE is enabled by and heavily relies on [MLIR](https://mlir.llvm.org).\n\u003c/details\u003e\n\n## License\n\nnod.ai AMDSHARK is licensed under the terms of the Apache 2.0 License with LLVM Exceptions.\nSee [LICENSE](LICENSE) for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnod-ai%2FAMD-SHARK-Studio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnod-ai%2FAMD-SHARK-Studio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnod-ai%2FAMD-SHARK-Studio/lists"}