https://github.com/do-me/mlx_parallm_ui

A minimal UI for parallel inferencing based on mlx_parallm
https://github.com/do-me/mlx_parallm_ui

Last synced: 2 months ago
JSON representation

A minimal UI for parallel inferencing based on mlx_parallm

• Host: GitHub
• URL: https://github.com/do-me/mlx_parallm_ui
• Owner: do-me
• License: mit
• Created: 2024-07-07T10:53:03.000Z (10 months ago)
• Default Branch: main
• Last Pushed: 2024-07-07T11:36:56.000Z (10 months ago)
• Last Synced: 2025-01-16T03:43:55.190Z (4 months ago)
• Language: HTML
• Size: 12.7 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# MLX ParaLLM UI

A minimal UI for parallel inferencing based on https://github.com/willccbb/mlx_parallm. Includes a fastapi backend and a tailwind-based frontend.

![image](https://github.com/do-me/mlx_parallm_ui/assets/47481567/6d43145c-5922-4332-944c-91de7cd684ff)

## Installation 

1. Follow the setup instructions in https://github.com/willccbb/mlx_parallm, best install dependencies in a virtual env

2. Copy the files from this repo into `mlx_parallm` (main dir) 

3. `pip install fastapi==0.95.1 pydantic==1.10.4 starlette==0.27.0 uvicorn==0.22.0` to install missing dependencies

4. Run the fastapi server with `python fastapi_server.py` and open localhost:8000 in your browser

Inferencing for the first time downloads the model once to your computer.

## Idea 

Keep it super simple to set up. Using tailwind css and marked js dependencies from a cdn. 

As soon as mlx_parallm becomes a pip package, installation will become easier.

## Notes 

Note that MLX is specific for Apple Silicon, so you can only run it on the M-series of Macs. Tested on a Mac M3 Max 128Gb.