https://github.com/preternatural-explore/mlx-swift-chat

A multi-platform SwiftUI frontend for running local LLMs with Apple's MLX framework.
https://github.com/preternatural-explore/mlx-swift-chat

ios llm-inference macos mlx mlx-swift swiftui

Last synced: 7 days ago
JSON representation

A multi-platform SwiftUI frontend for running local LLMs with Apple's MLX framework.

• Host: GitHub
• URL: https://github.com/preternatural-explore/mlx-swift-chat
• Owner: preternatural-explore
• License: mit
• Created: 2024-02-26T23:03:47.000Z (9 months ago)
• Default Branch: main
• Last Pushed: 2024-10-27T10:12:57.000Z (18 days ago)
• Last Synced: 2024-10-27T11:36:11.438Z (17 days ago)
• Topics: ios, llm-inference, macos, mlx, mlx-swift, swiftui
• Language: Swift
• Homepage:
• Size: 39.1 KB
• Stars: 352
• Watchers: 12
• Forks: 22
• Open Issues: 5
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

        
# MLX Swift Chat: Run LLM models locally with MLX!

https://github.com/PreternaturalAI/mlx-swift-chat/assets/8635253/f20862f3-8cab-4803-ba6e-44108b075c9b

### Run LLM models locally with MLX!

> MLX is an efficient machine learning framework specifically designed for Apple silicon (i.e. your laptop!)

>

> – [@awnihannun](https://twitter.com/awnihannun/status/1732184443451019431)

This project is a fully native SwiftUI app that allows you to run local LLMs (e.g. Llama, Mistral) on Apple silicon in real-time using [MLX](https://github.com/ml-explore/mlx).

## Installation

1. Open the Xcode project.

2. Go to **Signing & Capabilities**.

3. Change the **Team** to your own team.

4. Set the destination to **My Mac**.

5. Click **Run**.

Support for iOS is coming next week.

## Usage

1. Click on **Manage Models** in the inspector view.

2. Download and install a model (we recommend starting with `Nous-Hermes-2-Mistral-7B-DPO-4bit-MLX`).

3. Go back to the inspector and select the downloaded model from the model picker.

4. Wait for the model to load, the status bar will flash "Ready" once it is loaded.

5. Click the run button.



## Roadmap

- [ ] Fix iOS builds

- [ ] Implement support for StableLM

- [ ] Implement basic support automatically adding model-specific chat templates to the prompt

- [ ] Add support for stop sequences

- [ ] Add more model suggestions

- [ ] ... (many, _many_ more items to be added soon pending sleep)

## Frequently Asked Questions

### What models are currently supported?

| Model   | Status                      |

| ------- | --------------------------- |

| Mistral | Supported                   |

| Llama   | Supported                   |

| Phi     | Supported                   |

| Gemma   | Supported (May have issues) |

### How do I add new models?

Models are downloaded from Hugging Face. To add a new model, visit the [MLX Community on HuggingFace](https://huggingface.co/mlx-community) and search for the model you want, then add it via **Manage Models** → **Add Model**

> [!IMPORTANT]

> Note that this project is still under active development and some models may require additional implementation to run correctly.

### Is this suitable for production?

No. This is not intended for deploying into production.

### What are the minimum hardware and software requirements?

- Apple Silicon Mac (M1/M2/M3) with macOS 14.0 or newer.

- Any A-Series chip (iPad, iPhone) with iOS 17.2 or newer.

### Does this collect any data?

No. Everything is run locally on device.

### What are the parameters?

- **Temperature**: Controls randomness. Lowering results in less random completions. As the temperature approaches zero, the model will become deterministic and repetitive.

- **Top K**: Sort predicted tokens by probability and discards those below the k-th one. A top-k value of 1 is equivalent to greedy search (select the most probable token).

- **Maximum length**: The maximum number of tokens to generate. Requests can use up to 2,048 tokens shared between prompt and completion. The exact limit varies by model. (One token is roughly 4 characters for normal English text)

## Acknowledgements

- [buh/CompactSlider](https://github.com/buh/CompactSlider)

- [ml-explore/mlx-swift](https://github.com/ml-explore/mlx-swift)

- [huggingface/swift-chat](https://github.com/huggingface/swift-chat)

Special thanks to [Awni Hannun](https://github.com/awni) and [David Koski](https://github.com/davidkoski) for early testing and feedback

Much ❤️ to all the folks who made MLX (especially mlx-swift) possible!