Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/depasqualeorg/mlx-intro

Introduction to MLX for Swift developers
https://github.com/depasqualeorg/mlx-intro

ai apple deep-learning image-generation llm machine-learning mlx python speech-recognition speech-to-text swift text-generation text-to-speech

Last synced: 14 days ago
JSON representation

Introduction to MLX for Swift developers

Host: GitHub
URL: https://github.com/depasqualeorg/mlx-intro
Owner: DePasqualeOrg
Created: 2024-10-15T18:59:15.000Z (about 1 month ago)
Default Branch: main
Last Pushed: 2024-10-27T10:51:28.000Z (20 days ago)
Last Synced: 2024-10-27T11:50:13.750Z (20 days ago)
Topics: ai, apple, deep-learning, image-generation, llm, machine-learning, mlx, python, speech-recognition, speech-to-text, swift, text-generation, text-to-speech
Language: Jupyter Notebook
Homepage:
Size: 3.91 KB
Stars: 1
Watchers: 1
Forks: 0
Open Issues: 1
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

        ## Introduction to MLX for Swift developers

This introduction was created for my talk at the NSBarcelona meetup at Glovo on 5 November 2024.

**Advantages of MLX**

- [Optimized](https://github.com/ml-explore/mlx/issues/12#issuecomment-1843956313) for unified memory on Apple silicon

- Closely follows NumPy and PyTorch APIs

- Has a Swift API, allowing apps using MLX to be compiled for macOS, iOS, and visionOS

- Easier to work with most open-source models compared to Core ML

  - Core ML uses the Neural Engine in addition to the GPU and CPU. MLX doesn't use the Neural Engine due to its [constraints](https://github.com/ml-explore/mlx/issues/18#issuecomment-1846188238).

- Open source and evolving quickly with the field

**General introductions to deep learning**

- The [fast.ai course](https://course.fast.ai) is a great introduction to deep learning that starts from a high level and progressively delves into lower-level concepts.

- I also highly recommend François Chollet's *Deep Learning with Python* ([second edition](https://www.manning.com/books/deep-learning-with-python-second-edition) currently available, [third edition](https://www.manning.com/books/deep-learning-with-python-third-edition) forthcoming in spring 2025).

**Converting models to MLX**

New models generally get converted to MLX and quantized within hours of their release. You can find them in the [MLX Community](https://huggingface.co/mlx-community) on Hugging Face. If you need to convert and/or quantize a model, I've included an example notebook in this repository.

**Porting MLX code from Python to Swift**

- Claude 3.5 Sonnet has knowledge about MLX and generally does a good job of porting Python MLX code to Swift when provided with existing examples in both languages. Minor corrections are usually necessary.

**Designing for compatibility and stability in Swift**

- MLX is compatible with devices that support Metal version 7 or later (A14/M1 chip or later).

  - `MTLCreateSystemDefaultDevice()?.supportsFamily(.apple7)`

- Ensure that the model you're using will fit in the device's available memory.

  - `MTLDevice.recommendedMaxWorkingSetSize`

- Ensure that the operations you're running will fit in the device's maximum buffer length.

  - `MTLDevice.maxBufferLength`

- Carefully manage concurrency.

- Stop running operations on the GPU when an app enters the background (on iOS) and before it is quit by the user.

**Tools and examples**

- [mlx-examples](https://github.com/ml-explore/mlx-examples): Tools and examples in Python

- [mlx-swift-examples](https://github.com/ml-explore/mlx-swift-examples): Tools and examples in Swift

- [swift-transformers](https://github.com/huggingface/swift-transformers): Swift port of Hugging Face's [transformers](https://github.com/huggingface/transformers) library, including tools for downloading models and using tokenizers

  - [transformers.js](https://github.com/xenova/transformers.js) and [huggingface.js](https://github.com/huggingface/huggingface.js) include JavaScript implementations that can be useful when adding functionality to swift-transformers.

**Projects using MLX**

- Text generation

  - [LM Studio](https://lmstudio.ai): Cross-platform (desktop) chat app with Python MLX backend

  - [mlx-vlm](https://github.com/Blaizzy/mlx-vlm): Vision language models in Python

  - [fullmoon](https://github.com/mainframecomputer/fullmoon-ios): iOS chat app in Swift

  - [ChatMLX](https://github.com/maiqingqiang/ChatMLX): macOS chat app in Swift

  - [LLMEval](https://github.com/ml-explore/mlx-swift-examples/blob/main/Applications/LLMEval/README.md): Simple example chat app in Swift for macOS, iOS, and visionOS

  - [Local Chat](https://localchat.co): My chat app for macOS and iOS

- Text to speech

  - https://github.com/JosefAlbers/e2tts-mlx

  - https://github.com/lucasnewman/e2-tts-mlx

  - https://github.com/lucasnewman/f5-tts-mlx

- Speech to text

  - https://github.com/mustafaaljadery/lightning-whisper-mlx

  - https://github.com/JosefAlbers/whisper-turbo-mlx

- Image generation

  - https://github.com/filipstrand/mflux

  - https://github.com/mzbac/flux.swift

**People to follow on 𝕏**

- [Awni Hannun](https://x.com/awnihannun) (Apple)

- [Angelos Katharopoulos](https://x.com/angeloskath) (Apple)

- [Pedro Cuenca](https://x.com/pcuenq) (Hugging Face)

- [Prince Canuma](https://x.com/Prince_Canuma) (Arcee AI)

- [Me](https://x.com/DePasqualeOrg) (independent)

**Call to action for Swift developers**

Contribute to [mlx-swift-examples](https://github.com/ml-explore/mlx-swift-examples) and [swift-transformers](https://github.com/huggingface/swift-transformers) to help bring these tools and examples up to par with their Python equivalents.