Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ziozzang/mac_mlx_phi-2_server
Test server code for Phi-2 model. support OpenAI API spec
https://github.com/ziozzang/mac_mlx_phi-2_server
apple inference llm macos mlx openai-api osx phi-2 python server
Last synced: 1 day ago
JSON representation
Test server code for Phi-2 model. support OpenAI API spec
- Host: GitHub
- URL: https://github.com/ziozzang/mac_mlx_phi-2_server
- Owner: ziozzang
- License: mit
- Created: 2023-12-15T08:23:57.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2023-12-15T08:43:56.000Z (about 1 year ago)
- Last Synced: 2024-01-26T01:46:37.510Z (about 1 year ago)
- Topics: apple, inference, llm, macos, mlx, openai-api, osx, phi-2, python, server
- Language: Python
- Homepage:
- Size: 12.7 KB
- Stars: 13
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Mac_mlx_phi-2_server
# Disclaimer
- WARNING: this code is just for Fun, toy test code!
1. There's no acclerated way to run phi-2 model on Mac except using MLX. pytorch using MPS is slow to run/play. but MLX is fast enough to.
2. llama.cpp is not yet support to run phi-2. =(
3. so, this is some toy code to run. =)
Original inference codes are came from https://github.com/ml-explore/mlx-examples/tree/main/phi2
# Feature
Test server code for Phi-2 model. support OpenAI API spec. using MacOSX system.
you can use same api with OpenAI API spec.
# How to use?
```
# Install requirements
pip install -r requirements.txt
# Convert Model to MLX format
python convert.py
# Run Server.
python phi2-server.py
# Use anyway.
curl http://localhost:5000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [{"role": "user", "content": "answer why sky is so blue?"}],
"temperature": 0.5
}'
```
# Issue
- Need to change prompt to get better inference. this is just test. any pull-requests are welcome!
----
# Phi-2
Phi-2 is a 2.7B parameter language model released by Microsoft with
performance that rivals much larger models.[^1] It was trained on a mixture of
GPT-4 outputs and clean web text.
Phi-2 efficiently runs on Apple silicon devices with 8GB of memory in 16-bit
precision.
## Setup
Download and convert the model:
```sh
python convert.py
```
This will make the `weights.npz` file which MLX can read.
## Generate
To generate text with the default prompt:
```sh
python phi2.py
```
Should give the output:
```
Answer: Mathematics is like a lighthouse that guides us through the darkness of
uncertainty. Just as a lighthouse emits a steady beam of light, mathematics
provides us with a clear path to navigate through complex problems. It
illuminates our understanding and helps us make sense of the world around us.
Exercise 2:
Compare and contrast the role of logic in mathematics and the role of a compass
in navigation.
Answer: Logic in mathematics is like a compass in navigation. It helps
```
To use your own prompt:
```sh
python phi2.py --prompt --max_tokens
```
To see a list of options run:
```sh
python phi2.py --help
```
[^1]: For more details on the model see the [blog post](
https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/)
and the [Hugging Face repo](https://huggingface.co/microsoft/phi-2)