https://github.com/Prismadic/magnet
the small distributed language model toolkit; fine-tune state-of-the-art LLMs anywhere, rapidly
https://github.com/Prismadic/magnet
apple-silicon claude distributed-computing distributed-systems embeddings fine-tuning finetuning-llms gemini huggingface inference-api langchain llm-training milvus mistral mlx nats nats-messaging nats-streaming sentence-splitting tokenizers
Last synced: about 2 months ago
JSON representation
the small distributed language model toolkit; fine-tune state-of-the-art LLMs anywhere, rapidly
- Host: GitHub
- URL: https://github.com/Prismadic/magnet
- Owner: Prismadic
- License: mit
- Created: 2023-12-12T14:11:21.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-03-29T02:28:43.000Z (over 1 year ago)
- Last Synced: 2024-05-02T15:18:57.022Z (over 1 year ago)
- Topics: apple-silicon, claude, distributed-computing, distributed-systems, embeddings, fine-tuning, finetuning-llms, gemini, huggingface, inference-api, langchain, llm-training, milvus, mistral, mlx, nats, nats-messaging, nats-streaming, sentence-splitting, tokenizers
- Language: Python
- Homepage: https://prismadic.github.io/magnet/
- Size: 8.82 MB
- Stars: 18
- Watchers: 3
- Forks: 1
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
magnet
📖 docs | 💻 examples | 📓 substack
the small distributed language model toolkit
⚡️ fine-tune state-of-the-art LLMs anywhere, rapidly ⚡️
## 🧬 Installation
``` bash
pip install llm-magnet
```
or
``` bash
python3 setup.py install
```
## 🎉 usage
[check out the example notebooks](./examples/)
a snippet to get you started
``` python
from magnet.base import Magnet
from magnet.base import EmbeddedMagnet
cluster = EmbeddedMagnet()
cluster.start()
magnet = cluster.create_magnet()
await magnet.align()
config = {
"host": "127.0.0.1",
"credentials": None,
"domain": None,
"name": "my_stream",
"category": "my_category",
"kv_name": "my_kv",
"session": "my_session",
"os_name": "my_object_store",
"index": {
"milvus_uri": "127.0.0.1",
"milvus_port": 19530,
"milvus_user": "test",
"milvus_password": "test",
"dimension": 1024,
"model": "BAAI/bge-large-en-v1.5",
"name": "test",
"options": {
'metric_type': 'COSINE',
'index_type':'HNSW',
'params': {
"efConstruction": 40
, "M": 48
}
}
}
}
magnet = Magnet(config)
await magnet.align()
```
## 🔮 features
- ⚡️ **It's Fast**
- fast on consumer hardware
- _very_ fast on Apple Silicon
- **extremely** fast on ROCm/CUDA
- 🫵 **Automatic or your way**
- rely on established transformer patterns to let `magnet` do the work
- keep your existing data processing functions, bring them to `magnet`!
- 🛰️ **100% Distributed**
- processing, embedding, storage, retrieval, querying, or inference from anywhere
- as much or as little compute as you need
- 🧮 **Choose Inference Method**
- HuggingFace
- vLLM node
- GPU
- mlx
- 🌎 **Huge Volumes**
- handle gigantic amounts of data inexpensively
- fault-tolerant by design
- decentralized workloads
- 🔐 **Secure**
- JWT
- Basic
- 🪵 **World-Class Comprehension**
- `magnet` optionally logs its own code as it's executed (yes, really)
- build a self-aware system and allow it to learn from itself
- emojis are the future
## 🧲 why
- build a distributed LLM research node with any hardware, from Rasbperry Pi to the expensive cloud
- Apple silicon first-class citizen with [mlx](https://github.com/ml-explore/mlx)
- embed & index to vector db with [milvus](https://milvus.io)
- distributed processing with [NATS](https://nats.io)
- upload to S3
- ideal cyberpunk vision of LLM power users in vectorspace