https://github.com/pplmx/llm
Last synced: 6 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/pplmx/llm
- Owner: pplmx
- License: apache-2.0
- Created: 2025-02-20T10:52:28.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2026-02-10T15:02:03.000Z (6 months ago)
- Last Synced: 2026-02-10T19:55:46.630Z (6 months ago)
- Language: Python
- Homepage: http://blog.yoooo.fun/llm/
- Size: 1.14 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE-APACHE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
- Roadmap: ROADMAP.md
- Agents: AGENTS.md
Awesome Lists containing this project
README
# llm
[](https://github.com/x-pt/template)
[](https://github.com/pplmx/llm/actions)
[](https://coveralls.io/github/pplmx/llm?branch=main)
[](https://badge.fury.io/py/llm)
[](https://pypi.org/project/llm/)
## Table of Contents
- [Overview](#overview)
- [Features](#features)
- [Quick Start](#quick-start)
- [Installation](#installation)
- [Usage](#usage)
- [Development](#development)
- [Troubleshooting](#troubleshooting)
- [Contributing](#contributing)
- [License](#license)
- [Roadmap](#roadmap)
- [Changelog](#changelog)
- [Contact](#contact)
- [Acknowledgements](#acknowledgements)
## Overview
`llm` is a modular and extensible PyTorch framework for training and experimenting with Large Language Models (LLMs). It provides a robust infrastructure for building, training, and evaluating LLMs, focusing on modularity, scalability, and ease of experimentation.
## Features
- **Flexible & Pluggable Architecture**:
- **Component Registry**: Switch between MHA/MoE or future implementations (FlashAttn) via config.
- Grouped Query Attention (GQA) & SwiGLU support.
- Unified QKV projection & Flexible Norms (RMSNorm, LayerNorm).
- **Robust Training Framework**:
- Distributed Data Parallel (DDP) & Automatic Mixed Precision (AMP).
- Type-safe Configuration via Pydantic & Typer CLI.
- `torch.compile` optimization integration.
- **Parameter-Efficient Fine-Tuning**:
- **LoRA**: Low-rank adaptation with merge/unmerge support.
- **QLoRA**: 4-bit NF4 quantized base + fp16 LoRA adapters (~4x memory reduction).
- **Inference & Serving**:
- **OpenAI-Compatible API**: `/v1/chat/completions` endpoint works with OpenAI SDK.
- **KVCache**: Pre-allocated cache for efficient autoregressive generation.
- Streaming support (Server-Sent Events).
- Prometheus metrics & API key authentication.
- **Data & Tokenization Abstraction**:
- **HuggingFace Integration**: Direct support for HF Tokenizers (GPT-2, Llama, etc.).
- Modular `DataModule` design for text datasets.
- Legacy Character-level tokenizer for simple experiments.
- **CLI Tools**: `llm-train` and `llm-serve` commands for easy usage.
## Quick Start
To quickly get started with training a model using the framework, follow these steps:
1. **Initialize Project**: Ensure your environment is set up by running the initialization command (see [Installation](#installation) for details).
```bash
make init
```
2. **Run a Training Example**: Execute the main training script with a language modeling task.
```bash
llm-train --task lm --epochs 1 --batch-size 32
```
This command will train a language model for 1 epoch with a batch size of 32.
3. **Start Inference Server** (optional):
```bash
llm-serve
```
The server will be available at `http://localhost:8000` with OpenAI-compatible API.
## Installation
### Requirements
- Python 3.13+
- [uv](https://github.com/astral-sh/uv): A fast Python package installer and resolver, written in Rust.
- [make](https://www.gnu.org/software/make/): A build automation tool (typically pre-installed on Linux/macOS, available via Chocolatey/Scoop on Windows).
### Setting up the Environment
This project uses `uv` for dependency management and `Makefile` for common development tasks.
1. **Install `uv`**: If you don't have `uv` installed, follow the official instructions [here](https://github.com/astral-sh/uv#installation).
2. **Initialize Project**: Navigate to the project root directory and run the `make init` command. This will set up the virtual environment, install all necessary dependencies, and install pre-commit hooks.
```bash
make init
```
3. **Synchronize Dependencies (if needed)**: If `pyproject.toml` or `uv.lock` changes, you can re-synchronize dependencies:
```bash
make sync
```
### User Installation (Distribution)
If this project were to be distributed as a package, users would typically install it using pip:
```bash
pip install llm # Assuming 'llm' is the package name on PyPI
```
However, for development, `make init` is the recommended way to set up the environment.
## Usage
For comprehensive documentation, including detailed usage examples, development guides, and troubleshooting, please refer to our dedicated documentation section:
- [Architecture Guide](docs/architecture.md)
- [Fine-Tuning Guide (LoRA/QLoRA)](docs/guide-finetuning.md)
- [Inference Optimization Guide](docs/guide-inference.md)
- [Development Guide](docs/development.md)
- [CPU LLM Tutorial](docs/tutorial-cpu-llm.md)
- [Project Troubleshooting](docs/troubleshooting.md)
- [Training Framework Documentation](docs/training/README.md)
- [Examples](examples/README.md)
- [Scripts](scripts/README.md)
- [Notebooks](notebooks/README.md)
## Development
For detailed information on setting up the development environment, running tests, maintaining code quality, and other development workflows, please refer to our comprehensive [Development Guide](docs/development.md).
## Troubleshooting
If you encounter any issues while using `llm`, please check our [Troubleshooting Guide](docs/troubleshooting.md) for common problems and their solutions. If you can't find a solution to your problem, please [open an issue](https://github.com/pplmx/llm/issues) on our GitHub repository.
## Contributing
We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details on how to submit pull requests, report issues, or suggest improvements.
## License
This project is licensed under either of:
- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or )
- MIT license ([LICENSE-MIT](LICENSE-MIT) or )
at your option.
## Roadmap
For our detailed development roadmap and future plans, including upcoming features like inference API, Flash Attention integration, and RLHF support, please see [ROADMAP.md](ROADMAP.md).
## Changelog
For a detailed history of changes to this project, please refer to our [CHANGELOG.md](CHANGELOG.md).
## Contact
For questions, suggestions, or support, please open an issue on our GitHub repository.
## Acknowledgements
We acknowledge all contributors, open-source projects, and resources that have inspired and supported the development of this project.