{"id":44343398,"url":"https://github.com/pplmx/llm","last_synced_at":"2026-02-11T13:11:08.601Z","repository":{"id":278548536,"uuid":"935988302","full_name":"pplmx/llm","owner":"pplmx","description":null,"archived":false,"fork":false,"pushed_at":"2026-02-10T15:02:03.000Z","size":1197,"stargazers_count":1,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-10T19:55:46.630Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://blog.yoooo.fun/llm/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pplmx.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE-APACHE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-02-20T10:52:28.000Z","updated_at":"2026-01-14T07:34:30.000Z","dependencies_parsed_at":"2026-01-06T04:04:59.622Z","dependency_job_id":null,"html_url":"https://github.com/pplmx/llm","commit_stats":null,"previous_names":["pplmx/llm"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/pplmx/llm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplmx%2Fllm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplmx%2Fllm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplmx%2Fllm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplmx%2Fllm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pplmx","download_url":"https://codeload.github.com/pplmx/llm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplmx%2Fllm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29333170,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-11T12:42:24.625Z","status":"ssl_error","status_checked_at":"2026-02-11T12:41:23.344Z","response_time":97,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-11T13:11:07.821Z","updated_at":"2026-02-11T13:11:08.585Z","avatar_url":"https://github.com/pplmx.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# llm\n\n[![Generated by](https://img.shields.io/badge/Generated%20by-x--pt%2Ftemplate-blue)](https://github.com/x-pt/template)\n[![CI](https://github.com/pplmx/llm/workflows/CI/badge.svg)](https://github.com/pplmx/llm/actions)\n[![Coverage Status](https://coveralls.io/repos/github/pplmx/llm/badge.svg?branch=main)](https://coveralls.io/github/pplmx/llm?branch=main)\n[![PyPI version](https://badge.fury.io/py/llm.svg)](https://badge.fury.io/py/llm)\n[![Python Versions](https://img.shields.io/pypi/pyversions/llm.svg)](https://pypi.org/project/llm/)\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Features](#features)\n- [Quick Start](#quick-start)\n- [Installation](#installation)\n- [Usage](#usage)\n- [Development](#development)\n- [Troubleshooting](#troubleshooting)\n- [Contributing](#contributing)\n- [License](#license)\n- [Roadmap](#roadmap)\n- [Changelog](#changelog)\n- [Contact](#contact)\n- [Acknowledgements](#acknowledgements)\n\n## Overview\n\n`llm` is a modular and extensible PyTorch framework for training and experimenting with Large Language Models (LLMs). It provides a robust infrastructure for building, training, and evaluating LLMs, focusing on modularity, scalability, and ease of experimentation.\n\n## Features\n\n- **Flexible \u0026 Pluggable Architecture**:\n    - **Component Registry**: Switch between MHA/MoE or future implementations (FlashAttn) via config.\n    - Grouped Query Attention (GQA) \u0026 SwiGLU support.\n    - Unified QKV projection \u0026 Flexible Norms (RMSNorm, LayerNorm).\n- **Robust Training Framework**:\n    - Distributed Data Parallel (DDP) \u0026 Automatic Mixed Precision (AMP).\n    - Type-safe Configuration via Pydantic \u0026 Typer CLI.\n    - `torch.compile` optimization integration.\n- **Parameter-Efficient Fine-Tuning**:\n    - **LoRA**: Low-rank adaptation with merge/unmerge support.\n    - **QLoRA**: 4-bit NF4 quantized base + fp16 LoRA adapters (~4x memory reduction).\n- **Inference \u0026 Serving**:\n    - **OpenAI-Compatible API**: `/v1/chat/completions` endpoint works with OpenAI SDK.\n    - **KVCache**: Pre-allocated cache for efficient autoregressive generation.\n    - Streaming support (Server-Sent Events).\n    - Prometheus metrics \u0026 API key authentication.\n- **Data \u0026 Tokenization Abstraction**:\n    - **HuggingFace Integration**: Direct support for HF Tokenizers (GPT-2, Llama, etc.).\n    - Modular `DataModule` design for text datasets.\n    - Legacy Character-level tokenizer for simple experiments.\n- **CLI Tools**: `llm-train` and `llm-serve` commands for easy usage.\n\n## Quick Start\n\nTo quickly get started with training a model using the framework, follow these steps:\n\n1. **Initialize Project**: Ensure your environment is set up by running the initialization command (see [Installation](#installation) for details).\n\n    ```bash\n    make init\n    ```\n\n2. **Run a Training Example**: Execute the main training script with a language modeling task.\n\n    ```bash\n    llm-train --task lm --epochs 1 --batch-size 32\n    ```\n\n    This command will train a language model for 1 epoch with a batch size of 32.\n\n3. **Start Inference Server** (optional):\n\n    ```bash\n    llm-serve\n    ```\n\n    The server will be available at `http://localhost:8000` with OpenAI-compatible API.\n\n## Installation\n\n### Requirements\n\n- Python 3.13+\n- [uv](https://github.com/astral-sh/uv): A fast Python package installer and resolver, written in Rust.\n- [make](https://www.gnu.org/software/make/): A build automation tool (typically pre-installed on Linux/macOS, available via Chocolatey/Scoop on Windows).\n\n### Setting up the Environment\n\nThis project uses `uv` for dependency management and `Makefile` for common development tasks.\n\n1. **Install `uv`**: If you don't have `uv` installed, follow the official instructions [here](https://github.com/astral-sh/uv#installation).\n2. **Initialize Project**: Navigate to the project root directory and run the `make init` command. This will set up the virtual environment, install all necessary dependencies, and install pre-commit hooks.\n\n    ```bash\n    make init\n    ```\n\n3. **Synchronize Dependencies (if needed)**: If `pyproject.toml` or `uv.lock` changes, you can re-synchronize dependencies:\n\n    ```bash\n    make sync\n    ```\n\n### User Installation (Distribution)\n\nIf this project were to be distributed as a package, users would typically install it using pip:\n\n```bash\npip install llm # Assuming 'llm' is the package name on PyPI\n```\n\nHowever, for development, `make init` is the recommended way to set up the environment.\n\n## Usage\n\nFor comprehensive documentation, including detailed usage examples, development guides, and troubleshooting, please refer to our dedicated documentation section:\n\n- [Architecture Guide](docs/architecture.md)\n- [Fine-Tuning Guide (LoRA/QLoRA)](docs/guide-finetuning.md)\n- [Inference Optimization Guide](docs/guide-inference.md)\n- [Development Guide](docs/development.md)\n- [CPU LLM Tutorial](docs/tutorial-cpu-llm.md)\n- [Project Troubleshooting](docs/troubleshooting.md)\n- [Training Framework Documentation](docs/training/README.md)\n- [Examples](examples/README.md)\n- [Scripts](scripts/README.md)\n- [Notebooks](notebooks/README.md)\n\n## Development\n\nFor detailed information on setting up the development environment, running tests, maintaining code quality, and other development workflows, please refer to our comprehensive [Development Guide](docs/development.md).\n\n## Troubleshooting\n\nIf you encounter any issues while using `llm`, please check our [Troubleshooting Guide](docs/troubleshooting.md) for common problems and their solutions. If you can't find a solution to your problem, please [open an issue](https://github.com/pplmx/llm/issues) on our GitHub repository.\n\n## Contributing\n\nWe welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details on how to submit pull requests, report issues, or suggest improvements.\n\n## License\n\nThis project is licensed under either of:\n\n- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or \u003chttp://www.apache.org/licenses/LICENSE-2.0\u003e)\n- MIT license ([LICENSE-MIT](LICENSE-MIT) or \u003chttp://opensource.org/licenses/MIT\u003e)\n\nat your option.\n\n## Roadmap\n\nFor our detailed development roadmap and future plans, including upcoming features like inference API, Flash Attention integration, and RLHF support, please see [ROADMAP.md](ROADMAP.md).\n\n## Changelog\n\nFor a detailed history of changes to this project, please refer to our [CHANGELOG.md](CHANGELOG.md).\n\n## Contact\n\nFor questions, suggestions, or support, please open an issue on our GitHub repository.\n\n## Acknowledgements\n\nWe acknowledge all contributors, open-source projects, and resources that have inspired and supported the development of this project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpplmx%2Fllm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpplmx%2Fllm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpplmx%2Fllm/lists"}