{"id":30821168,"url":"https://github.com/helpingai/llm-trainer","last_synced_at":"2025-09-06T10:47:38.510Z","repository":{"id":305814012,"uuid":"1024022558","full_name":"HelpingAI/llm-trainer","owner":"HelpingAI","description":"A complete framework for training Large Language Models from scratch","archived":false,"fork":false,"pushed_at":"2025-08-28T08:00:29.000Z","size":228,"stargazers_count":7,"open_issues_count":0,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-28T14:44:11.929Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HelpingAI.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-22T04:35:46.000Z","updated_at":"2025-08-28T08:00:33.000Z","dependencies_parsed_at":"2025-07-22T06:48:01.579Z","dependency_job_id":"61026257-865e-467a-b5bb-438910f8c6fc","html_url":"https://github.com/HelpingAI/llm-trainer","commit_stats":null,"previous_names":["oevortex/llm-trainer","helpingai/llm-trainer"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/HelpingAI/llm-trainer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HelpingAI%2Fllm-trainer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HelpingAI%2Fllm-trainer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HelpingAI%2Fllm-trainer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HelpingAI%2Fllm-trainer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HelpingAI","download_url":"https://codeload.github.com/HelpingAI/llm-trainer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HelpingAI%2Fllm-trainer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273892840,"owners_count":25186561,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-06T02:00:13.247Z","response_time":2576,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-06T10:47:37.419Z","updated_at":"2025-09-06T10:47:38.486Z","avatar_url":"https://github.com/HelpingAI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLM Trainer\n\n\u003cdiv align=\"center\"\u003e\n\n[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)\n[![PyTorch](https://img.shields.io/badge/PyTorch-2.0+-red.svg)](https://pytorch.org/)\n[![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](https://opensource.org/licenses/Apache-2.0)\n[![GitHub](https://img.shields.io/badge/GitHub-HelpingAI/llm--trainer-black.svg)](https://github.com/HelpingAI/llm-trainer)\n[![SafeTensors](https://img.shields.io/badge/SafeTensors-Supported-brightgreen.svg)](https://github.com/huggingface/safetensors)\n[![Version](https://img.shields.io/badge/version-0.2.4-blue.svg)](https://github.com/HelpingAI/llm-trainer/releases)\n\n*A production-ready framework for training Large Language Models from scratch with modern PyTorch*\n\n\u003c/div\u003e\n\n## What's New in v0.2.4\n\n- **Memory Optimizations**: Efficient training with kernel optimizations\n- **SafeTensors Support**: Secure model serialization with automatic sharding for large models\n- **HuggingFace Integration**: Use any pretrained tokenizer via `HFTokenizerWrapper`\n- **Accelerate Support**: Distributed training with `use_accelerate=true`\n- **LoRA/PEFT**: Parameter-efficient fine-tuning with `use_peft=true`\n- **Backward Compatible**: Existing PyTorch models continue to work\n- **Patching System**: Kernel optimizations and memory-efficient training\n\n## Features\n\n### Core Architecture\n- **Custom Transformer Implementation**: Multi-head attention, feed-forward networks, positional encodings\n- **SafeTensors Integration**: Secure model serialization with automatic sharding\n- **Modular Design**: Easy to extend and customize for research and production\n\n### Tokenization\n- **BPE Tokenizer**: From-scratch BPE with Unicode and emoji support\n- **HuggingFace Integration**: Use any pretrained tokenizer (Mistral, Llama, GPT-2, etc.)\n- **WordPiece Support**: Alternative tokenization strategies\n\n### Data Pipeline\n- **HuggingFace Datasets**: Efficient loading with preprocessing and batching\n- **Memory Optimization**: Smart sequence packing and data streaming\n- **Multi-Processing**: Parallel data preprocessing for faster training\n\n### Training \u0026 Inference\n- **CPU/GPU Support**: Optimized configurations for both CPU and GPU training\n- **Distributed Training**: Multi-GPU support via Accelerate and DeepSpeed\n- **Parameter-Efficient**: LoRA/PEFT adapters for memory-efficient fine-tuning\n- **Mixed Precision**: FP16/BF16 automatic mixed precision\n- **Multiple Decoding Strategies**: Greedy, beam search, nucleus (top-p), top-k sampling\n- **Enhanced Trainer**: TRL-style training methods with familiar APIs\n- **Memory-Efficient Optimizers**: Optimized implementations for better performance\n- **Kernel Optimizations**: Fused operations for better performance\n- **Low VRAM Training**: Gradient checkpointing and memory-efficient techniques\n\n### Monitoring \u0026 Evaluation\n- **TensorBoard Integration**: Real-time training metrics and visualizations\n- **Weights \u0026 Biases**: Experiment tracking and hyperparameter optimization\n- **Comprehensive Metrics**: Perplexity, cross-entropy loss, generation quality\n\n## Requirements\n\n- Python 3.8 or higher\n- PyTorch 2.0 or higher\n- GPU: CUDA-compatible GPU (recommended) or CPU-only mode\n- Memory: 8GB RAM minimum (16GB+ recommended)\n\n## Installation\n\n### Basic Installation\n```bash\ngit clone https://github.com/HelpingAI/llm-trainer.git\ncd llm-trainer\npip install -e .\n```\n\n### Optional Dependencies\n\n```bash\n# Development tools\npip install -e \".[dev]\"\n\n# SafeTensors support (recommended)\npip install -e \".[safetensors]\"\n\n# Distributed training\npip install -e \".[distributed]\"\n\n# All features\npip install -e \".[full]\"\n```\n\n## Quick Start\n\n### Python API - Enhanced Training\n\n```python\nfrom llm_trainer import Trainer, TrainingConfig\nfrom llm_trainer.models import TransformerLM\nfrom llm_trainer.config import ModelConfig\nfrom llm_trainer.tokenizer import BPETokenizer\n\n# Create model and tokenizer\nmodel_config = ModelConfig(\n    vocab_size=32000,\n    d_model=512,\n    n_heads=8,\n    n_layers=6,\n    max_seq_len=1024\n)\nmodel = TransformerLM(model_config)\ntokenizer = BPETokenizer()\n\n# Configure training with TRL-style parameters\ntraining_config = TrainingConfig(\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=2,\n    learning_rate=2e-5,\n    num_train_epochs=3,\n    logging_steps=10,\n    save_steps=100,\n    optim=\"adamw\"  # TRL-style parameter\n)\n\n# Create trainer and train\ntrainer = Trainer(\n    model=model,\n    tokenizer=tokenizer,\n    config=training_config\n)\n\n# TRL-style training methods\ntrainer.train()  # Standard training\ntrainer.sft_train()  # Supervised fine-tuning\ntrainer.dpo_train()  # Direct preference optimization\n```\n\n### HuggingFace Integration with PEFT\n\n```python\nfrom llm_trainer import Trainer, TrainingConfig\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom peft import LoraConfig, TaskType\n\n# Load pretrained model and tokenizer\nmodel = AutoModelForCausalLM.from_pretrained(\"gpt2\")\ntokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\n# Configure LoRA (PEFT)\nlora_config = LoraConfig(\n    r=8,\n    lora_alpha=16,\n    lora_dropout=0.05,\n    task_type=TaskType.CAUSAL_LM\n)\n\n# Create trainer with PEFT\ntrainer = Trainer(\n    model=model,\n    tokenizer=tokenizer,\n    config=TrainingConfig(),\n    peft_config=lora_config  # Pass PEFT config directly\n)\n\n# Show parameter efficiency\ntrainer.print_trainable_parameters()\n\n# Train with familiar API\ntrainer.train()\n```\n\n### Memory-Efficient Optimizers\n\n```python\nfrom llm_trainer.training import create_optimizer\n\n# Create memory-efficient optimizer\noptimizer = create_optimizer(\n    model,\n    optimizer_name=\"adamw\",\n    learning_rate=5e-5,\n    weight_decay=0.01\n)\n```\n\n### Patching for Transformers/TRL\n\n```python\nfrom llm_trainer import patch_transformers, patch_trl\n\n# Patch Hugging Face Transformers with memory-efficient optimizations\npatch_transformers()\n\n# Patch TRL with memory-efficient optimizations\npatch_trl()\n\n# Now you can use enhanced Transformers/TRL classes with memory-efficient methods\nfrom transformers import Trainer, TrainingArguments\nfrom trl import SFTTrainer\n\n# These trainers now have enhanced methods\ntrainer = SFTTrainer(...)\ntrainer.print_trainable_parameters()  # Added by patching\ntrainer.prepare_model_for_kbit_training()  # Added by patching\n```\n\n### Kernel Optimizations for Fast Training\n\n```python\nfrom llm_trainer.kernels import (\n    FusedLinear, FusedRMSNorm, fused_cross_entropy,\n    gradient_checkpointing, LowVRAMLinear, empty_cache\n)\n\n# Use fused operations for better performance\nfused_linear = FusedLinear(in_features=512, out_features=512)\nfused_norm = FusedRMSNorm(dim=512)\n\n# Use gradient checkpointing to reduce memory usage\ndef forward_pass_with_checkpointing(model, inputs):\n    return gradient_checkpointing(model, inputs)\n\n# Use low VRAM linear layers for memory-efficient training\nlow_vram_linear = LowVRAMLinear(in_features=512, out_features=512)\n\n# Clear cache to free up memory\nempty_cache()\n```\n\n### Command Line\n\n```bash\n# GPU Training\npython scripts/train.py --config configs/small_model.yaml --output_dir ./output\n\n# CPU Training (no GPU required)\npython scripts/train.py --config configs/cpu_small_model.yaml --output_dir ./output\n\n# Text Generation\npython scripts/generate.py --model_path ./output --prompts \"The quick brown fox\" --interactive\n\n# Model Evaluation\npython scripts/evaluate.py --model_path ./output --dataset_config configs/eval_config.json\n```\n\n## Configuration\n\nThe framework uses YAML/JSON configuration files for reproducible experiments:\n\n### Small Model (Quick Start)\n```yaml\nmodel:\n  d_model: 512\n  n_heads: 8\n  n_layers: 6\n  vocab_size: 32000\n  max_seq_len: 1024\n\ntraining:\n  batch_size: 16\n  learning_rate: 1e-4\n  num_epochs: 3\n  use_amp: true\n  gradient_accumulation_steps: 4\n```\n\n### CPU-Optimized Training\n```yaml\ndevice: \"cpu\"\nmodel:\n  d_model: 256\n  n_heads: 4\n  n_layers: 4\n  max_seq_len: 512\n\ntraining:\n  batch_size: 2\n  use_amp: false\n  gradient_accumulation_steps: 8\n  dataloader_num_workers: 2\n```\n\n### Advanced Configuration\n```yaml\nmodel:\n  d_model: 768\n  n_heads: 12\n  n_layers: 12\n\ntraining:\n  use_accelerate: true\n  accelerate_mixed_precision: \"fp16\"\n  use_peft: true\n  peft_type: \"lora\"\n  peft_r: 8\n  peft_alpha: 16\n\n# SafeTensors settings\nsave_format: \"safetensors\"\nmax_shard_size: \"2GB\"\n```\n\n## Project Structure\n\n```\nllm-trainer/\n├── src/llm_trainer/              # Main package\n│   ├── models/                   # Model architectures\n│   │   ├── base_model.py         # Base model interface\n│   │   ├── transformer.py        # Custom Transformer implementation\n│   │   ├── safetensors_utils.py  # SafeTensors utilities\n│   │   └── attention.py          # Attention mechanisms\n│   ├── tokenizer/                # Tokenization\n│   │   ├── bpe_tokenizer.py      # BPE implementation\n│   │   ├── hf_tokenizer.py       # HuggingFace wrapper\n│   │   └── wordpiece_tokenizer.py # WordPiece implementation\n│   ├── data/                     # Data pipeline\n│   │   ├── dataset.py            # Dataset classes\n│   │   ├── dataloader.py         # Data loading\n│   │   └── preprocessing.py      # Data preprocessing\n│   ├── training/                 # Training infrastructure\n│   │   ├── trainer.py            # Enhanced trainer with TRL-style APIs\n│   │   ├── optimizer.py          # Standard optimizers\n│   │   └── scheduler.py          # Learning rate schedulers\n│   ├── kernels/                  # Kernel optimizations\n│   │   ├── fused_ops.py          # Fused operations\n│   │   └── memory_efficient.py   # Memory-efficient operations\n│   ├── patching/                 # Patching system\n│   │   ├── patch_transformers.py # Transformers patching\n│   │   └── patch_trl.py          # TRL patching\n│   ├── utils/                    # Utilities\n│   │   ├── generation.py         # Text generation\n│   │   ├── inference.py          # Inference utilities\n│   │   └── metrics.py            # Evaluation metrics\n│   └── config/                   # Configuration\n│       ├── model_config.py       # Model configuration\n│       └── training_config.py    # Training configuration\n├── scripts/                      # CLI tools\n│   ├── train.py                  # Training script\n│   ├── generate.py               # Text generation\n│   └── evaluate.py               # Model evaluation\n├── configs/                      # Pre-configured setups\n│   ├── small_model.yaml          # Small GPU model\n│   ├── medium_model.yaml         # Medium GPU model\n│   ├── cpu_small_model.yaml      # CPU-optimized small\n│   └── cpu_medium_model.yaml     # CPU-optimized medium\n├── examples/                     # Usage examples\n│   ├── complete_pipeline.py      # End-to-end example\n│   ├── safetensors_example.py    # SafeTensors demo\n│   └── train_small_model.py      # Quick start example\n└── docs/                         # Documentation\n```\n\n## Documentation\n\n- [Getting Started Guide](docs/getting_started.md) — Complete setup and first steps\n- [Model Architecture](docs/architecture.md) — Transformer implementation details\n- [Training Guide](docs/training.md) — Comprehensive training tutorial\n- [CPU Training Guide](docs/cpu_training.md) — Dedicated CPU training documentation\n- [Tokenizer Details](docs/tokenizer.md) — BPE tokenizer documentation\n- [API Reference](docs/api.md) — Complete API documentation\n\n## Development\n\n### Running Tests\n```bash\npip install -e \".[dev]\"\npytest tests/\n```\n\n### Code Quality\n```bash\nblack src/ scripts/ examples/\nflake8 src/ scripts/ examples/\nmypy src/\n```\n\n## Contributing\n\nWe welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md) for details.\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n\n## Support\n\n- **Bug Reports**: [GitHub Issues](https://github.com/HelpingAI/llm-trainer/issues)\n- **Discussions**: [GitHub Discussions](https://github.com/HelpingAI/llm-trainer/discussions)\n- **Documentation**: [Read the Docs](https://github.com/HelpingAI/llm-trainer/tree/main/docs)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhelpingai%2Fllm-trainer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhelpingai%2Fllm-trainer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhelpingai%2Fllm-trainer/lists"}