{"id":31753711,"url":"https://github.com/servicenow/training_llm_hf","last_synced_at":"2025-10-09T17:53:45.838Z","repository":{"id":317271348,"uuid":"1064241011","full_name":"ServiceNow/training_llm_hf","owner":"ServiceNow","description":"distributed training of llms using huggingface","archived":false,"fork":false,"pushed_at":"2025-09-29T21:13:20.000Z","size":70,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-29T23:33:40.423Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ServiceNow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-25T18:39:43.000Z","updated_at":"2025-09-29T21:13:23.000Z","dependencies_parsed_at":"2025-10-01T08:19:18.147Z","dependency_job_id":null,"html_url":"https://github.com/ServiceNow/training_llm_hf","commit_stats":null,"previous_names":["servicenow/training_llm_hf"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ServiceNow/training_llm_hf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ServiceNow%2Ftraining_llm_hf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ServiceNow%2Ftraining_llm_hf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ServiceNow%2Ftraining_llm_hf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ServiceNow%2Ftraining_llm_hf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ServiceNow","download_url":"https://codeload.github.com/ServiceNow/training_llm_hf/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ServiceNow%2Ftraining_llm_hf/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001804,"owners_count":26083197,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-09T17:53:41.947Z","updated_at":"2025-10-09T17:53:45.828Z","avatar_url":"https://github.com/ServiceNow.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SFT Training Pipeline for Large Language Models\n\nA comprehensive, production-ready Supervised Fine-Tuning (SFT) pipeline for Large Language Models using HuggingFace Transformers with distributed training support.\n\n## Features\n\n- **Flexible Data Loading**: Support for HuggingFace datasets, local files (JSON, JSONL, CSV, TXT)\n- **Distributed Training**: Multi-GPU training with PyTorch DDP\n- **Memory Optimization**: Gradient checkpointing, mixed precision (FP16/BF16), quantization support\n- **Parameter-Efficient Fine-tuning**: LoRA support for resource-constrained training\n- **Advanced Monitoring**: Wandb integration, comprehensive metrics, automatic batch size calculation\n- **Production Ready**: Robust error handling, checkpointing, resumable training\n- **Modular Design**: Clean, extensible codebase with separate concerns\n\n## Quick Start\n\n### 1. Installation\n\n```bash\n# Clone the repository\ngit clone \u003cyour-repo-url\u003e\ncd sft-training-pipeline\n\n# Install dependencies\npip install -r requirements.txt\n\n# Optional: Install development dependencies\npip install -r requirements-dev.txt\n```\n\n### 2. Prepare Your Data\n\nThe pipeline supports multiple data formats:\n\n#### JSONL Format (Recommended)\n```json\n{\"instruction\": \"Translate to French\", \"input\": \"Hello world\", \"output\": \"Bonjour le monde\"}\n{\"instruction\": \"Summarize\", \"input\": \"Long text...\", \"output\": \"Summary...\"}\n```\n\n#### Chat Format\n```json\n{\"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}, {\"role\": \"assistant\", \"content\": \"Hi there!\"}]}\n```\n\n#### Simple Text Format\n```json\n{\"text\": \"This is a training example with input and expected output.\"}\n```\n\n### 3. Configure Training\n\nCreate or modify the configuration file:\n\n```bash\ncp sample_config.yaml my_config.yaml\n# Edit my_config.yaml according to your needs\n```\n\nKey configuration options:\n```yaml\n# Model\nmodel_name_or_path: \"microsoft/DialoGPT-medium\"\n\n# Data\ndataset_path: \"./data\"\ntrain_file: \"train.jsonl\"\nvalidation_file: \"validation.jsonl\"\nmax_seq_length: 512\n\n# Training\nper_device_train_batch_size: 4\ngradient_accumulation_steps: 1\nlearning_rate: 5e-5\nnum_train_epochs: 3\nnum_gpus: 2  # Number of GPUs to use\n\n# Optimization\nfp16: true\ngradient_checkpointing: true\n\n# Optional: LoRA for parameter-efficient training\nlora_config:\n  r: 16\n  lora_alpha: 32\n  target_modules: [\"q_proj\", \"v_proj\"]\n```\n\n### 4. Launch Training\n\n#### Single GPU Training\n```bash\npython sft_main.py --config_file my_config.yaml\n```\n\n#### Multi-GPU Training\n```bash\n# Using the launch script (recommended)\n./launch_training.sh --config my_config.yaml --gpus 4\n\n# Or using torchrun directly\ntorchrun --nproc_per_node=4 sft_main.py --config_file my_config.yaml\n```\n\n#### SLURM Cluster\n```bash\nsbatch slurm_job.sh  # See examples/slurm_job.sh\n```\n\n## Architecture Overview\n\n```\nsft-training-pipeline/\n├── sft_main.py              # Main training script\n├── config_manager.py        # Configuration management\n├── data_handler.py          # Data loading and preprocessing\n├── model_manager.py         # Model and tokenizer management\n├── trainer_utils.py         # Custom trainer with enhanced features\n├── distributed_utils.py     # Distributed training utilities\n├── sample_config.yaml       # Sample configuration file\n├── launch_training.sh       # Distributed training launcher\n└── requirements.txt         # Python dependencies\n```\n\n## Configuration Guide\n\n### Data Configuration\n\n```yaml\n# Load from HuggingFace Hub\ndataset_name: \"squad\"\ndataset_config_name: \"v2.0\"\n\n# Or load from local files\ndataset_path: \"./data\"\ntrain_file: \"train.jsonl\"\nvalidation_file: \"val.jsonl\"\n\n# Preprocessing\nmax_seq_length: 512\npreprocessing_num_workers: 4\n```\n\n### Training Configuration\n\n```yaml\n# Batch size and optimization\nper_device_train_batch_size: 4      # Adjust based on GPU memory\ngradient_accumulation_steps: 2      # Effective batch size = 4 * 2 * num_gpus\nlearning_rate: 5e-5\nweight_decay: 0.01\n\n# Training schedule\nnum_train_epochs: 3\nwarmup_steps: 500\nlr_scheduler_type: \"linear\"\n\n# Memory optimization\nfp16: true                          # Mixed precision training\ngradient_checkpointing: true        # Trade compute for memory\n```\n\n### Distributed Training\n\n```yaml\nnum_gpus: 4                         # Number of GPUs\nddp_backend: \"nccl\"                 # Communication backend\n```\n\n### Parameter-Efficient Fine-tuning\n\n```yaml\nlora_config:\n  r: 16                             # Rank of adaptation\n  lora_alpha: 32                    # LoRA scaling parameter\n  target_modules: [\"q_proj\", \"v_proj\", \"k_proj\", \"o_proj\"]\n  lora_dropout: 0.1\n  bias: \"none\"\n  task_type: \"CAUSAL_LM\"\n```\n\n### Quantization (Memory Saving)\n\n```yaml\nquantization_config:\n  load_in_4bit: true\n  bnb_4bit_quant_type: \"nf4\"\n  bnb_4bit_compute_dtype: \"float16\"\n  bnb_4bit_use_double_quant: true\n```\n\n## Advanced Features\n\n### Automatic Batch Size Calculation\n\nThe pipeline automatically calculates optimal batch sizes based on:\n- GPU memory available\n- Model size\n- Sequence length\n- Target memory utilization\n\n### Comprehensive Metrics\n\n- **Perplexity**: Language model quality metric\n- **Token Accuracy**: Token-level prediction accuracy\n- **Sequence Accuracy**: Full sequence match accuracy\n- **Training Speed**: Samples per second, GPU utilization\n- **Memory Usage**: GPU memory allocation and utilization\n\n### Monitoring and Logging\n\n#### Weights \u0026 Biases Integration\n```yaml\nreport_to: \"wandb\"\nrun_name: \"my_sft_experiment\"\nwandb_project: \"llm-fine-tuning\"\n```\n\n#### TensorBoard Support\n```yaml\nreport_to: \"tensorboard\"\nlogging_dir: \"./logs\"\n```\n\n### Checkpointing and Resuming\n\n```yaml\nresume_from_checkpoint: \"./sft_output/checkpoint-1000\"\nsave_steps: 500\nsave_total_limit: 3\nload_best_model_at_end: true\n```\n\n## Data Format Examples\n\n### Instruction-Following Format (Alpaca Style)\n```json\n{\n  \"instruction\": \"Write a haiku about programming\",\n  \"input\": \"\",\n  \"output\": \"Code flows like water\\nBugs emerge from hidden depths\\nDebug, then release\"\n}\n```\n\n### Chat/Conversation Format\n```json\n{\n  \"messages\": [\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n    {\"role\": \"user\", \"content\": \"What is machine learning?\"},\n    {\"role\": \"assistant\", \"content\": \"Machine learning is a subset of AI...\"}\n  ]\n}\n```\n\n### Question-Answer Format\n```json\n{\n  \"input\": \"What is the capital of France?\",\n  \"output\": \"The capital of France is Paris.\"\n}\n```\n\n### Simple Text Completion\n```json\n{\n  \"text\": \"The quick brown fox jumps over the lazy dog.\"\n}\n```\n\n## Performance Optimization Tips\n\n### Memory Optimization\n1. **Gradient Checkpointing**: Trade compute for memory\n   ```yaml\n   gradient_checkpointing: true\n   ```\n\n2. **Mixed Precision**: Use FP16 or BF16\n   ```yaml\n   fp16: true  # or bf16: true for newer hardware\n   ```\n\n3. **Quantization**: 4-bit or 8-bit quantization\n   ```yaml\n   quantization_config:\n     load_in_4bit: true\n   ```\n\n4. **LoRA**: Parameter-efficient fine-tuning\n   ```yaml\n   lora_config:\n     r: 16\n     target_modules: [\"q_proj\", \"v_proj\"]\n   ```\n\n### Training Speed Optimization\n1. **Batch Size**: Use largest batch size that fits in memory\n2. **Gradient Accumulation**: Simulate larger batches\n3. **DataLoader Workers**: Parallel data loading\n   ```yaml\n   dataloader_num_workers: 4\n   preprocessing_num_workers: 8\n   ```\n\n### Distributed Training Best Practices\n1. **Backend Selection**: Use NCCL for GPU training\n2. **Network**: Use high-bandwidth interconnects (InfiniBand)\n3. **Data Sharding**: Ensure balanced data distribution\n\n## Troubleshooting\n\n### Common Issues\n\n#### Out of Memory (OOM) Errors\n```bash\n# Reduce batch size\nper_device_train_batch_size: 2\n\n# Enable gradient checkpointing\ngradient_checkpointing: true\n\n# Use mixed precision\nfp16: true\n\n# Consider quantization\nquantization_config:\n  load_in_4bit: true\n```\n\n#### Distributed Training Issues\n```bash\n# Check CUDA devices\nnvidia-smi\n\n# Verify network connectivity\nping \u003cother_node_ip\u003e\n\n# Check for hanging processes\nps aux | grep python\n\n# Kill hanging processes\npkill -f \"python.*sft_main.py\"\n```\n\n#### Data Loading Issues\n```bash\n# Check file formats\nhead -n 1 data/train.jsonl | python -m json.tool\n\n# Validate data\npython -c \"\nimport json\nwith open('data/train.jsonl') as f:\n    for i, line in enumerate(f):\n        try:\n            json.loads(line)\n        except:\n            print(f'Invalid JSON at line {i+1}: {line[:100]}')\n        if i \u003e 10:\n            break\n\"\n```\n\n### Performance Monitoring\n\n#### System Resources\n```bash\n# Monitor GPU usage\nwatch -n 1 nvidia-smi\n\n# Monitor CPU and memory\nhtop\n\n# Monitor network (for distributed training)\niftop\n```\n\n#### Training Progress\n```python\n# Custom monitoring script\nimport wandb\n\n# Log custom metrics\nwandb.log({\n    \"custom_metric\": value,\n    \"epoch\": epoch,\n    \"step\": step\n})\n```\n\n## Examples\n\n### Basic Fine-tuning\n```yaml\nmodel_name_or_path: \"gpt2\"\ndataset_path: \"./data\"\ntrain_file: \"train.jsonl\"\nper_device_train_batch_size: 8\nnum_train_epochs: 3\nlearning_rate: 5e-5\n```\n\n### Large Model with LoRA\n```yaml\nmodel_name_or_path: \"microsoft/DialoGPT-large\"\nper_device_train_batch_size: 2\ngradient_checkpointing: true\nfp16: true\nlora_config:\n  r: 32\n  lora_alpha: 64\n  target_modules: [\"c_attn\", \"c_proj\"]\n```\n\n### Multi-GPU Training\n```yaml\nnum_gpus: 4\nper_device_train_batch_size: 4\ngradient_accumulation_steps: 2\n# Effective batch size: 4 * 4 * 2 = 32\n```\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch: `git checkout -b feature-name`\n3. Make changes and add tests\n4. Run tests: `pytest tests/`\n5. Submit a pull request\n\n## License\n\nMIT License - see LICENSE file for details.\n\n## Citation\n\nIf you use this pipeline in your research, please cite:\n\n```bibtex\n@software{sft_training_pipeline,\n  title={SFT Training Pipeline for Large Language Models},\n  author={Your Name},\n  year={2024},\n  url={https://github.com/your-username/sft-training-pipeline}\n}\n```\n\n## Support\n\nFor questions and support:\n- Create an issue on GitHub\n- Check the documentation\n- Join our Discord community\n\n---\n\n**Happy Fine-tuning! 🚀**","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fservicenow%2Ftraining_llm_hf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fservicenow%2Ftraining_llm_hf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fservicenow%2Ftraining_llm_hf/lists"}