{"id":29169084,"url":"https://github.com/astorfi/llm-alignment-project","last_synced_at":"2025-07-22T16:06:11.812Z","repository":{"id":264305408,"uuid":"892955085","full_name":"astorfi/LLM-Alignment-Project","owner":"astorfi","description":"A comprehensive template for aligning large language models (LLMs) using Reinforcement Learning from Human Feedback (RLHF), transfer learning, and more. Build your own customizable LLM alignment solution with ease.","archived":false,"fork":false,"pushed_at":"2024-12-15T05:25:50.000Z","size":634,"stargazers_count":33,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-17T17:55:18.694Z","etag":null,"topics":["ai","alignment","deep-learning","generative-ai","large-language-models","llms","machine-learning","rlhf","template"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/astorfi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"docs/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"docs/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-23T06:16:56.000Z","updated_at":"2025-05-18T18:16:38.000Z","dependencies_parsed_at":"2024-11-27T06:32:02.323Z","dependency_job_id":null,"html_url":"https://github.com/astorfi/LLM-Alignment-Project","commit_stats":null,"previous_names":["astorfi/llm-alignment-template","astorfi/llm-alignment-project-template","astorfi/llm-alignment-project"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/astorfi/LLM-Alignment-Project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astorfi%2FLLM-Alignment-Project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astorfi%2FLLM-Alignment-Project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astorfi%2FLLM-Alignment-Project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astorfi%2FLLM-Alignment-Project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/astorfi","download_url":"https://codeload.github.com/astorfi/LLM-Alignment-Project/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astorfi%2FLLM-Alignment-Project/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266524894,"owners_count":23942834,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","alignment","deep-learning","generative-ai","large-language-models","llms","machine-learning","rlhf","template"],"created_at":"2025-07-01T11:09:26.609Z","updated_at":"2025-07-22T16:06:11.786Z","avatar_url":"https://github.com/astorfi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🌌 LLM Alignment Template - Your Template for Aligning Language Models\n\n[![Build Status](https://github.com/astorfi/LLM-Alignment-Project-Template/actions/workflows/ci.yml/badge.svg)](https://github.com/astorfi/LLM-Alignment-Project-Template/actions)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Contributions Welcome](https://img.shields.io/badge/contributions-welcome-brightgreen)](https://github.com/astorfi/LLM-Alignment-Project-Template/issues)\n[![Python Version](https://img.shields.io/badge/python-3.8%2B-blue)](https://www.python.org/)\n\n## 📌 Introduction\n\n# LLM Alignment Project\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"_img/LLMalignment.png\" alt=\"LLM Alignment Project\" width=\"600\"\u003e\n  \u003cbr\u003e\n  \u003ci\u003eFigure 1: Take a look at: \u003ca href=\"https://arxiv.org/abs/2308.05374\" target=\"_blank\"\u003earXiv:2308.05374\u003c/a\u003e\u003c/i\u003e\n\u003c/p\u003e\n\n\n\n**LLM Alignment Template** is not just a comprehensive tool for aligning large language models (LLMs), but also serves as a **powerful template** for building your own LLM alignment application. Inspired by project templates like **PyTorch Project Template**, this repository is designed to provide a full stack of functionality, acting as a starting point to customize and extend for your own LLM alignment needs. Whether you are a researcher, developer, or data scientist, this template provides a solid foundation for efficiently creating and deploying LLMs tailored to align with human values and objectives.\n\n## 🚀 Overview\n\n**LLM Alignment Template** provides a full stack of functionality, including training, fine-tuning, deploying, and monitoring LLMs using Reinforcement Learning from Human Feedback (RLHF). This project also integrates evaluation metrics to ensure ethical and effective use of language models. The interface offers a user-friendly experience for managing alignment, visualizing training metrics, and deploying at scale.\n\n## ✨ Features\n\n- **🌐 Interactive Web Interface**: A user-friendly interface for interacting with the LLM, training models, and viewing alignment metrics.\n- **🧠 Training with RLHF**: Reinforcement Learning from Human Feedback to ensure model alignment with human preferences.\n- **🛠️ Data Augmentation \u0026 Preprocessing**: Advanced preprocessing, tokenization, and data augmentation with back-translation and paraphrasing.\n- **🔄 Transfer Learning**: Utilize pre-trained models like BERT for improved performance on specific tasks.\n- **📦 Scalable Deployment**: Docker and Kubernetes-based deployment with Horizontal Pod Autoscaling (HPA).\n- **🔍 Model Explainability**: SHAP-based dashboards for understanding model decisions.\n- **📊 User Feedback Loop**: Collection of user ratings for fine-tuning models continuously.\n\n## 📂 Table of Contents\n\n- [Introduction](#-introduction)\n- [Overview](#-overview)\n- [Features](#-features)\n- [Project Structure](#-project-structure)\n- [Setup](#️-setup)\n  - [Prerequisites](#prerequisites)\n  - [Installation](#installation)\n  - [Running Locally](#running-locally)\n- [Deployment](#-deployment)\n  - [Kubernetes Deployment](#kubernetes-deployment)\n  - [Canary Deployment](#canary-deployment)\n  - [Monitoring and Logging](#monitoring-and-logging)\n- [Training and Evaluation](#-training-and-evaluation)\n- [Testing](#-testing)\n- [Future Work](#-future-work)\n- [Contributing](#-contributing)\n- [License](#-license)\n- [Contact](#-contact)\n\n## 📂 Project Structure\n\n- **`app/`**: Contains API and UI code.\n  - `auth.py`, `feedback.py`, `ui.py`: API endpoints for user interaction, feedback collection, and general interface management.\n  - **Static Files**: JavaScript (`app.js`, `chart.js`), CSS (`styles.css`), and Swagger API documentation (`swagger.json`).\n  - **Templates**: HTML templates (`chat.html`, `feedback.html`, `index.html`) for UI rendering.\n\n- **`src/`**: Core logic and utilities for preprocessing and training.\n  - **Preprocessing** (`preprocessing/`):\n    - `preprocess_data.py`: Combines original and augmented datasets and applies text cleaning.\n    - `tokenization.py`: Handles tokenization.\n  - **Training** (`training/`):\n    - `fine_tuning.py`, `transfer_learning.py`, `retrain_model.py`: Scripts for training and retraining models.\n    - `rlhf.py`, `reward_model.py`: Scripts for reward model training using RLHF.\n  - **Utilities** (`utils/`): Common utilities (`config.py`, `logging.py`, `validation.py`).\n\n- **`dashboards/`**: Performance and explainability dashboards for monitoring and model insights.\n  - `performance_dashboard.py`: Displays training metrics, validation loss, and accuracy.\n  - `explainability_dashboard.py`: Visualizes SHAP values to provide insight into model decisions.\n\n- **`tests/`**: Unit, integration, and end-to-end tests.\n  - `test_api.py`, `test_preprocessing.py`, `test_training.py`: Various unit and integration tests.\n  - **End-to-End Tests** (`e2e/`): Cypress-based UI tests (`ui_tests.spec.js`).\n  - **Load Testing** (`load_testing/`): Uses Locust (`locustfile.py`) for load testing.\n\n- **`deployment/`**: Configuration files for deployment and monitoring.\n  - **Kubernetes Configurations** (`kubernetes/`): Deployment and Ingress configurations for scaling and canary releases.\n  - **Monitoring** (`monitoring/`): Prometheus (`prometheus.yml`) and Grafana (`grafana_dashboard.json`) for performance and system health monitoring.\n\n## ⚙️ Setup\n\n### Prerequisites\n\n- 🐍 Python 3.8+\n- 🐳 Docker \u0026 Docker Compose\n- ☸️ Kubernetes (Minikube or a cloud provider)\n- 🟢 Node.js (for front-end dependencies)\n\n### 📦 Installation\n\n1. **Clone the Repository**:\n   ```bash\n   git clone https://github.com/yourusername/LLM-Alignment-Template.git\n   cd LLM-Alignment-Template\n   ```\n\n2. **Install Dependencies**:\n   - Python dependencies:\n     ```bash\n     pip install -r requirements.txt\n     ```\n   - Node.js dependencies (optional for UI improvements):\n     ```bash\n     cd app/static\n     npm install\n     ```\n\n### 🏃 Running Locally\n\n1. **Build Docker Images**:\n   ```bash\n   docker-compose up --build\n   ```\n\n2. **Access the Application**:\n   - Open a browser and visit `http://localhost:5000`.\n\n## 🚢 Deployment\n\n### ☸️ Kubernetes Deployment\n\n- **Deploy to Kubernetes**:\n  - Apply the deployment and service configurations:\n    ```bash\n    kubectl apply -f deployment/kubernetes/deployment.yml\n    kubectl apply -f deployment/kubernetes/service.yml\n    ```\n  - **Horizontal Pod Autoscaler**:\n    ```bash\n    kubectl apply -f deployment/kubernetes/hpa.yml\n    ```\n\n### 🌟 Canary Deployment\n\n- Canary deployments are configured using `deployment/kubernetes/canary_deployment.yml` to roll out new versions safely.\n\n### 📈 Monitoring and Logging\n\n- **Prometheus and Grafana**:\n  - Apply Prometheus and Grafana configurations in `deployment/monitoring/` to enable monitoring dashboards.\n- **📋 Centralized Logging**: The **ELK Stack** is configured with Docker using `docker-compose.logging.yml` for centralized logs.\n\n## 🧠 Training and Evaluation\n\n### 🔄 Transfer Learning\n\nThe training module (`src/training/transfer_learning.py`) uses pre-trained models like **BERT** to adapt to custom tasks, providing a significant performance boost.\n\n### 📊 Data Augmentation\n\nThe `data_augmentation.py` script (`src/data/`) applies augmentation techniques like back-translation and paraphrasing to improve data quality.\n\n### 🧠 Reinforcement Learning from Human Feedback (RLHF)\n\n- **Reward Model Training**: Uses the `rlhf.py` and `reward_model.py` scripts to fine-tune models based on human feedback.\n- **Feedback Collection**: Users rate responses via the feedback form (`feedback.html`), and the model retrains with `retrain_model.py`.\n\n### 🔍 Explainability Dashboard\n\nThe `explainability_dashboard.py` script uses **SHAP** values to help users understand why a model made specific predictions.\n\n## 🧪 Testing\n\n- **✅ Unit Tests**: Located in `tests/`, covering API, preprocessing, and training functionalities.\n- **🖥️ End-to-End Tests**: Uses **Cypress** to test UI interactions.\n- **📊 Load Testing**: Implemented with **Locust** (`tests/load_testing/locustfile.py`) to ensure stability under load.\n\n## 🔮 Future Work\n\n- **🔑 User Roles and Permissions**: Adding a role-based access control system.\n- **📉 Advanced Monitoring**: Further enhance Prometheus alerts for anomaly detection.\n- **🚀 Public Demo Deployment**: Deploy a public version on Heroku or AWS for showcasing.\n\n## 🤝 Contributing\n\nContributions are welcome! Please submit pull requests or issues for improvements or new features.\n\n## 📜 License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more information.\n\n## 📬 Contact\n\n- **📧 Email**: [amirsina.torfi@gmail.com](mailto:amirsina.torfi@gmail.com)\n- **🌐 Website**: [Author Website](https://astorfi.github.io)\n\n## Main Collaborators\n\n| [\u003cimg src=\"https://github.com/astorfi.png\" width=\"100px;\"/\u003e](https://github.com/astorfi)\u003cbr/\u003e [\u003csub\u003eAmirsina Torfi\u003c/sub\u003e](https://github.com/astorfi) | [\u003cimg src=\"https://github.com/HRajoliN.png\" width=\"100px;\"/\u003e](https://github.com/HRajoliN)\u003cbr/\u003e [\u003csub\u003eHossein Rajoli\u003c/sub\u003e](https://github.com/HRajoliN) |\n| --- | --- |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastorfi%2Fllm-alignment-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fastorfi%2Fllm-alignment-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastorfi%2Fllm-alignment-project/lists"}