{"id":20173003,"url":"https://github.com/humansignal/rlhf","last_synced_at":"2025-04-09T18:18:01.505Z","repository":{"id":166166953,"uuid":"614860225","full_name":"HumanSignal/RLHF","owner":"HumanSignal","description":"Collection of links, tutorials and best practices of how to collect the data and build end-to-end RLHF system to finetune Generative AI models","archived":false,"fork":false,"pushed_at":"2023-07-24T17:54:26.000Z","size":3230,"stargazers_count":216,"open_issues_count":4,"forks_count":44,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-09T18:17:57.025Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HumanSignal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-16T13:19:57.000Z","updated_at":"2025-04-05T19:35:37.000Z","dependencies_parsed_at":"2024-11-14T01:33:11.251Z","dependency_job_id":"ba4fae69-4faf-45d0-b490-2f1d11763aae","html_url":"https://github.com/HumanSignal/RLHF","commit_stats":null,"previous_names":["humansignal/rlhf"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanSignal%2FRLHF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanSignal%2FRLHF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanSignal%2FRLHF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HumanSignal%2FRLHF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HumanSignal","download_url":"https://codeload.github.com/HumanSignal/RLHF/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248085328,"owners_count":21045139,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T01:33:06.952Z","updated_at":"2025-04-09T18:18:01.482Z","avatar_url":"https://github.com/HumanSignal.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Aligning ML Models with Human Feedback\n\n- [What this repository contains:](#what-this-repository-contains)\n  - [This repo may be of interest to you if:](#this-repo-may-be-of-interest-to-you-if)\n  - [If you have any questions about this repo, or need a hand:](#if-you-have-any-questions-about-this-repo-or-need-a-hand)\n- [Establishing Supervised Model Baseline](#establishing-supervised-model-baseline)\n- [Gathering and Incorporating Human Feedback](#gathering-and-incorporating-human-feedback)\n- [Training and Assessing the Final Model with Reinforcement Learning](#training-and-assessing-the-final-model-with-reinforcement-learning)\n\n![Stages of model alignment](./images/RLHF.png)\n\n## What this repository contains:\nThis repository contains a collection of tutorials, best practices, and references for developers, data scientists, and machine learning professionals of all skill levels. \n\n### This repo may be of interest to you if: \n- You've attended the PyData Berlin Talk by [Nikolai Liubimov](https://www.linkedin.com/in/liubimov/) and [Erin Mikail Staples](https://www.linkedin.com/in/erinmikail/)\n- You're interested in data-centric model development\n- You're using [Label Studio](https://labelstud.io) to improve your ML process and models.\n\n### If you have any questions about this repo, or need a hand:\n- This repo is maintained by the [Label Studio](https://labelstud.io) team. \n- Reach out to the [Label Studio Community Team](mailto:community@labelstud.io) or open an issue with any questions.\n- Join the [Label Studio Community Slack](https://slack.labelstud.io/?source=github-RLHF)\n- Check out more resources around Human in the Loop / RHLF in our [Awesome List](https://github.com/heartexlabs/awesome-human-in-the-loop/tree/master)\n\n\n## Establishing Supervised Model Baseline\nIn this step, we collect labeled text data to train an initial Large Language Model (LLM), focusing on task-specific performance improvements. This stage involves gathering instructions and responses to adapt the base model to a broad range of tasks, enhancing its ability to generate accurate and contextually relevant responses.\nTypically this step involves:\n- Selecting baseline Foundational Model (FM) that can perform fairly well on general tasks (like [GPT2](https://huggingface.co/gpt2) or [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6b))\n- Generate dataset of pairs prompt input followed by response. You can manually label dataset or generate the data like it is [provided in the example](https://github.com/tatsu-lab/stanford_alpaca)\n- Peform supervised model finetuning.\n\n## Gathering and Incorporating Human Feedback\nThis stage involves collecting comparison data to establish human preferences for the responses generated by the supervised model. By ranking multiple responses based on quality, we can train a reward model that effectively captures human preferences. This reward model plays a crucial role in reinforcement learning, optimizing the performance of the fine-tuned foundational model.\n\n[Gathering Human Feedback Tutorial](tutorials/RLHF_with_Custom_Datasets.ipynb) - This Jupyter Notebook tutorial will guide you through the process of collecting comparison data, establishing human preferences, and incorporating this feedback into the reward model training.\n\n## Training and Assessing the Final Model with Reinforcement Learning\nThe training stage is a challenging process, and the final model assessment is a critical component in evaluating your model's quality. It is essential to determine whether the model adheres to the provided instructions, avoids biases, and maintains a high standard of performance.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhumansignal%2Frlhf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhumansignal%2Frlhf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhumansignal%2Frlhf/lists"}