{"id":16819106,"url":"https://github.com/davanstrien/haiku-dpo","last_synced_at":"2025-03-22T03:31:31.234Z","repository":{"id":217358362,"uuid":"743119442","full_name":"davanstrien/haiku-dpo","owner":"davanstrien","description":"Using open source LLMs to build synthetic datasets for direct preference optimization ","archived":false,"fork":false,"pushed_at":"2024-02-29T13:44:40.000Z","size":1558,"stargazers_count":58,"open_issues_count":0,"forks_count":8,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-15T00:39:56.207Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davanstrien.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-14T11:50:49.000Z","updated_at":"2025-02-25T07:19:30.000Z","dependencies_parsed_at":"2024-02-29T14:46:15.803Z","dependency_job_id":null,"html_url":"https://github.com/davanstrien/haiku-dpo","commit_stats":null,"previous_names":["davanstrien/haiku-dpo"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davanstrien%2Fhaiku-dpo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davanstrien%2Fhaiku-dpo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davanstrien%2Fhaiku-dpo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davanstrien%2Fhaiku-dpo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davanstrien","download_url":"https://codeload.github.com/davanstrien/haiku-dpo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244902929,"owners_count":20529114,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T10:52:08.151Z","updated_at":"2025-03-22T03:31:29.801Z","avatar_url":"https://github.com/davanstrien.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e🌸 Synthetic Haiku DPO 🌸\u003c/h1\u003e\n\u003cp align=\"center\"\u003e\u003cem\u003e Using open source LLMs to build synthetic datasets for direct preference optimization \u003c/em\u003e\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://cdn-uploads.huggingface.co/production/uploads/60107b385ac3e86b3ea4fc34/nmz7lvu64BytxDvPMm1C5.png\" alt=\"Banner for a dataset card featuring a fusion of digital and traditional Japanese elements. The design includes stylized digital prompts and haiku within text bubbles and on digital screens, set against a backdrop of delicate cherry blossoms and a serene Japanese landscape. The color scheme is dominated by soft pastel pink tones, creating a harmonious blend of modern technology and classical poetry aesthetics.\" width=\"500\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003cem\u003eLines in code retreat,\u003cbr\u003eSynthetic haiku compete,\u003cbr\u003eNature's rhythm, neat.\u003cbr\u003e\u003c/em\u003e\u003c/p\u003e\n\n## 📖 About\n\n[\u003cimg src=\"https://raw.githubusercontent.com/argilla-io/distilabel/main/docs/assets/distilabel-badge-light.png\" alt=\"Built with Distilabel\" width=\"200\" height=\"32\"/\u003e](https://github.com/argilla-io/distilabel)\n\n\u003e Haiku (俳句) is a type of short form poetry that originated in Japan. Traditional Japanese haiku consist of three phrases composed of 17 phonetic units (called on in Japanese, which are similar to syllables) in a 5, 7, 5 pattern  - [https://en.wikipedia.org/wiki/Haiku](https://en.wikipedia.org/wiki/Haiku)\n\n\nA current request for a haiku looks like when using the [TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0) model looks like this:\n\n```\n\u003e\u003e\u003e Write a haiku about moss.\nMosses soft, green and shining,\nGracefully drifting in the breeze,\nA symphony of light and sound.\n```\n\nThis has some of the properties of a traditional haiku (as expressed in the English language), it has three lines, and the themes of nature and the seasons are present. However, the syllable count is off. \n\nWhat happens if we do a little prompt engineering and ask the model to write a haiku with a 5, 7, 5 syllable pattern? \n\n```\n\u003e\u003e\u003e Write a haiku about moss. Follow the traditional 5-7-5 syllable pattern.\nMosses, tiny green eyes, inchworm life,\nGrowing tall, they're hard to see,\nSyllables flow, as the world moves by: haiku of life.\n```\n\nPrompt engineering didn't help use here. What if we use a slightly larger model? The [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B) model is a 7B parameter model, which I've found to perform very well for many tasks. \n\n```\n\u003e\u003e\u003e Write a haiku about moss. Follow the traditional 5-7-5 syllable pattern.\nMoss, soft emerald hue,\nCreeps on stone, whispers secrets -\nNature's silent truth.\n```\n\n\nThis repository contains the code used to generate the [Synthetic Haiku DPO](https://huggingface.co/datasets/davanstrien/haiku_dpo) dataset, a dataset of synthetic haiku generated using [Distilabel](https://github.com/argilla-io/distilabel). \n\nThe goal of this repo is to help the author explore the process of using synthetic data to train a model for direct preference optimization (DPO). \n\n## 📝 The Plan\n\n1. Generate synthetic haiku prompt/response pairs from [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B) \n2. Rank the generated haiku using a syllable counter\n3. Train a model on the generated haiku\n4. Evaluate the model's performance in generating \"technically\" correct haiku\n\nThe next step will involve\n5. annotating haiku generated by the model by aesthetic quality i.e. which haiku do I like the most\n6. seeing the extent to which a DPO dataset can be used to optimize for aesthetic quality\n7. Profit?\n\n## 📚 Notebooks\n\nThese notebooks work through the first few stages of this project; the end result is a DPO dataset. The plan is to also add notebooks for the other stages over time. \n\n1. [01_generate_haiku_prompts.ipynb](01_generate_haiku_prompts.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/davanstrien/haiku-dpo/blob/main/01_generate_haiku_prompts.ipynb) -  This notebook contains code for generating haiku prompts using [Distilabel](https://github.com/argilla-io/distilabel)\n2. [02_generate_haiku.ipynb](02_generate_haiku.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/davanstrien/haiku-dpo/blob/main/02_generate_haiku.ipynb)  -  This notebook generates the actual haiku\n3. [03_rule_based_ranking.ipynb](03_rule_based_ranking.ipynb) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/davanstrien/haiku-dpo/blob/main/03_rule_based_ranking.ipynb) -  This notebook carries out the rule-based ranking of the generated haiku which can then be used to create a preference dataset.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavanstrien%2Fhaiku-dpo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavanstrien%2Fhaiku-dpo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavanstrien%2Fhaiku-dpo/lists"}