{"id":13569259,"url":"https://github.com/declare-lab/flan-alpaca","last_synced_at":"2025-04-06T16:12:35.284Z","repository":{"id":148130892,"uuid":"617431627","full_name":"declare-lab/flan-alpaca","owner":"declare-lab","description":"This repository contains code for extending the Stanford Alpaca synthetic instruction tuning to existing instruction-tuned models such as Flan-T5.","archived":false,"fork":false,"pushed_at":"2023-07-04T23:49:23.000Z","size":34,"stargazers_count":351,"open_issues_count":8,"forks_count":38,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-30T14:11:11.817Z","etag":null,"topics":["alpaca","flan-t5","language-model","llm","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/declare-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-03-22T11:35:19.000Z","updated_at":"2025-02-28T06:21:51.000Z","dependencies_parsed_at":"2024-01-17T00:55:15.134Z","dependency_job_id":"35044160-53f1-4a98-b1d2-577eef7ca119","html_url":"https://github.com/declare-lab/flan-alpaca","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2Fflan-alpaca","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2Fflan-alpaca/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2Fflan-alpaca/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/declare-lab%2Fflan-alpaca/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/declare-lab","download_url":"https://codeload.github.com/declare-lab/flan-alpaca/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247509234,"owners_count":20950232,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alpaca","flan-t5","language-model","llm","transformers"],"created_at":"2024-08-01T14:00:37.714Z","updated_at":"2025-04-06T16:12:35.262Z","avatar_url":"https://github.com/declare-lab.png","language":"Python","funding_links":[],"categories":["Open Source LLM","LLM-List","Python"],"sub_categories":["Open-LLM"],"readme":"## 🍮 🦙 Flan-Alpaca: Instruction Tuning from Humans and Machines\n\n📣 We developed Flacuna by fine-tuning Vicuna-13B on the Flan collection. Flacuna is better than Vicuna at problem-solving. Access the model here [https://huggingface.co/declare-lab/flacuna-13b-v1.0](https://huggingface.co/declare-lab/flacuna-13b-v1.0).\n\n📣 **FLAN-T5** is also useful in text-to-audio generation. Find our work at [https://github.com/declare-lab/tango](https://github.com/declare-lab/tango) if you are interested.\n\n\nThis repository contains code for extending the [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca)\nsynthetic instruction tuning to existing instruction-tuned models such as [Flan-T5](https://arxiv.org/abs/2210.11416).\nWe have a [live interactive demo](https://huggingface.co/spaces/joaogante/transformers_streaming) thanks\nto [Joao Gante](https://huggingface.co/joaogante)!\nWe are also benchmarking many instruction-tuned models\nat [declare-lab/flan-eval](https://github.com/declare-lab/flan-eval).\nOur pretrained models are fully available on HuggingFace 🤗 :\n\n| Model                                                                            | Parameters | Instruction Data                                                                                                                                   | Training GPUs   |\n|----------------------------------------------------------------------------------|------------|----------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|\n| [Flan-Alpaca-Base](https://huggingface.co/declare-lab/flan-alpaca-base)          | 220M       | [Flan](https://github.com/google-research/FLAN), [Alpaca](https://github.com/tatsu-lab/stanford_alpaca)                                            | 1x A6000        |\n| [Flan-Alpaca-Large](https://huggingface.co/declare-lab/flan-alpaca-large)        | 770M       | [Flan](https://github.com/google-research/FLAN), [Alpaca](https://github.com/tatsu-lab/stanford_alpaca)                                            | 1x A6000        |\n| [Flan-Alpaca-XL](https://huggingface.co/declare-lab/flan-alpaca-xl)              | 3B         | [Flan](https://github.com/google-research/FLAN), [Alpaca](https://github.com/tatsu-lab/stanford_alpaca)                                            | 1x A6000        |\n| [Flan-Alpaca-XXL](https://huggingface.co/declare-lab/flan-alpaca-xxl)            | 11B        | [Flan](https://github.com/google-research/FLAN), [Alpaca](https://github.com/tatsu-lab/stanford_alpaca)                                            | 4x A6000 (FSDP) |\n| [Flan-GPT4All-XL](https://huggingface.co/declare-lab/flan-gpt4all-xl)            | 3B         | [Flan](https://github.com/google-research/FLAN), [GPT4All](https://github.com/nomic-ai/gpt4all)                                                    | 1x A6000        |\n| [Flan-ShareGPT-XL](https://huggingface.co/declare-lab/flan-sharegpt-xl)          | 3B         | [Flan](https://github.com/google-research/FLAN), [ShareGPT](https://github.com/domeccleston/sharegpt)/[Vicuna](https://github.com/lm-sys/FastChat) | 1x A6000        |\n| [Flan-Alpaca-GPT4-XL*](https://huggingface.co/declare-lab/flan-alpaca-gpt4-xl)   | 3B         | [Flan](https://github.com/google-research/FLAN), [GPT4-Alpaca](https://github.com/Instruction-Tuning-with-GPT-4/GPT-4-LLM)                         | 1x A6000        |\n\n*recommended for better performance\n\n### Why?\n\n[Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) represents an exciting new direction\nto approximate the performance of large language models (LLMs) like ChatGPT cheaply and easily.\nConcretely, they leverage an LLM such as GPT-3 to generate instructions as synthetic training data.\nThe synthetic data which covers more than 50k tasks can then be used to finetune a smaller model.\nHowever, the original implementation is less accessible due to licensing constraints of the\nunderlying [LLaMA](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) model.\nFurthermore, users have noted [potential noise](https://github.com/tloen/alpaca-lora/issues/65) in the synthetic\ndataset. Hence, it may be better to explore a fully accessible model that is already trained on high-quality (but\nless diverse) instructions such as [Flan-T5](https://arxiv.org/abs/2210.11416).\n\n### Usage\n\n```\nfrom transformers import pipeline\n\nprompt = \"Write an email about an alpaca that likes flan\"\nmodel = pipeline(model=\"declare-lab/flan-alpaca-gpt4-xl\")\nmodel(prompt, max_length=128, do_sample=True)\n\n# Dear AlpacaFriend,\n# My name is Alpaca and I'm 10 years old.\n# I'm excited to announce that I'm a big fan of flan!\n# We like to eat it as a snack and I believe that it can help with our overall growth.\n# I'd love to hear your feedback on this idea. \n# Have a great day! \n# Best, AL Paca\n```\n\n### Setup\n\nInstall dependencies and download data. We used the cleaned data\nfrom [Alpaca-LoRA](https://github.com/tloen/alpaca-lora.git) for training.\n\n```\nconda create -n paca python=3.8 -y\nconda activate paca\npip install -r requirements.txt\nmkdir -p data\nwget https://github.com/declare-lab/flan-alpaca/releases/download/v0.1.0/alpaca_data.json -O data/alpaca.json\nwget https://github.com/declare-lab/flan-alpaca/releases/download/v0.1.0/alpaca_data_cleaned.json -O data/alpaca_clean.json\nwget https://github.com/declare-lab/flan-alpaca/releases/download/v0.1.0/alpaca_gpt4_data.json -O data/alpaca_gpt4.json\n```\n\nPreprocess [Cleaned Alpaca](https://github.com/tloen/alpaca-lora/blob/main/alpaca_data_cleaned.json) training dataset:\n\n```\npython data_loading.py preprocess_alpaca \\\n--path_in data/alpaca_gpt4.json \\\n--path_out data/train.json\n```\n\nIf you want to use [GPT4All](https://github.com/nomic-ai/gpt4all) data, you can use this command:\n\n```\npython data_loading.py preprocess_gpt4all --path_out data/train.json\n```\n\nIf you want to use [ShareGPT](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) data, you can\nuse this command:\n\n```\nwget https://github.com/declare-lab/flan-alpaca/releases/download/v0.1.0/ShareGPT_unfiltered_cleaned_split.json -O data/sharegpt.json\npython data_loading.py preprocess_sharegpt --path_out data/train.json\n```\n\n### Training\n\nThe following command will finetune the Flan-T5-XL (8hrs on a single A6000 GPU).\n\n```\npython training.py --output_dir outputs/model/xl \\\n--use_compile \\\n--train_epochs 3 \\\n--max_source_length 64 \\\n--max_target_length 512 \\\n--data_path data/train.json \\\n--model_name_or_path \"google/flan-t5-xl\" \\\n--train_batch_size 1 \\\n--gradient_accumulation_steps 64\n```\n\nIf the model does not fit into memory, and you have multiple GPUs, you can\ntry [fully-sharded data parallel](https://engineering.fb.com/2021/07/15/open-source/fsdp/) by replacing `--use_compile`\nwith `--use_fsdp`.\n\n### Inference\n\n```\npython inference.py test_model \\\n--path \"outputs/model/xl/epoch=2-step=2439.ckpt\" \\\n--prompt \"Write an email about an alpaca that likes flan\"\n```\n\n### Exporting to HuggingFace Hub\n\nReplace `declare-lab/flan-alpaca-xl` with your desired HuggingFace repo.\n\n```\nhuggingface-cli login\n\npython inference.py export_to_hub \\\n--path \"outputs/model/xl/epoch=2-step=2439.ckpt\" \\\n--repo declare-lab/flan-alpaca-xl\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeclare-lab%2Fflan-alpaca","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeclare-lab%2Fflan-alpaca","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeclare-lab%2Fflan-alpaca/lists"}