{"id":13754127,"url":"https://github.com/zjukg/KnowPAT","last_synced_at":"2025-05-09T22:31:03.279Z","repository":{"id":206653600,"uuid":"716509477","full_name":"zjukg/KnowPAT","owner":"zjukg","description":"[Paper][ACL 2024 Findings] Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering","archived":false,"fork":false,"pushed_at":"2024-06-10T09:09:45.000Z","size":9466,"stargazers_count":176,"open_issues_count":0,"forks_count":16,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-08-03T09:06:51.571Z","etag":null,"topics":["domain-specific-qa","instruction-tuning","knowledge-graph","large-language-models","preference-alignment","question-answering"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2311.06503","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zjukg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-09T09:39:20.000Z","updated_at":"2024-07-24T03:37:09.000Z","dependencies_parsed_at":"2024-08-03T09:17:17.704Z","dependency_job_id":null,"html_url":"https://github.com/zjukg/KnowPAT","commit_stats":null,"previous_names":["zjukg/knowpat"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjukg%2FKnowPAT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjukg%2FKnowPAT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjukg%2FKnowPAT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjukg%2FKnowPAT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zjukg","download_url":"https://codeload.github.com/zjukg/KnowPAT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224884612,"owners_count":17386121,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["domain-specific-qa","instruction-tuning","knowledge-graph","large-language-models","preference-alignment","question-answering"],"created_at":"2024-08-03T09:01:41.261Z","updated_at":"2024-11-16T06:31:41.287Z","avatar_url":"https://github.com/zjukg.png","language":"Python","funding_links":[],"categories":["A01_文本生成_文本对话"],"sub_categories":["大语言对话模型及数据"],"readme":"# Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering\n![](https://img.shields.io/badge/version-1.0.1-blue)\n[![license](https://img.shields.io/github/license/mashape/apistatus.svg?maxAge=2592000)](https://github.com/zjukg/KnowPAT/main/LICENSE)\n[![Preprint](https://img.shields.io/badge/ACL'24(Findings)-brightgreen)](https://arxiv.org/abs/2311.06503)\n[![Pytorch](https://img.shields.io/badge/PyTorch-%23EE4C2C.svg?e\u0026logo=PyTorch\u0026logoColor=white)](https://pytorch.org/)\n - [Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering](https://arxiv.org/abs/2311.06503)\n\n\u003e For domain-specific application of large language models (LLMs), external knowledge and LLMs should work together to achieve best user experience. LLMs should acquire an ability to **make the right choices** about retrieved external knowledge to meet the human needs. Knowledgeable Preference AlignmenT (KnowPAT) is a new pipeline to align LLMs with human's knowledge preference. KnowPAT incorporates domain knowledge graphs to construct preference set and design new alignment objective to fine-tune the LLMs.\n\n## 🌈 Model Architecture\n![Model_architecture](figure/model.png)\n\n## 💻 Data preparation\nFor reasons of **commercial confidentiality**, the dataset in our paper will **not** be made public. However, we provide examples of the data in `data/` so that you can construct your own preference dataset and try out our alignment method according to the given format.\n\nFor each data instance, you should prepare a question and several answers, and the human preference score of each answer. A higher score means a better and preferred answer. Note that the answers should be sorted in the **score descending** order, which means the better answer and its score should be in the front.\n\n- **News**: 🎉🎉🥰🎉🎉 Our paper has been accepted by Findings of ACL 2024.\n- **News**: We now add a new Chinese QA datasets [RJUA](https://github.com/alipay/RJU_Ant_QA) and it is open-sourced. You can try to conduct experiments on this dataset.\n\n\n## 🔬 Dependencies\nOur code is developed based on [RRHF](https://github.com/GanjinZero/RRHF). Please build the Python environment following the instruction like RRHF.\n\nTo set up, you can use the following command lines to set up python3.8 and pytorch requirements:\n\n```shell\nconda create -n alignment python=3.8\npip install torch==1.13.0+cu116 torchvision==0.14.0+cu116 torchaudio==0.13.0 --extra-index-url https://download.pytorch.org/whl/cu116\n\n```\nInstall other packages:\n\n```shell\npip install -r requirements.txt\n```\n\n## 📕 Training \u0026 Test\n\n- run KnowPAT training on RJUA datasets\n\n```shell\n\nexport MODEL_PATH=\"YOUR LLM PATH\"\nexport SAVE_PATH=\"YOUR SAVE PATH\"\nexport DATA_PATH=\"data/RJUA_train.json\"\nexport WANDB_DISABLED=true\nwandb offline\n\nCUDA_VISIBLE_DEVICES=1 nohup python train.py \\\n    --model_name_or_path $MODEL_PATH \\\n    --data_path $DATA_PATH \\\n    --bf16 True \\\n    --output_dir $SAVE_PATH \\\n    --num_train_epochs 3 \\\n    --per_device_train_batch_size 1 \\\n    --per_device_eval_batch_size 1 \\\n    --gradient_accumulation_steps 8 \\\n    --evaluation_strategy \"no\" \\\n    --save_strategy \"steps\" \\\n    --save_steps 300 \\\n    --save_total_limit 40 \\\n    --learning_rate 1e-4 \\\n    --weight_decay 0. \\\n    --warmup_ratio 0.03 \\\n    --lr_scheduler_type \"cosine\" \\\n    --logging_steps 1 \\\n    --model_max_length 512 --rrhf_weight 0.01 \u003e log_rjua.txt \u0026\n\n```\nYou may need to fill your model/save/data path before running. The model path should be a llama-architecture LLM.\n\n- run inference\n```shell\nCUDA_VISIBLE_DEVICES=0 python inference.py\n```\n\n\n## 🤝 Cite:\nPlease consider citing this paper if you use the code from our work.\nThanks a lot :)\n\n```bigquery\n\n@article{DBLP:journals/corr/abs-2311-06503,\n  author       = {Yichi Zhang and\n                  Zhuo Chen and\n                  Yin Fang and\n                  Lei Cheng and\n                  Yanxi Lu and\n                  Fangming Li and\n                  Wen Zhang and\n                  Huajun Chen},\n  title        = {Knowledgeable Preference Alignment for LLMs in Domain-specific Question\n                  Answering},\n  journal      = {CoRR},\n  volume       = {abs/2311.06503},\n  year         = {2023}\n}\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjukg%2FKnowPAT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzjukg%2FKnowPAT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjukg%2FKnowPAT/lists"}