{"id":28676570,"url":"https://github.com/zjunlp/kformer","last_synced_at":"2025-06-13T23:05:13.444Z","repository":{"id":39703808,"uuid":"448163786","full_name":"zjunlp/Kformer","owner":"zjunlp","description":"[NLPCC 2022] Kformer: Knowledge Injection in Transformer Feed-Forward Layers","archived":false,"fork":false,"pushed_at":"2022-10-20T10:57:56.000Z","size":5671,"stargazers_count":33,"open_issues_count":0,"forks_count":5,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-07-16T05:54:39.042Z","etag":null,"topics":["ffn","kformer","knowledge-injection","nlpcc","nlpcc-2022","pytorch","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zjunlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-01-15T02:40:37.000Z","updated_at":"2024-03-28T02:57:21.000Z","dependencies_parsed_at":"2022-08-24T05:01:09.601Z","dependency_job_id":null,"html_url":"https://github.com/zjunlp/Kformer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zjunlp/Kformer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FKformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FKformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FKformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FKformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zjunlp","download_url":"https://codeload.github.com/zjunlp/Kformer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FKformer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259732770,"owners_count":22903087,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ffn","kformer","knowledge-injection","nlpcc","nlpcc-2022","pytorch","transformer"],"created_at":"2025-06-13T23:05:12.695Z","updated_at":"2025-06-13T23:05:13.438Z","avatar_url":"https://github.com/zjunlp.png","language":"Python","readme":"# Kformer\n\nCode for our NLPCC 2022 paper ***[Kformer: Knowlede Injection in Transformer Feed-Forward Layers](https://arxiv.org/pdf/2201.05742.pdf)***\n\nThe project is based on [Fairseq](https://github.com/pytorch/fairseq).\n\n\n\u003cdiv align=center\u003e\u003cimg src=\"model.png\" width=\"40%\" height=\"40%\" /\u003e\u003c/div\u003e\n\n## Requirements\nTo install requirements:\n\n```\ncd fairseq\n./setup.sh\n```\n\n## Download Model\n```\nmkdir models\ncd models\nwget https://dl.fbaipublicfiles.com/fairseq/models/roberta.base.tar.gz\ntar -zxvf roberta.base.tar.gz\n```\n\n## Data\nYou can download the data from [ZJU Cloud](https://pan.zju.edu.cn/share/486968b763a8bd334360556dc0) and put it under the `.\\data\\`.\nThe data we provide here is the question with the retrieved knowledge using bm25.\n## Run the experiments\n\n### Finetuning\n\n#### Social IQA\n\nUse the command below to finetune SocialIQA on Kformer. You can change the layer to inject by editing the arg `--knowledge_layer`.\n `--knowledge_layer` contains two arguments [a,b) denoting the interval of the layer of Roberta. You need to change [this line](https://github.com/zjunlp/Kformer/blob/main/fairseq/fairseq/modules/transformer_sentence_encoder.py#L263) to change the number of the knowledge used for infusion.\n\n```shell\n./fairseq/run_social.sh\n```\n\n #### MedQA\n\nUse the command below to finetune MedQA on Kformer.\n\n```shell\n./fairseq/run_med.sh\n```\n\n### Evaluation\nUse the following command to evalute the finetuned model. Set the `--knowledge_layer` the same as the arg during finetuning.\n\n```\nexport ModelPath = $ModelPath$\nexport DataPath = $DataPath$\npython fairseq/test_social.py --model_path $ModelPath$ --knowledge_layer 9 12 --data_file $DataPath$\n```\n\nChange fairseq/test_social.py to test_med.py to evaluate MedQA.\n\n\n## If you find this repo helpful...\nPlease give us a :star: and cite our paper as\n```bibtex\n@article{Yao2022KformerKI,\n  title={Kformer: Knowledge Injection in Transformer Feed-Forward Layers},\n  author={Yunzhi Yao and Shaohan Huang and Li Dong and Furu Wei and Huajun Chen and Ningyu Zhang},\n  journal={ArXiv},\n  year={2022},\n  volume={abs/2201.05742}\n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjunlp%2Fkformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzjunlp%2Fkformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjunlp%2Fkformer/lists"}