{"id":28676576,"url":"https://github.com/zjunlp/pitfallsknowledgeediting","last_synced_at":"2025-06-13T23:05:14.065Z","repository":{"id":197934646,"uuid":"698033057","full_name":"zjunlp/PitfallsKnowledgeEditing","owner":"zjunlp","description":"[ICLR 2024] Unveiling the Pitfalls of Knowledge Editing for Large Language Models","archived":false,"fork":false,"pushed_at":"2024-06-13T03:58:42.000Z","size":9506,"stargazers_count":19,"open_issues_count":0,"forks_count":2,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-06-13T07:35:55.794Z","etag":null,"topics":["ai","artificial-intelligence","easyedit","knowledge-editing","knowlm","large-language-models","model-editing","natural-language-processing","nlp"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zjunlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-29T02:07:54.000Z","updated_at":"2024-06-13T03:58:45.000Z","dependencies_parsed_at":"2024-02-26T07:47:55.438Z","dependency_job_id":null,"html_url":"https://github.com/zjunlp/PitfallsKnowledgeEditing","commit_stats":null,"previous_names":["zjunlp/pitfallsknowledgeediting"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zjunlp/PitfallsKnowledgeEditing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FPitfallsKnowledgeEditing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FPitfallsKnowledgeEditing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FPitfallsKnowledgeEditing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FPitfallsKnowledgeEditing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zjunlp","download_url":"https://codeload.github.com/zjunlp/PitfallsKnowledgeEditing/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FPitfallsKnowledgeEditing/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259732770,"owners_count":22903087,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","artificial-intelligence","easyedit","knowledge-editing","knowlm","large-language-models","model-editing","natural-language-processing","nlp"],"created_at":"2025-06-13T23:05:13.011Z","updated_at":"2025-06-13T23:05:14.050Z","avatar_url":"https://github.com/zjunlp.png","language":"Python","readme":"\u003ch1 align='center'\u003ePitfalls of Editing LLMs\u003c/h1\u003e\n\nCode for the ICLR2024 paper: \"[Unveiling the Pitfalls of Knowledge Editing for Large Language Models](https://arxiv.org/abs/2310.02129)\".\n\n\n---\n\n\u003cdiv align=center\u003e\u003cimg src=\"img/main.png\" width=\"80%\" height=\"80%\" alt=\"main\"/\u003e\u003c/div\u003e\n\n[Knowledge Editing](https://github.com/zjunlp/KnowledgeEditingPapers) provides an efficient way to change the behavior of LLMs without resorting to an exhaustive retraining or continuous training procedure. As the number of edits increases, the model might manifest **Knowledge Conflict** when dealing with inputs involved with multiple consecutive edits. Meanwhile, each edit could potentially lead to ruptures in knowledge links within the model, resulting in **Knowledge Distortion**.\n\n---\n\n![overview](img/overview.png)\n\n**Overview:** (a) Through **Reverse Edit** and **Composite Edit**, we can observe that previous knowledge editing approaches may trigger Knowledge Conflict, leading to failures of knowledge editing; (b) Through **Round-Edit**, we notice that previous knowledge editing approaches may lead to Knowledge Distortion, and the underlying knowledge structure within LLMs can be disrupted.\n\n## Table of Contents\n\n- [Installation](#installation)\n- [Dataset Format](#dataset-format)\n- [Evaluation](#evaluation)\n- [Summerization](#summerization)\n- [Experimental Results](#experimental-results)\n- [How to Cite](#how-to-cite)\n- [Acknowledgements](#acknowledgements)\n\n## Installation\n\nPlease use Python 3.9+ to get started, install `conda` and run:\n```bash\nconda create -n EditLLMs python=3.9.7\npip install -r requirements.txt\n```\n\n**Note:** We recommend `conda` for managing Python, CUDA, and PyTorch=1.12.1.\n\n## Dataset Format\n\n### ConflictEdit\n\nEach dataset split in this part contains 2500 data, except 2000 data in `./data/GPT2-XL/composite_edit.json`. Take `reverse_edit.json` for example:\n\n```json\n{\n    \"rule\": \"Logical Rule\", \n    \"triples\": [\n        {\n            \"relation\": {\n                \"id\": \"ID in WikiData\",\n                \"prompt\": \"Prompt of Relation\",\n                \"query\": \"Prompt of Relation in the query format\",\n                \"label\": \"Relation Description\"\n            },\n            \"subject\": {\n                \"id\": \"ID in WikiData\",\n                \"label\": \"Entity Description\"\n            },\n            \"object\": {\n                \"id\": \"ID in WikiData\",\n                \"label\": \"Entity Description\"\n            }\n        },                  // Triple 1\n        \"... Triple 2\"\n    ],\n    \"prerequisites\": [],    // Tied Fact Dependency\n    \"type\": \"reverse\",      // Edit Type\n    \"edits\": [\n        {\n            \"relation\": \"Same as above\",\n            \"subject\": \"Same as above\",\n            \"object\": \"Object to be edit\",\n            \"new_object\": \"Target Object of editing\"\n        },                  // Edit 1\n        \"... Edit 2\"\n    ]\n}\n```\n\n### RoundEdit\n\nEach dataset split in this part contains 2500 data.Take `easy.json` for example:\n\n```json\n{\n    \"type\": \"1-N@RelationID\",   // N means 1-n relation\n    \"edit\": {\n        \"relation\": \"Same as above\",\n        \"subject\": \"Same as above\",\n        \"new_object\": \"Intermediate object in Round-Edit\",\n        \"object\": \"Target object in Round-Edit\"\n    },\n    \"true_objects\": [\n        {\n            \"id\": \"ID in WikiData\",\n            \"label\": \"Entity Description\"\n        },                      // True object 1\n        \"... True objects\"\n    ]\n}\n```\n\n## Evaluation\n\n### Knowledge Conflict\n\nTo evaluate Knowledge Conflict, simply utilize the scripts as:\n\n```shell\nbash run_conflictedit.sh\n```\n\nThe dataset split can be changed by modified the `mode` in [`run_conflictedit.sh`](scripts/run_conflictedit.sh) and also the model type, hyperparameters and editing methods. The experimental results are written in `./{ModelName}/conflict_results/`\n\n### Knowledge Distortion\n\nTo evaluate Knowledge Conflict, please follow the **Steps** as:\n\n- **Step 1:** Obtain the results on the original model by running:\n\n```shell\nbash run_model.sh\n```\n\n- **Step 2:** Obtain the main results as:\n\n```shell\nbash run_roundedit.sh\n```\n\n- **Step 3:** Obtain the **Multi-Label Edit (MLE)** results as:\n\n```shell\nbash run_MLE.sh\n```\n\nThe dataset split can be changed by modified the `mode` in each script and also the model type, hyperparameters and editing methods. The experimental results are written in `./{ModelName}/round_results/`\n\n\n**Note:** We train MEND on our datasets and the checkpoints are available in [Google Drive](https://drive.google.com/drive/folders/1D9kQDY6DkBAJPM85nv1ancwqmaBvVkvY?usp=sharing).\n\n## Summerization\n\nTo summarize the results, you can use [`experiments/summarize.py`](experiments/summarize.py):\n\n```bash\npython3 -m experiments.summarize --res_dir=GPT-J\n```\n\n## Experimental Results\n\n### Knowledge Conflict\n\n![KnowledgeConflict](img/knowledge_conflict_results.png)\n\n### Knowledge Distortion\n\n\u003cdiv align=center\u003e\u003cimg src=\"img/knowledge_distortion_results.png\" width=\"85%\" height=\"85%\" alt=\"KnowledgeDistortion\"/\u003e\u003c/div\u003e\n\n## How to Cite\n\n```bibtex\n@article{li2023unveiling,\n  title={Unveiling the pitfalls of knowledge editing for large language models},\n  author={Li, Zhoubo and Zhang, Ningyu and Yao, Yunzhi and Wang, Mengru and Chen, Xi and Chen, Huajun},\n  journal={arXiv preprint arXiv:2310.02129},\n  year={2023}\n}\n```\n\n## Acknowledgements\n\nWe appreciate [OpenAI GPT4 Service](https://openai.com/gpt-4), [MEMIT](https://github.com/kmeng01/memit), [EasyEdit](https://github.com/zjunlp/EasyEdit) and many other related works for their open-source contributions.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjunlp%2Fpitfallsknowledgeediting","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzjunlp%2Fpitfallsknowledgeediting","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjunlp%2Fpitfallsknowledgeediting/lists"}