{"id":50528060,"url":"https://github.com/sakanaai/doc-to-lora","last_synced_at":"2026-06-03T10:01:09.667Z","repository":{"id":358093688,"uuid":"1155266070","full_name":"SakanaAI/doc-to-lora","owner":"SakanaAI","description":"Hypernetworks that update LLMs to remember factual information","archived":false,"fork":false,"pushed_at":"2026-05-25T07:12:27.000Z","size":6830,"stargazers_count":727,"open_issues_count":0,"forks_count":89,"subscribers_count":6,"default_branch":"main","last_synced_at":"2026-05-25T09:15:22.024Z","etag":null,"topics":["ai","ai-agent","hypernetworks","llm","llm-agent","lora","machine-learning","memory"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2602.15902","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SakanaAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-11T10:11:52.000Z","updated_at":"2026-05-25T07:12:32.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/SakanaAI/doc-to-lora","commit_stats":null,"previous_names":["sakanaai/doc-to-lora"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SakanaAI/doc-to-lora","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SakanaAI%2Fdoc-to-lora","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SakanaAI%2Fdoc-to-lora/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SakanaAI%2Fdoc-to-lora/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SakanaAI%2Fdoc-to-lora/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SakanaAI","download_url":"https://codeload.github.com/SakanaAI/doc-to-lora/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SakanaAI%2Fdoc-to-lora/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33858578,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-03T02:00:06.370Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-agent","hypernetworks","llm","llm-agent","lora","machine-learning","memory"],"created_at":"2026-06-03T10:01:08.216Z","updated_at":"2026-06-03T10:01:09.656Z","avatar_url":"https://github.com/SakanaAI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n    \u003ch1\u003eDoc-to-LoRA (D2L): Learning to Instantly Internalize Contexts\u003c/h1\u003e\n    :sparkles:\u003ca href=\"https://pub.sakana.ai/doc-to-lora/\"\u003eInteractive Web\u003c/a\u003e |\n    :newspaper:\u003ca href=\"https://x.com/SakanaAILabs\"\u003eX\u003c/a\u003e |\n    :scroll:\u003ca href=\"https://arxiv.org/abs/2602.15902\"\u003ePaper\u003c/a\u003e |\n    :hugs:\u003ca href=\"https://huggingface.co/SakanaAI\"\u003eHugging Face\u003c/a\u003e |\n    :octocat:\u003ca href=\"https://github.com/SakanaAI/doc-to-lora\"\u003eGitHub\u003c/a\u003e\n\u003cbr\u003eA reference implementation of Doc-to-LoRA (D2L).\u003cbr\u003e\n\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\n    \u003cimg height=\"300px\" src=\"assets/overview_animation.gif\" /\u003e\n\u003c/div\u003e\n\n---\n\n## 🛠️ Installation\n```\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n./install.sh\n```\n\n## 🤗 Pre-Trained Models\n```\nuv run huggingface-cli login\nuv run huggingface-cli download SakanaAI/doc-to-lora --local-dir trained_d2l --include \"*/\"\n```\n\n## 🚀 Python API Usage\n```python\n# caveat: this interface only supports non-batched inputs\n# for batched inference please see `src/ctx_to_lora/modeling/hypernet.py`\nimport torch\n\nfrom ctx_to_lora.model_loading import get_tokenizer\nfrom ctx_to_lora.modeling.hypernet import ModulatedPretrainedModel\n\n# model loading\ncheckpoint_path = \"trained_d2l/gemma_demo/checkpoint-80000/pytorch_model.bin\"\nstate_dict = torch.load(checkpoint_path, weights_only=False)\nmodel = ModulatedPretrainedModel.from_state_dict(\n    state_dict, train=False, use_sequence_packing=False\n)\nmodel.reset()\ntokenizer = get_tokenizer(model.base_model.name_or_path)\n\n# prepare data\ndoc = open(\"data/sakana_wiki.txt\", \"r\").read()\nchat = [{\"role\": \"user\", \"content\": \"Tell me about Sakana AI.\"}]\nchat_ids = tokenizer.apply_chat_template(\n    chat,\n    add_special_tokens=False,\n    return_attention_mask=False,\n    add_generation_prompt=True,\n    return_tensors=\"pt\",\n).to(model.device)\n\n\n# calls after internalization will be influenced by internalized info\nmodel.internalize(doc)\n\noutputs = model.generate(input_ids=chat_ids, max_new_tokens=512)\nprint(tokenizer.decode(outputs[0]))\n\n\n# remove internalized info\n# model.reset()\n\n# without internalized info, the model will halucinate\n# outputs = model.generate(input_ids=chat_ids, max_new_tokens=512)\n# print(tokenizer.decode(outputs[0]))\n```\n\n### 🎮 Interactive Demo\n```bash\nuv run demo/app.py\n```\n\u003cdiv align=\"center\"\u003e\n    \u003ch3\u003eVideo Demo\u003c/h3\u003e\n    \u003cvideo src=\"https://github.com/user-attachments/assets/16781365-5ec2-4c1c-b4f4-aeeebe3c2be5\" controls autoplay muted playsinline preload=\"metadata\" width=\"900\"\u003e\u003c/video\u003e\n\u003c/div\u003e\n\n### 🧪 Experimental Scripts\nTo run any of the following scripts, use `uv run $PATH_TO_SCRIPT` from the root of this project.\n\n\n| Experiment                           | Data prep                             | Training                      | Evaluation                   | Notes                                                                                                                               |\n| ------------------------------------ | ------------------------------------- | ----------------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |\n| [Main experiment](scripts/main_exp/) | `scripts/main_exp/0-download_data.sh` | `scripts/main_exp/1-train.sh` | `scripts/main_exp/eval/*.sh` | Downloading data is fastest; regenerate only if you need fresh synthetic data. Evaluation scripts reproduce the main paper metrics. |\n| [NIAH](scripts/niah/)                | `scripts/niah/0-gen_data.sh`          | `scripts/niah/1-train.sh`     | `scripts/niah/2-eval.sh`     | Run the scripts in order; data generation only needs to happen once                                                                 |\n\n\n### 🔬 Self-Generated Data Viewer\nAfter downloading/generating the data, we can see samples of the data using this script.\n```bash\nuv run webui/self_gen_viewer.py\n```\nSee more info at [webui/SELF_GEN_VIEWER.md](webui/SELF_GEN_VIEWER.md).\n\n### 📚 Citation\n```bibtex\n@inproceedings{charakorn2026doctolora,\n  title       ={Doc-to-Lo{RA}: Learning to Instantly Internalize Contexts},\n  author      ={Rujikorn Charakorn and Edoardo Cetin and Shinnosuke Uesaka and Robert Tjarko Lange},\n  booktitle   ={Forty-third International Conference on Machine Learning},\n  year        ={2026},\n  url         ={https://openreview.net/forum?id=iW1oBBO72S}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsakanaai%2Fdoc-to-lora","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsakanaai%2Fdoc-to-lora","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsakanaai%2Fdoc-to-lora/lists"}