{"id":28653875,"url":"https://github.com/tiger-ai-lab/scholarcopilot","last_synced_at":"2025-06-13T07:07:59.294Z","repository":{"id":267114273,"uuid":"899998746","full_name":"TIGER-AI-Lab/ScholarCopilot","owner":"TIGER-AI-Lab","description":"ScholarCopilot: Training Large Language Models for Academic Writing with Accurate Citations","archived":false,"fork":false,"pushed_at":"2025-04-10T03:03:19.000Z","size":13208,"stargazers_count":119,"open_issues_count":1,"forks_count":15,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-10T03:35:02.646Z","etag":null,"topics":["copilot","lm","rag"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2504.00824","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TIGER-AI-Lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-07T15:36:55.000Z","updated_at":"2025-04-10T03:03:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"13b0819e-1a13-4102-b6c2-23864ec8134f","html_url":"https://github.com/TIGER-AI-Lab/ScholarCopilot","commit_stats":null,"previous_names":["tiger-ai-lab/scholarcopilot"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TIGER-AI-Lab/ScholarCopilot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FScholarCopilot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FScholarCopilot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FScholarCopilot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FScholarCopilot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TIGER-AI-Lab","download_url":"https://codeload.github.com/TIGER-AI-Lab/ScholarCopilot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FScholarCopilot/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259599331,"owners_count":22882357,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["copilot","lm","rag"],"created_at":"2025-06-13T07:07:57.247Z","updated_at":"2025-06-13T07:07:59.269Z","avatar_url":"https://github.com/TIGER-AI-Lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ScholarCopilot\n\n| [**🚀Project Page**](https://tiger-ai-lab.github.io/ScholarCopilot/) | [**📖Paper**](https://arxiv.org/abs/2504.00824) | [**🤗Data**](https://huggingface.co/datasets/TIGER-Lab/ScholarCopilot-Data-v1/) | [**🤗Model**](https://huggingface.co/TIGER-Lab/ScholarCopilot-v1) | [**🤗Demo**](https://huggingface.co/spaces/TIGER-Lab/ScholarCopilot) |\n\nScholar Copilot is an intelligent academic writing assistant that enhances the research writing process through AI-powered text completion and citation suggestions. Built by [TIGER-Lab](https://huggingface.co/TIGER-Lab), it aims to streamline academic writing while maintaining high scholarly standards.\n\n## 🌟 Key Features\n\n### 📝 Smart Text Generation\n- **Next-3-Sentence Suggestions**: Get contextually relevant suggestions for your next three sentences\n- **Full Section Auto-Completion**: Generate complete sections with appropriate academic structure and flow\n- **Context-Aware Writing**: All generations consider your existing text to maintain coherence\n\n### 📚 Intelligent Citation Management\n- **Real-time Citation Suggestions**: Receive relevant paper citations based on your writing context\n- **One-Click Citation Insertion**: Easily select and insert citations in proper academic format\n- **Citation Bibtex Generation**: Automatically generate and export bibtex entries for your citations\n\n## Inference Pipeline Overview\n\nScholar Copilot employs a unified model architecture that seamlessly integrates retrieval and generation through a dynamic switching mechanism. During the generation process, the model autonomously determines appropriate citation points using learned citation patterns. When a citation is deemed necessary, the model temporarily halts generation, utilizes the hidden states of the citation token to retrieve relevant papers from the corpus, inserts the selected references, and then resumes coherent text generation.\n\n\u003cimg width=\"1022\" alt=\"image\" src=\"https://github.com/user-attachments/assets/487890f7-c450-49d6-ac3c-da2d9fb48eba\"\u003e\n\n\n## 🚀 Getting Started\n\nTo set up the ScholarCopilot demo on your own server, follow these simple steps:\n\n1. Clone the repository:\n```bash\ngit clone git@github.com:TIGER-AI-Lab/ScholarCopilot.git\ncd ScholarCopilot/run_demo\n```\n\n2. Set up the environment:\n```bash\npip install -r requirements.txt\n```\n\n3. Download the required model and data:\n```bash\nbash download.sh\n```\n\n4. Launch the demo:\n```\nbash run_demo.sh\n```\n\n## Update new papers to the corpus\nTo update your corpus with the latest papers, follow these steps:\n\n1. Download the most recent arXiv metadata from Kaggle and save it to your chosen ARXIV_META_DATA_PATH\n2. Run the data processing script:\n```bash\ncd utils/\npython process_arxiv_meta_data.py ARXIV_META_DATA_PATH ../data/corpus_data_arxiv_1215.jsonl\n```\n3. Generate the embedding of the corpus:\n```bash\nbash encode_corpus.sh\n```\n\n4. Convert the embedding to HNSW index for efficient search:\n```\npython build_hnsw_index.py --input_dir \u003cembedding dir\u003e --output_dir \u003chnsw index dir\u003e\n```\n\n## 📖 Demo Video\n\n[![Scholar Copilot Demo Video](https://img.youtube.com/vi/QlY7S52sWDA/maxresdefault.jpg)](https://www.youtube.com/watch?v=QlY7S52sWDA)\n\n\n## Train your own model\n\n1. Download the training data:\n```bash\ncd train/\nbash download.sh\n```\n\n2. Configure and run the training script (To reproduce our results, you can use the hyperparameters in the script and 4 machines with 8 GPUs each (32 GPUs in total).)\n```bash\ncd src/\nbash start_train.sh\n```\n\n## Citation\n```\n@article{wang2024scholarcopilot,\n  title={ScholarCopilot: Training Large Language Models for Academic Writing with Accurate Citations},\n  author = {Wang, Yubo and Ma, Xueguang and Nie, Ping and Zeng, Huaye and Lyu, Zhiheng and Zhang, Yuxuan and Schneider, Benjamin and Lu, Yi and Yue, Xiang and Chen, Wenhu},\n  journal={arXiv preprint arXiv:2504.00824},\n  year={2025}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftiger-ai-lab%2Fscholarcopilot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftiger-ai-lab%2Fscholarcopilot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftiger-ai-lab%2Fscholarcopilot/lists"}