{"id":24532543,"url":"https://github.com/alpha-innovator/chimera","last_synced_at":"2025-04-07T06:10:35.461Z","repository":{"id":267408983,"uuid":"895413240","full_name":"Alpha-Innovator/Chimera","owner":"Alpha-Innovator","description":"Improving Generalist Model with Domain-Specific Experts","archived":false,"fork":false,"pushed_at":"2025-01-10T05:01:32.000Z","size":4571,"stargazers_count":85,"open_issues_count":0,"forks_count":11,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-31T05:04:33.224Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Alpha-Innovator.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-28T06:55:20.000Z","updated_at":"2025-03-04T23:49:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"bcc02fc8-c097-4f9f-b3c0-2fc3c1ed3934","html_url":"https://github.com/Alpha-Innovator/Chimera","commit_stats":null,"previous_names":["unimodal4reasoning/chimera","alpha-innovator/chimera"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-Innovator%2FChimera","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-Innovator%2FChimera/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-Innovator%2FChimera/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alpha-Innovator%2FChimera/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Alpha-Innovator","download_url":"https://codeload.github.com/Alpha-Innovator/Chimera/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247601448,"owners_count":20964864,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-22T10:07:47.549Z","updated_at":"2025-04-07T06:10:35.440Z","avatar_url":"https://github.com/Alpha-Innovator.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1\u003eChimera: Improving Generalist Model with\u003cbr\u003eDomain-Specific Experts\u003c/h1\u003e\n\n\n[[ Paper ]](https://arxiv.org/abs/2412.05983) [[ Website ]](https://unimodal4reasoning.github.io/chimera_page/) [[ Dataset🤗 ]]() [[ Models🤗 ]](https://huggingface.co/collections/U4R/chimera-10-6749542e2f0dfa09414232c0) \n\n\u003c/div\u003e\n\n## News :fire:\n- [ ] :fire: :fire: We are developing the Chimera-enhanced version to achieve superior multimodal reasoning by integrating stronger expert models, effective general models, and optimized fusion strategies\n- [ ] Release the training code and data recipe\n- [x] Release the inference code and model checkpoints\n\n## 🛠️ Installation\n\n- Clone this repository:\n\n  ```bash\n  git clone https://github.com/UniModal4Reasoning/Chimera.git\n  ```\n\n- Create a conda virtual environment and activate it:\n\n  ```bash\n  conda create -n chimera python=3.9 -y\n  conda activate chimera\n  ```\n\n- Install dependencies using `requirements.txt`:\n\n  ```bash\n  pip install -r requirements.txt\n  ```\n\n- Install other requirements:\n\n  ```bash\n  cd chimera/\n  pip install --upgrade pip  # enable PEP 660 support\n  pip install -e .\n  ```\n\n### Additional Instructions\n\n- Install `flash-attn==2.3.4`:\n\n  ```bash\n  pip install flash-attn==2.3.4 --no-build-isolation\n  ```\n\n  Alternatively you can compile from source:\n\n  ```bash\n  git clone https://github.com/Dao-AILab/flash-attention.git\n  cd flash-attention\n  git checkout v2.3.4\n  python setup.py install\n  ```\n\n\n## Quick Start\n### Multi-modal reasoning\n```python\nfrom chimera.chimera_infer import Chimera4easyuse\nimport torch\nfrom PIL import Image\n\n# prepare model\n# model_path = \"U4R/Chimera-Reasoner-2B\"\n# model_path = \"U4R/Chimera-Reasoner-4B\"\nmodel_path = \"U4R/Chimera-Reasoner-8B\"\ngeneration_config = dict(max_new_tokens=256, do_sample=False)\nmodel = Chimera4easyuse(model_path, dtype = torch.bfloat16, generation_config= generation_config)\n\n# prepare input\nimage_path = \"path/to/image\"\nuser_prompt = \"\u003cimage\u003e\\nuser prompt\"\ninput_image = Image.open(image_path).convert('RGB')\nresponse = model.get_response(user_prompt, [input_image])\nprint(response)\n```\n\n### Visual content extraction\n```python\nfrom chimera.chimera_infer import Chimera4easyuse\nimport torch\nfrom PIL import Image\n\n# prepare model\nmodel_path = \"U4R/Chimera-Extractor-1B\"\ngeneration_config = dict(max_new_tokens=4096, do_sample=False, no_repeat_ngram_size = 20)\nmodel = Chimera4easyuse(model_path, dtype = torch.float16, generation_config= generation_config)\n\n# prepare input\nimage_path = \"path/to/document\"\nuser_prompt = \"\u003cimage\u003e\\nAs a smart PDF to Markdown conversion tool, please convert the content of the provided PDF into Markdown format.\"\ninput_image = Image.open(image_path).convert('RGB')\nresponse = model.get_response(user_prompt, [input_image])\nprint(response)\n```\n\n\n## License\nChimera is released under the [Apache License 2.0](LICENSE)\n\n## Citation\nIf you find our models / code / papers useful in your research, please consider giving ⭐ and citations 📝, thx :)  \n```bibtex\n@misc{peng2024chimeraimprovinggeneralistmodel,\n      title={Chimera: Improving Generalist Model with Domain-Specific Experts}, \n      author={Tianshuo Peng and Mingsheng Li and Hongbin Zhou and Renqiu Xia and Renrui Zhang and Lei Bai and Song Mao and Bin Wang and Conghui He and Aojun Zhou and Botian Shi and Tao Chen and Bo Zhang and Xiangyu Yue},\n      year={2024},\n      eprint={2412.05983},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2412.05983}, \n}\n```\n\n## Contact Us\nIf you encounter any issues or have questions, please feel free to contact us via bo.zhangzx@gmail.com or zhangbo@pjlab.org.cn.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpha-innovator%2Fchimera","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falpha-innovator%2Fchimera","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpha-innovator%2Fchimera/lists"}