{"id":13456538,"url":"https://github.com/Qihoo360/360zhinao","last_synced_at":"2025-03-24T11:30:40.163Z","repository":{"id":232913557,"uuid":"779225422","full_name":"Qihoo360/360zhinao","owner":"Qihoo360","description":"360zhinao","archived":false,"fork":false,"pushed_at":"2024-09-11T02:03:09.000Z","size":8461,"stargazers_count":275,"open_issues_count":9,"forks_count":23,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-09-11T07:08:04.621Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Qihoo360.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-29T10:26:24.000Z","updated_at":"2024-09-11T02:03:12.000Z","dependencies_parsed_at":"2024-07-31T08:24:43.484Z","dependency_job_id":null,"html_url":"https://github.com/Qihoo360/360zhinao","commit_stats":null,"previous_names":["qihoo360/360zhinao"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qihoo360%2F360zhinao","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qihoo360%2F360zhinao/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qihoo360%2F360zhinao/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qihoo360%2F360zhinao/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Qihoo360","download_url":"https://codeload.github.com/Qihoo360/360zhinao/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221962409,"owners_count":16908336,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T08:01:23.725Z","updated_at":"2024-10-29T00:30:38.494Z","avatar_url":"https://github.com/Qihoo360.png","language":"Python","funding_links":[],"categories":["Python","A01_文本生成_文本对话"],"sub_categories":["大语言对话模型及数据"],"readme":"\u003cp align=\"left\"\u003e\n    \u003ca href=\"./README_CN.md\"\u003e中文\u003c/a\u003e ｜ \u0026nbsp English\u003c/a\u003e\u0026nbsp\n\u003c/p\u003e\n\u003cbr\u003e\n\n\u003cdiv align=\"center\"\u003e\n\u003ch1\u003e\n  360Zhinao (360智脑)\n\u003c/h1\u003e\n\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\n    🤗 \u003ca href=\"https://huggingface.co/qihoo360\"\u003eHuggingFace\u003c/a\u003e\u0026nbsp\u0026nbsp | \u0026nbsp\u0026nbsp\n    🤖 \u003ca href=\"https://www.modelscope.cn/profile/qihoo360\"\u003eModelScope\u003c/a\u003e\u0026nbsp\u0026nbsp ｜ \u0026nbsp\u0026nbsp\n    💬 \u003ca href=\"./assets/WeChat.png\"\u003eWeChat (微信)\u003c/a\u003e\u0026nbsp\u0026nbsp ｜ \u0026nbsp\u0026nbsp\n    📑 \u003ca href=\"https://arxiv.org/abs/2405.13386\"\u003eTechnical Report\u003c/a\u003e\u0026nbsp\u0026nbsp\n\u003c/div\u003e\n\u003cbr\u003e\n\u003cp align=\"center\"\u003e\n Feel free to visit 360Zhinao's official website\u003ca href=\"https://ai.360.com\"\u003e https://ai.360.com\u003c/a\u003e for more experience.\n\u003c/p\u003e\n\n\u003cbr\u003e\n\n# Introduction\n 🎉🎉🎉 We released the 360Zhinao model series:\n - **360Zhinao-7B-Base**\n - **360Zhinao-7B-Chat-4K**\n - **360Zhinao-7B-Chat-32K**\n - **360Zhinao-7B-Chat-360K**\n - **360Zhinao-search**\n - **360Zhinao-1.8B-Reranking**\n\nNotable features of our 360Zhinao models are:\n\n- **Base Model:** Leveraging a high-quality corpus of 3.4 trillion tokens consisting of mainly Chinese, English and code, we achieved competitive performance on relevant benchmarks against other 7B models.\n- **Chat Models:** Powerful chat capabilities and three context lengths of 4K, 32K and 360K. 360K (around 500k Chinese characters) is the longest context length among Chinese open-sourced models upon release (Apr. 11, 2024).\n\n\u003cbr\u003e\n\n# News and Updates\n- [2024.05.23] We released two models, 360Zhinao-search and 360Zhinao-1.8B-Reranking, which ranked first respectively in the Retrieval and Reranking tasks of [C-MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) .\n- [2024.05.20] We extended llama3 and released **llama3-8B-360Zhinao-360k-Instruct**\u003ca href=\"https://huggingface.co/qihoo360/llama3-8B-360Zhinao-360k-Instruct\"\u003e🤗\u003c/a\u003e Details [here](./360k).\n- [2024.04.12] We released **360Zhinao-7B** v1.0, including the base model and three chat models with context lengths 4K, 32K and 360K. \nTechnical report is [here](./360ZHINAO_TECHNICAL_REPORT.pdf) and on [arXiv](https://arxiv.org/abs/2405.13386).\n\n\u003cbr\u003e\n\n# Table of contents\n- [Download URL](#Download-URL)\n- [Model Evaluation](#Model-Evaluation)\n- [Quickstart](#Quickstart)\n- [Model Inference](#Model-Inference)\n- [Model Finetune](#Model-Finetune)\n- [License](#License)\n\n\u003cbr\u003e\n\n# Download URL\n\n| Size | Model | BF16 | Int4|\n|-|-|-|-|\n| 7B | 360Zhinao-7B-Base | \u003ca href=\"https://www.modelscope.cn/models/qihoo360/360Zhinao-7B-Base/summary\"\u003e🤖\u003c/a\u003e  \u003ca href=\"https://huggingface.co/qihoo360/360Zhinao-7B-Base\"\u003e🤗\u003c/a\u003e |  |\n| 7B | 360Zhinao-7B-Chat-4K | \u003ca href=\"https://www.modelscope.cn/models/qihoo360/360Zhinao-7B-Chat-4K/summary\"\u003e🤖\u003c/a\u003e  \u003ca href=\"https://huggingface.co/qihoo360/360Zhinao-7B-Chat-4K\"\u003e🤗\u003c/a\u003e | \u003ca href=\"https://www.modelscope.cn/models/qihoo360/360Zhinao-7B-Chat-4K-Int4/summary\"\u003e🤖\u003c/a\u003e  \u003ca href=\"https://huggingface.co/qihoo360/360Zhinao-7B-Chat-4K-Int4\"\u003e🤗\u003c/a\u003e |\n| 7B | 360Zhinao-7B-Chat-32K | \u003ca href=\"https://www.modelscope.cn/models/qihoo360/360Zhinao-7B-Chat-32K/summary\"\u003e🤖\u003c/a\u003e  \u003ca href=\"https://huggingface.co/qihoo360/360Zhinao-7B-Chat-32K\"\u003e🤗\u003c/a\u003e | \u003ca href=\"https://www.modelscope.cn/models/qihoo360/360Zhinao-7B-Chat-32K-Int4/summary\"\u003e🤖\u003c/a\u003e  \u003ca href=\"https://huggingface.co/qihoo360/360Zhinao-7B-Chat-32K-Int4\"\u003e🤗\u003c/a\u003e |\n| 7B | 360Zhinao-7B-Chat-360K | \u003ca href=\"https://www.modelscope.cn/models/qihoo360/360Zhinao-7B-Chat-360K/summary\"\u003e🤖\u003c/a\u003e  \u003ca href=\"https://huggingface.co/qihoo360/360Zhinao-7B-Chat-360K\"\u003e🤗\u003c/a\u003e | \u003ca href=\"https://www.modelscope.cn/models/qihoo360/360Zhinao-7B-Chat-360K-Int4/summary\"\u003e🤖\u003c/a\u003e  \u003ca href=\"https://huggingface.co/qihoo360/360Zhinao-7B-Chat-360K-Int4\"\u003e🤗\u003c/a\u003e |\n| 325M | 360Zhinao-search | \u003ca href=\"https://huggingface.co/qihoo360/360Zhinao-search\"\u003e🤗\u003c/a\u003e |  |\n| 1.8B | 360Zhinao-1.8B-Reranking | \u003ca href=\"https://huggingface.co/qihoo360/360Zhinao-1.8B-Reranking\"\u003e🤗\u003c/a\u003e |  |\n\n\u003cbr\u003e\n\n# Model Evaluation\n## Base Model\nWe evaluate our model on [OpenCompass](https://opencompass.org.cn/home), more specifically on C-Eval, AGIEval, MMLU, CMMLU, HellaSwag, MATH, GSM8K, HumanEval, MBPP, BBH and LAMBADA. \nThese benchmarks test the model on\nnatural language understanding, knowledge, mathematics, code generation and logical reasoning, etc.\n\nResults are listed as follows and could be viewed or reproduced on [OpenCompass leaderboard](https://rank.opencompass.org.cn/leaderboard-llm).\n\n| \u003cdiv style=\"width: 100pt\"\u003eModel\u003c/div\u003e | AVG   | CEval | AGIEval | MMLU | CMMLU | HellaSwag | MATH | GSM8K | HumanEval | MBPP | BBH | LAMBADA |\n|:----------------------|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| Baichuan2-7B          | 41.49     | 56.3      | 34.6      | 54.7      | 57        | 67        | 5.4       | 24.6      | 17.7      | 24        | 41.8      | 73.3      |\n| Baichuan-7B           | 31.94     | 44.7      | 24.6      | 41.5      | 44.6      | 68.4      | 2.5       | 9.6       | 9.1       | 6.4       | 32.8      | 67.1      |\n| ChatGLM3-6B           | **58.67** | 67        | 47.4      | 62.8      | 66.5      | 76.5      | 19.2      | 61        | 44.5      | **57.2**  | **66.2**  | 77.1      |\n| DeepSeek-7B           | 39.8      | 45        | 24        | 49.3      | 46.8      | 73.4      | 4.2       | 18.3      | 25        | 36.4      | 42.8      | 72.6      |\n| InternLM2-7B          | 58.01     | 65.7      | 50.2      | 65.5      | 66.2      | 79.6      | 19.9      | **70.6**  | 41.5      | 42.4      | 64.4      | 72.1      |\n| InternLM-7B           | 39.33     | 53.4      | 36.9      | 51        | 51.8      | 70.6      | 6.3       | 31.2      | 13.4      | 14        | 37        | 67        |\n| LLaMA-2-7B            | 33.27     | 32.5      | 21.8      | 46.8      | 31.8      | 74        | 3.3       | 16.7      | 12.8      | 14.8      | 38.2      | 73.3      |\n| LLaMA-7B              | 30.35     | 27.3      | 20.6      | 35.6      | 26.8      | 74.3      | 2.9       | 10        | 12.8      | 16.8      | 33.5      | 73.3      |\n| Mistral-7B-v0.1       | 47.67     | 47.4      | 32.8      | 64.1      | 44.7      | 78.9      | 11.3      | 47.5      | 27.4      | 38.6      | 56.7      | 75        |\n| MPT-7B                | 30.06     | 23.5      | 21.3      | 27.5      | 25.9      | 75        | 2.9       | 9.1       | 17.1      | 22.8      | 35.6      | 70        |\n| Qwen1.5-7B            | 55.12     | 73.57     | **50.8**  | 62.15     | 71.84     | 72.62     | **20.36** | 54.36     | **53.05** | 36.8      | 40.01     | 70.74     |\n| Qwen-7B               | 49.53     | 63.4      | 45.3      | 59.7      | 62.5      | 75        | 13.3      | 54.1      | 27.4      | 31.4      | 45.2      | 67.5      |\n| XVERSE-7B             | 34.27     | 61.1      | 39        | 58.4      | 60.8      | 73.7      | 2.2       | 11.7      | 4.9       | 10.2      | 31        | 24        |\n| Yi-6B                 | 47.8      | 73        | 44.3      | 64        | **73.5**  | 73.1      | 6.3       | 39.9      | 15.2      | 23.6      | 44.9      | 68        |\n| **360Zhinao-7B**      | 56.15     | **74.11** | 49.49     | **67.44** | 72.38     | **83.05** | 16.38     | 53.83     | 35.98     | 42.4      | 43.95     | **78.59** |\n\n\n## Chat Models\n\nThe 4K and 32K models are trained separately with the same 4K SFT data.\n\nTo train the long-context models, we adopted a two-stage approach.\n\n**First stage**: We increased RoPE base and extended the context length to 32K.\n- Firstly, we performed Continual Pretraining on approximately 5B tokens with a 32K context window.\n- Then during the SFT stage, we finetuned the model using long data from various sources, including high-quality human-labeled 32K data.\n\n**Second stage**: We extended the context length to 360K, training with the following data:\n- A small amount of high-quality human-labeled super-long data.\n- Due to the scarcity of annotated super-long data, we constructed various forms of synthetic data.\n    - Multi-Doc QA: Similar to [Ziya-Reader](https://arxiv.org/abs/2311.09198), we generated multi-document QA pairs based on 360's database. Multiple QA pairs are constructed for one row of Multi-Doc QA data input, resulting in a multi-turn format and significantly improving the training efficiency.\n    - Single-Doc QA: Similar to [LLama2 Long](https://arxiv.org/abs/2309.16039), we constructed multi-turn QA data based on different segments within one row of long-text input.\n\nWe evaluated our models across various lengths and benchmarks.\n\n- ### Long Context Benchmarks\n\n\n  We evaluated our 32K and 360K models on [LongBench](https://github.com/THUDM/LongBench), a multi-task bilingual benchmark for long contexts. We report results on **Chinese** tasks most relevant to downstream applications: Single/Multi-Doc QA, Summarization, Few-Shot Learning and Code Completion.\n\n    | Model                     | Avg       | Single-Doc QA  | Multi-Doc QA | Summarization | Few-Shot Learning | Code Completion |\n    | :------------------------ |:---------:|:--------:|:---------:|:---------:|:------------:|:---------:|\n    | GPT-3.5-Turbo-16k         | 37.84     | 61.2     | 28.7      | 16        | 29.2         | 54.1      |\n    | ChatGLM2-6B-32k           | 37.16     | 51.6     | 37.6      | 16.2      | 27.7         | 52.7      |\n    | ChatGLM3-6B-32k           | 44.62     | **62.3** | 44.8      | 17.8      | 42           | 56.2      |\n    | InternLM2-Chat-7B         | 42.20     | 56.65    | 29.15     | **17.99** | 43.5         | **63.72** |\n    | Qwen1.5-Chat-7B           | 36.75     | 52.85    | 30.08     | 14.28     | 32           | 54.55     |\n    | Qwen1.5-Chat-14B          | 39.80     | 60.39    | 27.99     | 14.77     | 37           | 58.87     |\n    | 360Zhinao-7B-Chat-32K     | **45.18** | 57.18    | **48.06** | 15.03     | **44**       | 61.64     |\n\n- ### 360Zhinao-7B-Chat-360K on \"NeedleInAHaystack\"\n\n  [NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack) places one small piece of information in different positions of long text and queries this information as a test of LLM's long-context capabilities.\n\n  360Zhinao-7B-Chat-360K could achieve over 98% accuracy on both English and Chinese NeedleInAHaystack tasks.\n\n  - English version（same as [NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)）\n  \n    \u003cp align=\"center\"\u003e\n        \u003cimg src=\"assets/360Zhinao-7B-Chat-360K.en_score.png\" width=\"600\" /\u003e\n    \u003cp\u003e\n\n    **needle**：The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.\n\n    **query**：What is the best thing to do in San Francisco?\n\n\n  - Chinese version\n\n    \u003cp align=\"center\"\u003e\n        \u003cimg src=\"assets/360Zhinao-7B-Chat-360K.zh_score.png\" width=\"600\" /\u003e\n    \u003cp\u003e\n\n    We constructed the Chinese version following the [SuperCLUE-200K benchmark](https://mp.weixin.qq.com/s/QgoRf2LB-7vc3vTFOHJkpw):\n\n    **haystack**：Chinese novels.\n  \n    **needle**：(in Chinese) 王莽是一名勤奋的店员，他每天凌晨就起床，赶在第一缕阳光照亮大地之前到达店铺，为即将开始的一天做准备。他清扫店铺，整理货架，为顾客提供方便。他对五金的种类和用途了如指掌，无论顾客需要什么，他总能准确地找到。\\n然而，他的老板刘秀却总是对他吹毛求疵。刘秀是个挑剔的人，他总能在王莽的工作中找出一些小错误，然后以此为由扣他的工资。他对王莽的工作要求非常严格，甚至有些过分。即使王莽做得再好，刘秀也总能找出一些小问题，让王莽感到非常沮丧。\\n王莽虽然对此感到不满，但他并没有放弃。他知道，只有通过自己的努力，才能获得更好的生活。他坚持每天早起，尽管他知道那天可能会再次被刘秀扣工资。他始终保持微笑，尽管他知道刘秀可能会再次对他挑剔。\n\n    **query**：(in Chinese) 王莽在谁的手下工作？\n\n\u003cbr\u003e\n\n# Quickstart\nWe provide simple examples illustrating the use of 360Zhinao-7B-Base and 360Zhinao-7B-Chat on 🤖ModelScope and 🤗Transformers.\n\n## Dependency Installation\n- python \u003e= 3.8\n- pytorch \u003e= 2.0\n- transformers \u003e= 4.37.2\n- CUDA \u003e= 11.4\n\n```shell\npip install -r requirements.txt \n```\n\nOptionally, we recommend installing Flash-Attention 2 to improve performance and reduce memory footprint.\n\n\u003eflash-attn \u003e= 2.3.6\n```shell\nFLASH_ATTENTION_FORCE_BUILD=TRUE pip install flash-attn==2.3.6\n```\n\n## 🤗 Transformers\n### Demonstration of Base Model Inference\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\nfrom transformers.generation import GenerationConfig\n\nMODEL_NAME_OR_PATH = \"qihoo360/360Zhinao-7B-Base\"\n\ntokenizer = AutoTokenizer.from_pretrained(\n    MODEL_NAME_OR_PATH, \n    trust_remote_code=True)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    MODEL_NAME_OR_PATH,\n    device_map=\"auto\",\n    trust_remote_code=True)\n\ngeneration_config = GenerationConfig.from_pretrained(\n    MODEL_NAME_OR_PATH,\n    trust_remote_code=True)\n\ninputs = tokenizer('中国二十四节气\\n1. 立春\\n2. 雨水\\n3. 惊蛰\\n4. 春分\\n5. 清明\\n', return_tensors='pt')\ninputs = inputs.to(model.device)\n\npred = model.generate(input_ids=inputs[\"input_ids\"], generation_config=generation_config)\nprint(\"outputs:\\n\", tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))\n```\n### Demonstration of Chat Model Inference\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\nfrom transformers.generation import GenerationConfig\n\nMODEL_NAME_OR_PATH = \"qihoo360/360Zhinao-7B-Chat-4K\"\n\ntokenizer = AutoTokenizer.from_pretrained(\n    MODEL_NAME_OR_PATH, \n    trust_remote_code=True)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    MODEL_NAME_OR_PATH,\n    device_map=\"auto\",\n    trust_remote_code=True)\n\ngeneration_config = GenerationConfig.from_pretrained(\n    MODEL_NAME_OR_PATH,\n    trust_remote_code=True)\n\nmessages = []\n#round-1\nmessages.append({\"role\": \"user\", \"content\": \"介绍一下刘德华\"})\nresponse = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)\nmessages.append({\"role\": \"assistant\", \"content\": response})\nprint(messages)\n\n#round-2\nmessages.append({\"role\": \"user\", \"content\": \"他有什么代表作？\"})\nresponse = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)\nmessages.append({\"role\": \"assistant\", \"content\": response})\nprint(messages)\n```\n\n## 🤖 ModelScope\n### Demonstration of Base Model Inference\n\n```python\nfrom modelscope import AutoModelForCausalLM, AutoTokenizer\nfrom modelscope import GenerationConfig\n\nMODEL_NAME_OR_PATH = \"qihoo360/360Zhinao-7B-Base\"\n\ntokenizer = AutoTokenizer.from_pretrained(\n    MODEL_NAME_OR_PATH, \n    trust_remote_code=True)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    MODEL_NAME_OR_PATH,\n    device_map=\"auto\",\n    trust_remote_code=True)\n\ngeneration_config = GenerationConfig.from_pretrained(\n    MODEL_NAME_OR_PATH,\n    trust_remote_code=True)\n\ninputs = tokenizer('中国二十四节气\\n1. 立春\\n2. 雨水\\n3. 惊蛰\\n4. 春分\\n5. 清明\\n', return_tensors='pt')\ninputs = inputs.to(model.device)\n\npred = model.generate(input_ids=inputs[\"input_ids\"], generation_config=generation_config)\nprint(\"outputs:\\n\", tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))\n```\n\n### Demonstration of Chat Model Inference\n\n```python\nfrom modelscope import AutoModelForCausalLM, AutoTokenizer\nfrom modelscope import GenerationConfig\n\nMODEL_NAME_OR_PATH = \"qihoo360/360Zhinao-7B-Chat-4K\"\n\ntokenizer = AutoTokenizer.from_pretrained(\n    MODEL_NAME_OR_PATH, \n    trust_remote_code=True)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    MODEL_NAME_OR_PATH,\n    device_map=\"auto\",\n    trust_remote_code=True)\n\ngeneration_config = GenerationConfig.from_pretrained(\n    MODEL_NAME_OR_PATH,\n    trust_remote_code=True)\n\nmessages = []\n#round-1\nmessages.append({\"role\": \"user\", \"content\": \"介绍一下刘德华\"})\nresponse = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)\nmessages.append({\"role\": \"assistant\", \"content\": response})\nprint(messages)\n\n#round-2\nmessages.append({\"role\": \"user\", \"content\": \"他有什么代表作？\"})\nresponse = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)\nmessages.append({\"role\": \"assistant\", \"content\": response})\nprint(messages)\n```\n\n## CLI Demo\nUse terminal for command-line interface:\n\n```shell\npython cli_demo.py\n```\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"assets/cli_demo.gif\" width=\"600\" /\u003e\n\u003cp\u003e\n\nNote: for Mac users, `device = 'mps'` is not supported yet.\n\n## Web Demo\n\n```shell\nstreamlit run web_demo.py\n```\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"assets/web_demo.gif\" width=\"600\" /\u003e\n\u003cp\u003e\n\n## API Demo\nLaunch api:\n```shell\npython openai_api.py\n```\n\nThen request with parameters:\n```shell\ncurl 'http://localhost:8360/v1/chat/completions' \\\n-H 'Content-Type: application/json' \\\n-d '{\n    \"max_new_tokens\": 200,\n    \"do_sample\": true,\n    \"top_k\": 0,\n    \"top_p\": 0.8,\n    \"temperature\": 1.0,\n    \"repetition_penalty\": 1.0,\n    \"messages\": [\n        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n        {\"role\": \"user\", \"content\": \"你好\"}\n    ]\n}'\n```\n\n\u003cbr\u003e\n\n# Model Inference\n## Quantization\nWe provide quantization schemes based on AutoGPTQ and release the Int4 quantization models. \n\n## Deployment\n### vLLM Installation\nWe recommend using `vLLM==0.3.3`.\n\nIf you are using **CUDA 12.1 and PyTorch 2.1**, you can install vLLM directly with:\n```shell\npip install vllm==0.3.3\n```\n\nOtherwise, please refer to the official vLLM [Installation Instructions](https://docs.vllm.ai/en/latest/getting_started/installation.html).\n\nAfter installation, perform the following steps:\n1. Copy `vllm/zhinao.py` into `vllm/model_executor/models` in your vllm installation directory (in python/conda env).\n2. Copy `vllm/serving_chat.py` into `vllm/entrypoints/openai` in your vllm installation directory.\n3. Then add a line in `vllm/model_executor/models/__init__.py`\n\n    ```shell\n    \"ZhinaoForCausalLM\": (\"zhinao\", \"ZhinaoForCausalLM\"),\n    ```\n\n### vLLM Service Start\n\nStart the service:\n```shell\npython -m vllm.entrypoints.openai.api_server \\\n    --served-model-name 360Zhinao-7B-Chat-4K \\\n    --model qihoo360/360Zhinao-7B-Chat-4K \\\n    --trust-remote-code \\\n    --tensor-parallel-size 1 \\\n    --max-model-len 4096 \\\n    --host 0.0.0.0 \\\n    --port 8360\n```\n\nUse curl to request the service:\n```shell\ncurl http://localhost:8360/v1/chat/completions \\\n-H \"Content-Type: application/json\" \\\n-d '{\n    \"model\": \"360Zhinao-7B-Chat-4K\",\n    \"max_tokens\": 200,\n    \"top_k\": -1,\n    \"top_p\": 0.8,\n    \"temperature\": 1.0,\n    \"presence_penalty\": 0.0,\n    \"frequency_penalty\": 0.0,\n    \"messages\": [\n        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n        {\"role\": \"user\", \"content\": \"你好\"}\n    ],\n    \"stop\": [\n        \"\u003ceod\u003e\",\n        \"\u003c|im_end|\u003e\",\n        \"\u003c|im_start|\u003e\"\n    ]\n}'\n```\nUse python to request the service:\n```python\nfrom openai import OpenAI\nopenai_api_key = \"EMPTY\"\nopenai_api_base = \"http://localhost:8360/v1\"\n\nclient = OpenAI(\n    api_key=openai_api_key,\n    base_url=openai_api_base,\n)\n\nchat_response = client.chat.completions.create(\n    model=\"360Zhinao-7B-Chat-4K\",\n    messages=[\n        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n        {\"role\": \"user\", \"content\": \"你好\"},\n    ],\n    stop=[\n        \"\u003ceod\u003e\",\n        \"\u003c|im_end|\u003e\",\n        \"\u003c|im_start|\u003e\"\n    ],\n    presence_penalty=0.0,\n    frequency_penalty=0.0\n)\nprint(\"Chat response:\", chat_response)\n```\n\n\u003e If you need to enable repetition penalty, we recommend setting `presence_penalty` and `frequency_penalty` instead of `repetition_penalty`.\n\n\n\u003cbr\u003e\n\n# Model Finetune\n## Training data\n\nTraining Data: `data/training_data_sample.json`. This example data has 10,000 rows sampled from [multiturn_chat_0.8M](https://huggingface.co/datasets/BelleGroup/multiturn_chat_0.8M) with converted format.\n\nData Format:\n```json\n[\n  {\n    \"id\": 1,\n    \"conversations\": [\n        {\n            \"from\": \"system\",\n            \"value\": \"You are a helpful assistant.\"\n        },\n        {\n            \"from\": \"user\",\n            \"value\": \"您好啊\"\n        },\n        {\n            \"from\": \"assistant\",\n            \"value\": \"你好！我今天能为您做些什么？有什么问题或需要帮助吗? 我在这里为您提供服务。\"\n        }\n    ]\n  }\n]\n```\n## Finetuning scripts\n```shell\nset -x\n\nHOSTFILE=hostfile\nDS_CONFIG=./finetune/ds_config_zero2.json\n\n# PARAMS\nLR=5e-6\nEPOCHS=3\nMAX_LEN=4096\nBATCH_SIZE=4\nNUM_NODES=1\nNUM_GPUS=8\nMASTER_PORT=29500\n\nIS_CONCAT=False # Whether to concatenate to maximum length (MAX_LEN)\n\nDATA_PATH=\"./data/training_data_sample.json\"\nMODEL_PATH=\"qihoo360/360Zhinao-7B-Base\"\nOUTPUT_DIR=\"./outputs/\"\n\ndeepspeed --hostfile ${HOSTFILE} \\\n        --master_port ${MASTER_PORT} \\\n        --num_nodes ${NUM_NODES} \\\n        --num_gpus ${NUM_GPUS} \\\n        finetune.py \\\n        --report_to \"tensorboard\" \\\n        --data_path ${DATA_PATH} \\\n        --model_name_or_path ${MODEL_PATH} \\\n        --output_dir ${OUTPUT_DIR} \\\n        --model_max_length ${MAX_LEN} \\\n        --num_train_epochs ${EPOCHS} \\\n        --per_device_train_batch_size ${BATCH_SIZE} \\\n        --gradient_accumulation_steps 1 \\\n        --save_strategy steps \\\n        --save_steps 200 \\\n        --learning_rate ${LR} \\\n        --lr_scheduler_type cosine \\\n        --adam_beta1 0.9 \\\n        --adam_beta2 0.95 \\\n        --adam_epsilon 1e-8 \\\n        --max_grad_norm 1.0 \\\n        --weight_decay 0.1 \\\n        --warmup_ratio 0.01 \\\n        --gradient_checkpointing True \\\n        --bf16 True \\\n        --tf32 True \\\n        --deepspeed ${DS_CONFIG} \\\n        --is_concat ${IS_CONCAT} \\\n        --logging_steps 1 \\\n        --log_on_each_node False\n```\n```shell\nbash finetune/ds_finetune.sh\n```\n- Configuring `HOSTFILE` switches between single-machine and multi-machine training.\n- configuring `ds_config` switches between zero1, zero2 and zero3.\n- `fp16, bf16` could configure mixed precision training. bf16 is recommended to be consistent with the pretrained model.\n- `is_concat` configures whether the training data is concatenated or not.\n\n\u003cbr\u003e\n\n\n# 360Zhinao-search Model Introduction\n360Zhinao-search uses the self-developed BERT model as the base for multi-task fine-tuning, which has an average score of 75.05 on the Retriev    al task on the C-MTEB-Retrieval benchmark, currently ranking first.\n[C-MTEB-Retrieval leaderboard](https://huggingface.co/spaces/mteb/leaderboard) contains a total of 8 [query, passage] similarity retrieval sub    tasks in different fields, using NDCG@10 (Normalized Discounted Cumulative Gain @ 10) as the evaluation index.\n\n| Model | T2Retrieval | MMarcoRetrieval | DuRetrieval | CovidRetrieval | CmedqaRetrieval | EcomRetrieval | MedicalRetrieval | VideoRetrieval |     Avg |\n|:-------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|\n|**360Zhinao-search** | 87.12 | 83.32 | 87.57 | 85.02 | 46.73 | 68.9 | 63.69 | 78.09 | **75.05** |\n|AGE_Hybrid | 86.88 | 80.65 | 89.28 | 83.66 | 47.26 | 69.28 | 65.94 | 76.79 | 74.97 |\n|OpenSearch-text-hybrid | 86.76 | 79.93 | 87.85 | 84.03 | 46.56 | 68.79 | 65.92 | 75.43 | 74.41 |\n|piccolo-large-zh-v2 | 86.14 | 79.54 | 89.14 | 86.78 | 47.58 | 67.75 | 64.88 | 73.1 | 74.36 |\n|stella-large-zh-v3-1792d | 85.56 | 79.14 | 87.13 | 82.44 | 46.87 | 68.62 | 65.18 | 73.89 | 73.6 |\n\n## Optimization points\n1. Data filtering: Strictly prevent the C-MTEB-Retrieval test data from leaking, and clean all queries and passages in the test set;\n2. Data source enhancement: Use open source data and LLM synthetic data to improve data diversity;\n3. Negative example mining: Use multiple methods to deeply mine difficult-to-distinguish negative examples to improve information gain;\n4. Training efficiency: multi-machine multi-CPU + Deepspeed method to optimize GPU memory utilization.\n\n## Environmental requirements\n```bash\ncd Retrieval\npip install -r requirements.txt\n```\n\n## Training script\n```bash\ncd Retrieval/finetune\nsh train.sh\n```\n\n## Inference script\n```bash\ncd Retrieval/eval\npython test_model.py\n```\n\n## C-MTEB test script\n```bash\ncd Retrieval/eval\nsh eval.sh\n```\n\n## Reference\n[bge fine-tuning code](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune)\n[C-MTEB official test script](https://github.com/FlagOpen/FlagEmbedding/tree/master/C_MTEB)\n\n\n\n# 360Zhinao-1.8B-Reranking Model Introduction\nThe 360Zhinao-1.8B-Reranking model utilizes the self-developed 360Zhinao_1.8B_base model as its foundation. Our self-developed unidirectional     generative model, 360Zhinao_1.8B_reranking, achieved an average score of 70.13, currently ranking first overall and first among open-source mo    dels, opening up new possibilities for generative models to undertake discriminative tasks.\n\n[C-MTEB-Reranking leaderboard](https://huggingface.co/spaces/mteb/leaderboard) contains four subtasks, which are tasks of judging the similari    ty of user questions and answers in different fields. It uses MAP (Mean-average-precision) as the evaluation index. Currently, the open-source     models on this leaderboard are primarily bidirectional discriminative models (BERT-like models). The only unidirectional generative model (GP    T-like model) is gte-Qwen1.5-7B-instruct, which has an average score of 66.38, ranking 25th, with less than ideal results.\n\n\n| Model | T2Reranking | MMarcoReranking | CMedQAv1 | CMedQAv2 | Avg |\n|:-------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|\n| **360Zhinao-1.8B-Reranking** | **68.55** | **37.29** | **86.75** | **87.92** | **70.13** |\n| piccolo-large-zh-v2 | 67.15 | 33.39 | 90.14 | 89.31 | 70 |\n| Baichuan-text-embedding | 67.85 | 34.3 | 88.46 | 88.06 | 69.67 |\n| stella-mrl-large-zh-v3.5-1792d | 66.43 | 28.85 | 89.18 | 89.33 | 68.45 |\n| PEG | 69.43 | 33.55 | 86.56 | 84.09 | 68.41 |\n| bge-reranker-base | 67.28 | 35.46 | 81.27 | 84.1 | 67.03 |\n| bge-reranker-large | 67.6 | 37.17 | 82.14 | 84.19 | 67.78 |\n\n\n## Optimization points\nThrough iterative discovery and resolution of the following technical issues, it continuously stimulates the world knowledge inherent in the l    arge model during the pre-training phase, better bridging the gap between generative models and discriminative tasks.\n\n1. Data Processing: The model training did not utilize world knowledge, meaning it neither continued pre-training with domain-specific data no    r fine-tuned datasets outside of the four datasets on the leaderboard. It only used the four datasets within the leaderboard, carefully iterat    ing through data perception, and targeting different datasets for data cleaning and mining to ensure that the ranking in individual tasks coul    d reach the top three.\n2. Resolving Task Conflicts: When merging four tasks, due to different data domain distributions, answer patterns, training data volumes, conv    ergence steps, and even sequence lengths, conflicts exist between different tasks. Deeply resolving these conflict issues is crucial to obtain    ing a universal model with the best comprehensive indicators across different tasks.\n3. Resolving Training Instability: Unlike generative tasks that produce multiple characters, using generative models for discriminative tasks     requires the model to output a continuous value. Therefore, there is an oscillation problem during the training process. Deeply analyzing and     resolving training instability can result in a model with better generalization and robustness.\n\n\n# Environmental requirements\n```bash\ncd Reranking\npip install -r requirements.txt\n```\n\nIf your GPU supports fp16 or bf16 precision, we also recommend installing [flash-attention](https://github.com/Dao-AILab/flash-attention) (**n    ow with support for flash attention 2**) to improve your runtime efficiency and reduce memory usage. (**flash-attention is optional and not re    quired for running this project**)\n\n```bash\ngit clone https://github.com/Dao-AILab/flash-attention\ncd flash-attention \u0026\u0026 pip install .\n# The installation below is optional and might be slow.\n# pip install csrc/layer_norm\n# No need to install the following if the flash-attn version is above 2.1.1.\n# pip install csrc/rotary\n```\n\n## Input Format\n```json\n[\n  {\n    \"id\": \"identity_0\",\n    \"conversations\": [\n      {\n        \"from\": \"user\",\n        \"value\": \"What Color Is the Sky\\n\\nBlue\"\n      },\n      {\n        \"from\": \"assistant\",\n        \"value\": \"3\"\n      }\n    ]\n  }\n]\n```\n\n## Training Script\n```bash\ncd Reranking\nsh finetune/finetune_ds.sh\n```\n\n## Inference Script\n```bash\ncd Reranking\npython test_model.py\n```\n\n# Citation\n\nIf you find our work helpful, feel free to cite as:\n\n```\n@article{qwen,\n  title={360Zhinao Technical Report},\n  author={360Zhinao-Team},\n  journal={arXiv preprint arXiv:2405.13386},\n  year={2024}\n}\n```\n\n\n# License\n\nThe source code of this repository follows the open-source license Apache 2.0.\n\n360​Zhinao open-source models support commercial use. If you wish to use these models or continue training them for commercial purposes, please contact us via email (g-zhinao-opensource@360.cn) to apply. For the specific license agreement, please see [\u003c\u003c360 Zhinao Open-Source Model License\u003e\u003e](https://github.com/Qihoo360/360zhinao/blob/main/360%E6%99%BA%E8%84%91%E5%BC%80%E6%BA%90%E6%A8%A1%E5%9E%8B%E8%AE%B8%E5%8F%AF%E8%AF%81.txt).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FQihoo360%2F360zhinao","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FQihoo360%2F360zhinao","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FQihoo360%2F360zhinao/lists"}