Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/gmftbyGMFTBY/science-llm

A large-scale language model for scientific domain, trained on redpajama arXiv split
https://github.com/gmftbyGMFTBY/science-llm

Last synced: 6 days ago
JSON representation

A large-scale language model for scientific domain, trained on redpajama arXiv split

Host: GitHub
URL: https://github.com/gmftbyGMFTBY/science-llm
Owner: gmftbyGMFTBY
License: apache-2.0
Created: 2023-05-24T12:38:10.000Z (over 1 year ago)
Default Branch: main
Last Pushed: 2024-03-01T03:46:36.000Z (8 months ago)
Last Synced: 2024-08-02T06:19:28.764Z (3 months ago)
Language: Python
Homepage:
Size: 16.3 MB
Stars: 119
Watchers: 3
Forks: 14
Open Issues: 1
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

Awesome-Domain-LLM - Mozi (墨子)
StarryDivineSky - gmftbyGMFTBY/science-llm

README

# Mozi: A Scientific Large-scale Language Model

![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-green.svg)
![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)

![墨子](./asset/墨子_avatar.png)

**Team of Beijing Institute of Technology:** [Tian Lan](https://github.com/gmftbyGMFTBY)^\*, Tianyi Che*, [Zewen Chi](https://github.com/CZWin32768), Xuhao Hu and Xian-ling Mao

****

## Catalogue:
* 1. Introduction
* 2. Running Mozi Demo
* 2.1. Environment Installation
* 2.2. Prepare Mozi Checkpoint
* 2.5. Deploying Demo
* 3. Train Your Own SciDPR model
* 3. Train Your Own Mozi models
* 3.1. Data Preparation
* 3.2. Training Configurations
* 3.3. Training Mozi
* 3.3. Scientific Pre-Training Mozi Models
* 3.3. Supervised Fine-tuning Mozi Models
* Citation
* Technical Report
* Acknowledgments

****

### 1. Introduction: [Back to Top]

Mozi is the first large-scale language model for the scientific paper domain, such as question answering and emotional support.
With the help of the large-scale language and evidence retrieval models, SciDPR, Mozi generates concise and accurate responses to users' questions about specific papers and provides emotional support for academic researchers.

We will explore more real-world application scenarios for Mozi in the future, making it a foundation model for solving various scientific tasks.

****

### 2. Running Mozi Models: [Back to Top]

#### 2.1. Environment Installation:
To install the required environment, please run
```
pip install -r requirements.txt
```

Then install the Pytorch package with the correct cuda version, for example
```
pip install torch==1.13.1+cu117 -f https://download.pytorch.org/whl/torch/
```

#### 2.2. Prepare Mozi Checkpoint:
The Mozi model weights (pre-trained on scientific corpus) consists of the pre-trained large-scale language and the LoRA weights.

1. First of all, please download [LLaMA-7B checkpoint](https://huggingface.co/decapoda-research/llama-7b-hf) and [Baichuan-7B checkpoint](https://huggingface.co/baichuan-inc/baichuan-7B).
2. Then, please download the LoRA weights for these two models from:

| LoRA checkpoints | Huggingface Delta Weights Address |
| ---------------- | --------------------------------- |
| Baichuan-7B delta weight | [mozi_baichuan_7b](https://huggingface.co/DataHammer/mozi_baichuan_7b) |
| LLaMA-7B delta weight | [mozi_llama_7b](https://huggingface.co/DataHammer/mozi_llama_7b) |

3. We also release the delta LoRA model weights for scientific emotional dialogue, which can be found in [here](). The emotional dialogue delta weights are built on Baichuan-7B model. In the future, we will directly optimize this scientific emotional dialogue instruction tuning dataset with other instruction dataset, such as paper-ground question answering and scientific information retrieval.

Now, the model parameters are all prepared.

#### 2.3. Deploying Demo:
Upon completion of previous steps, you can run the demo locally as
```bash
./scripts/deploy.sh

# #!/bin/bash
# CUDA_VISIBLE_DEVICES=0 python deploy.py \
# --model scillm-sft\
# --model_path baichuan-inc/baichuan-7B\
# --delta_model_path ../ckpt/scillm-emotional-sft/18\
# --port 23333
```

This script runs the Mozi model emotional model on `23333` port and the input `POST` request should be like:

```json
{
"decoding_method": "greedy",
"top_p": 0.7,
"top_k": 10,
"penalty_alpha": 0.5,
"max_new_tokens": 128,
"history": [
"Human: 最近科研压力真的好大啊"
]
}
```

If you want to test the paper-ground dialog model, please replace the `--delta_model_path` with the corresponding model checkpoints weights that you download.
For the paper-ground dialog, the input `POST` request should be like:

```json
{
"decoding_method": "greedy",
"top_p": 0.7,
"top_k": 10,
"penalty_alpha": 0.5,
"max_new_tokens": 128,
"evidences": [
"During the first two decades of the 21st century, the sharing and processing of vast amounts of data has become pervasive ...",
"One way of circumventing this problem is to anonymise the data by removing, ...",
"Given that this paper is concerned with text documents (e.g. medical records), the involved techniques are related to Natural Language Processing (NLP) ..."
],
"question": "Which dataset do the author use in this paper?"
}
```

****

### 3. Train SciDPR model for Mozi Model: [Back to Top]

SciDPR servers the evidence retrieval component for Mozi paper-ground question answering models, which retrieves related evidences of user's queries.
For model details about SciDPR model, please refer to this [README.md](./SciDPR/README.md) file.

****

### 4. Train Your Own Mozi Model: [Back to Top]

**Prerequisites:** Before training the model, making sure the environment is properly installed and the checkpoints of LLaMA-7B and Baichuan-7B are downloaded.

#### 4.1. Data Preparation: [Back to Top]

**Declaimer:** To ensure the reproducibility of our results, we have released our training dataset. The dataset must be used for research purpose only. The use of the dataset must comply with the licenses from original sources, i.e. QASPER and SciMRC. These datasets may be taken down when requested by the original authors.

Due to the limited computation resources, we only collect 4B tokens from Redpajama arXiv corpus for the first version of scientific pre-training, and the downloading scripts could be found in [this scripts](./scillm/data/pretrain/download_from_hf.py).

After downloading, put the downloaded file and unzip them under the [./data/](./data/) directory, and the directory should look like:
```bash
.
├── pretrain
│   ├── download_from_hf.py
│   ├── test
│   │   └── collect.py
│   └── train
│   ├── combine_chinese_corpus.sh
│   └── split.sh
└── sft
├── alpaca
│   └── alpaca_data.json
├── combine.py
├── dolly
│   ├── download_from_hf.py
│   └── train.json
├── emotional
│   └── train.json
├── processed_qasper_test_set.json
├── processed_scimrc_test_set.json
├── qasper
│   ├── README-test.md
│   ├── README.md
│   ├── collect.py
│   ├── qasper-dev-v0.3.json
│   ├── qasper-test-and-evaluator-v0.3.tgz
│   ├── qasper-test-v0.3.json
│   ├── qasper-train-dev-v0.3.tgz
│   ├── qasper-train-v0.3.json
│   ├── qasper_dev_sft.json
│   ├── qasper_evaluator.py
│   ├── qasper_sft.json
│   ├── qasper_test_sft.json
│   ├── qasper_train_sft.json
│   ├── qasper_yes_no_test_sft.json
│   ├── qasper_yes_no_train_sft.json
│   └── scillm_test.json
├── scimrc
│   ├── collect.py
│   ├── scimrc_dev_sft.json
│   ├── scimrc_test_sft.json
│   ├── scimrc_train_sft.json
│   ├── scimrc_yes_no_test_sft.json
│   ├── scimrc_yes_no_train_sft.json
│   ├── smrc_dev.jsonl
│   ├── smrc_test.jsonl
│   └── smrc_train.jsonl
└── train.json
```

After downloading these datasets and saving them at the proper path, please refer to [Dataset Prepareing Tutorial](./data/README.md) for preprocessing these for corpus for following training.

#### 4.2 Training Configurations: [Back to Top]

To train the model properly, we use the [QLoRA](https://github.com/artidoro/qlora) and [deepspeed](http://deepspeed.readthedocs.io/) toolkit. Before running, please make sure these essential toolkit are downloaded.

The configurations about training are shown as follows:
| Models | Model Name | Training Configurations |
| ------ | ---------- | ----------------------- |
| Scientific Pretraining | scillm | [scillm-train](./config/scillm.json); [scillm-deepspeed](./dsconfig/scillm.json) |
| Paper-ground SFT | scillm-sft | [scillm-sft-train](./config/scillm-sft.json); [scillm-deepspeed](./dsconfig/scillm-sft.json) |
| Paper-ground SFT | scillm-sft | [scillm-sft-train](./config/scillm-sft.json); [scillm-deepspeed](./dsconfig/scillm-sft.json) |

Please refer to these configuration file for more training details.
Note that these models are pre-trained on 8 x 3090 (24G) GPUs for over 9 days (over 4B tokens). As for paper-ground question answering SFT, the training process cost less than 3 hours (with 2000 steps).

#### 4.3. Training Mozi Models: [Back to Top]

##### 4.3.1. Scientific Pre-training Mozi Models: [Back to Top]

To pre-train Mozi on scientific pre-training corpus with 4B tokens, please run the following commands:

```bash
./scripts/train_pretrain.sh
```

The key arguments of the training script are as follows:
* `--model`: The model name listed in `config/base.json`.
* `--model_path`: The checkpoint for large-scale langauge models, `baichuan-inc/baichuan-7B` for baichuan-7B model and `decapoda-research/llama-7b-hf` for LLaMA-7B.
* `--train_data_path`: The path saves the pretraining corpus.
* `--log_path`: The directory that saves the pre-trained log in tensorboard format. This directory will be automatically created.
* `--save_path`: The directory which saves the trained QLoRA delta weights. This directory will be automatically created.

Note that the total training steps can be set in the `total_step` argument at [./config/base.yaml](./config/base.yaml) file. Set this argument carefully to make sure all the tokens will be used during training.

##### 4.3.2. Supversied Fine-tuning Mozi Models: [Back to Top]

Furthermore, to supervised fine-tune Mozi models on paper-ground question answering corpus, first make sure the `dataset` is set as `QASPERDataset`, and then please run the following commands:
```bash
./scripts/train_sft.sh
```

The key arguments of the training script are as follows:
* `--model`: The model name listed in `config/base.json`.
* `--model_path`: The checkpoint for large-scale langauge models, `baichuan-inc/baichuan-7B` for baichuan-7B model and `decapoda-research/llama-7b-hf` for LLaMA-7B.
* `--delta_model_path`: The LoRA checkpoint weighted pre-traing in Section 3.3.1. The SFT process will continue optimize these LoRA weights for paper-ground question answering task.
* `--train_data_path`: The path saves the pretraining corpus.
* `--log_path`: The directory that saves the pre-trained log in tensorboard format. This directory will be automatically created.
* `--save_path`: The directory which saves the trained QLoRA delta weights. This directory will be automatically created.

If you want to train the emotional dialog task, just simply replace the dataset with the path of emotional dataset path,and make sure the `dataset_name` in `config/bash.yaml` should also be set as `EmotionalDataset`.

****

### Citation:

If you found Mozi models useful in your research or applications, please kindly cite using the following BibTeX:
```
...
```

### Technical Report:

You can refer to our technical report for more details, which is saved in [this path](./asset/mozi_technical_report.pdf).

****

### Acknowledgments:

This repo benefits from [OpenAlpaca](https://github.com/yxuansu/OpenAlpaca), [PandaGPT](https://panda-gpt.github.io/). Thanks for their wonderful works!