https://github.com/kamalkraj/e5-mistral-7b-instruct

Finetune mistral-7b-instruct for sentence embeddings
https://github.com/kamalkraj/e5-mistral-7b-instruct

finetuning huggingface lora mistral-7b peft pytorch sentence-embeddings transformers

Last synced: 10 days ago
JSON representation

Finetune mistral-7b-instruct for sentence embeddings

• Host: GitHub
• URL: https://github.com/kamalkraj/e5-mistral-7b-instruct
• Owner: kamalkraj
• License: apache-2.0
• Created: 2024-01-06T09:22:35.000Z (11 months ago)
• Default Branch: master
• Last Pushed: 2024-05-02T13:24:00.000Z (7 months ago)
• Last Synced: 2024-05-03T01:01:33.881Z (7 months ago)
• Topics: finetuning, huggingface, lora, mistral-7b, peft, pytorch, sentence-embeddings, transformers
• Language: Python
• Homepage:
• Size: 34.2 KB
• Stars: 53
• Watchers: 3
• Forks: 12
• Open Issues: 4
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# e5-mistral-7b-instruct

```bash

docker build -t pytorch .

```

```bash

docker run --gpus=all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -it --rm -v $(pwd):/e5-mistral-7b-instruct/ pytorch bash

```

### Prepare data

Run `prepare_dataset` to create a similarity dataset with one postive and negative pair from SNLI. 

```bash

python prepare_dataset.py

```

### Run model

set the model cache folder `export TRANSFORMERS_CACHE=.cache/`

First, run `accelerate config --config_file ds_zero3_cpu.yaml`

check the sample file for Single GPU [here](ds_zero3_cpu.yaml)

Below given parameter is taken from the paper for finetuning. 

Adjust accroding to your dataset and usecase.

```bash

accelerate launch \

    --config_file ds_zero3_cpu.yaml \

    peft_lora_embedding_semantic_search.py \

    --dataset_name similarity_dataset \

    --max_length 512 \

    --model_name_or_path intfloat/e5-mistral-7b-instruct \

    --per_device_train_batch_size 1 \

    --per_device_eval_batch_size 1 \

    --learning_rate 0.0001 \

    --weight_decay 0.01 \

    --max_train_steps 1000 \

    --gradient_accumulation_steps 2048 \

    --lr_scheduler_type linear \

    --num_warmup_steps 100 \

    --output_dir trained_model \

    --use_peft

```

[loss](loss.py) function copied from here -> https://github.com/RElbers/info-nce-pytorch