https://github.com/spycsh/qwen-vl-hpu

draft of enabling Qwen-VL with static shape on HPU
https://github.com/spycsh/qwen-vl-hpu

Last synced: 24 days ago
JSON representation

draft of enabling Qwen-VL with static shape on HPU

• Host: GitHub
• URL: https://github.com/spycsh/qwen-vl-hpu
• Owner: Spycsh
• Created: 2024-07-09T07:14:44.000Z (4 months ago)
• Default Branch: main
• Last Pushed: 2024-07-17T08:40:42.000Z (4 months ago)
• Last Synced: 2024-07-18T08:11:52.452Z (4 months ago)
• Language: Python
• Homepage:
• Size: 77.1 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
BKC on HPU

```

docker run -itd -p 8091:80  --runtime=habana -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host vault.habana.ai/gaudi-docker/1.16.1/ubuntu22.04/habanalabs/pytorch-installer-2.2.2:latest

pip install tiktoken matplotlib

pip install einops transformers_stream_generator

pip install accelerate

pip install --upgrade-strategy eager optimum[habana]

git clone https://github.com/Spycsh/optimum-habana.git

cd optimum-habana

git checkout enable_qwen_vl

export PYTHONPATH=/root:/usr/lib/habanalabs/:/optimum-habana/

python test_baseline_cpu.py

cp ./modeling_qwen.py /root/.cache/huggingface/modules/transformers_modules/Qwen/Qwen-VL-Chat/f57cfbd358cb56b710d963669ad1bcfb44cdcdd8/modeling_qwen.py

python test_hpu.py

```

Rough perf comparison (1 Gaudi card vs. 8380 xeon cpu)

hpu: 80ms/token

cpu: 650ms/token