https://github.com/yas-sim/openvino_genai_sample_codes
OpenVINO.genai sample codes with a helper class that supports vLLM-like iterator-based streaming output.
https://github.com/yas-sim/openvino_genai_sample_codes
chatbot chatgpt edge-ai inference intel llm openvino python vllm
Last synced: about 1 year ago
JSON representation
OpenVINO.genai sample codes with a helper class that supports vLLM-like iterator-based streaming output.
- Host: GitHub
- URL: https://github.com/yas-sim/openvino_genai_sample_codes
- Owner: yas-sim
- License: apache-2.0
- Created: 2024-09-08T10:09:08.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-09-09T08:24:23.000Z (almost 2 years ago)
- Last Synced: 2025-03-24T10:38:54.551Z (over 1 year ago)
- Topics: chatbot, chatgpt, edge-ai, inference, intel, llm, openvino, python, vllm
- Language: Python
- Homepage:
- Size: 6.84 KB
- Stars: 4
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# OpenVINO.genai simple sample codes
|#|File name|Description|
|---|---|---|
|1|openvino_gengi_simple.py|Most simple LLM Q&A sample code|
|2|openvino_genai_simple_streaming.py|Simple LLM Q&A sample code. Streaming output enabled.|
|3|openvino_genai_streamer_test.py|Simple LLM Q&A sample code with vLLM-like iterator-based streaming output enabled.|
## Model preparation
The sample programs above requires LLM model to run.
You can prepare the LLM model in OpenVINO IR format with `optimum-cli` command in the `optimum-intel` PyPI package.
You can download the `TinyLlama-1.1B` model with following command.
```sh
pip install optimum-intel
optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --trust-remote-code "TinyLlama-1.1B-Chat-v1.0"
```
Please refer to the original [OpenVINO.genai GitHub repo](https://github.com/openvinotoolkit/openvino.genai) for further details.