https://github.com/rhecosystemappeng/lightspeed-rag-pipeline
lightspeed rag pipeline
https://github.com/rhecosystemappeng/lightspeed-rag-pipeline
Last synced: 4 months ago
JSON representation
lightspeed rag pipeline
- Host: GitHub
- URL: https://github.com/rhecosystemappeng/lightspeed-rag-pipeline
- Owner: RHEcosystemAppEng
- Created: 2023-11-28T14:49:38.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-03-11T11:15:41.000Z (over 2 years ago)
- Last Synced: 2025-02-28T23:22:37.080Z (over 1 year ago)
- Language: Python
- Size: 83 KB
- Stars: 0
- Watchers: 3
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# LightSpeed RAG tetkon pipeline
This repo contains Tekton pipeline for creation of LLM embedding using Llama_index
- Loading data
- Indexing
- Storing
- Evaluating
## Prerequisites
- Tekton
- OpenShift
- Nvidia Operator installed
- Kostomize CLI
## Usage
- Build the embedding container under `images/rag` using the `build.sh` file.
- Add secrets resources to the secret folder and to link the secretes to the pipeline service account
- Copy and edit the pipeline configuration file `pipeline/data-pipeline.yaml`
- add you file adn resources to th `kustomization.yaml` file
- Deploy the pipeline pipeline using the following command:
```
oc apply -k .
```
- Run the pipeline
- Results:
- Container image in the specified location with the embedding json files
- GitHub Release zip file which contains the embedding files and metadata files
## Folders
- images - base images containing the tekton tasks
## References
https://docs.llamaindex.ai/en/stable/getting_started/concepts.html
https://docs.llamaindex.ai/en/stable/understanding/using_llms/using_llms.htm
https://blog.llamaindex.ai/build-and-scale-a-powerful-query-engine-with-llamaindex-and-ray-bfb456404bc4
https://tekton.dev/docs/pipelines/pipelines/