https://github.com/hariprasath-v/workshop_open_source_data_ingestion_rags_dlt

Data ingestion pipeline using dlt to load data from a REST API into LanceDB.
https://github.com/hariprasath-v/workshop_open_source_data_ingestion_rags_dlt

colab-notebook dlt lancedb llama2 ollama rest-api sentence-transformers

Last synced: 6 months ago
JSON representation

Data ingestion pipeline using dlt to load data from a REST API into LanceDB.

• Host: GitHub
• URL: https://github.com/hariprasath-v/workshop_open_source_data_ingestion_rags_dlt
• Owner: hariprasath-v
• License: mit
• Created: 2024-07-16T14:43:52.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2024-07-16T14:55:50.000Z (about 1 year ago)
• Last Synced: 2025-04-08T08:47:49.108Z (6 months ago)
• Topics: colab-notebook, dlt, lancedb, llama2, ollama, rest-api, sentence-transformers
• Language: Jupyter Notebook
• Homepage:
• Size: 91.8 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Workshop_Open_Source_Data_Ingestion_RAGs_dlt

### Build a data ingestion pipeline using dlt to load data from a REST API into LanceDB.

### Covered topics

* Extract data from REST APIs

* Loading and vectorizing into LanceDB, which unlike other vector DBs stores the data and the embeddings

* Incremental loading