https://github.com/lancedb/lance-deeplearning-recipes
Deep Learning how-to's using Lance file format
https://github.com/lancedb/lance-deeplearning-recipes
Last synced: 9 months ago
JSON representation
Deep Learning how-to's using Lance file format
- Host: GitHub
- URL: https://github.com/lancedb/lance-deeplearning-recipes
- Owner: lancedb
- License: apache-2.0
- Created: 2024-03-25T12:36:49.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-09-18T11:11:57.000Z (over 1 year ago)
- Last Synced: 2025-03-22T21:51:09.599Z (10 months ago)
- Language: Python
- Size: 910 KB
- Stars: 15
- Watchers: 7
- Forks: 5
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# Lance Deep Learning - recipes
Dive into building Deep learning pipelines using Lance datasets!
This repository contains examples to help you use Lance datasets for your Deep learning projects.
- These are built using Lance, a free, open-source, columnar data format that **requires no setup**.
- High-performance random access: More than **1000x faster** than Parquet.
- Zero-copy, automatic versioning: manage versions of your data automatically, and reduce redundancy with zero-copy logic built-in.
Join our community for support - Discord •
Twitter
---
Why Lance
Convinience
Lance columnar file format is designed for large scale DL workloads. Columnar format allows you to easily and efficiently manage complex and unstructred multi-modal datasets Updation, filtering and zero-copy versioning allow you to iterate faster on large datasets. It’s designed to be used with images, videos, 3D point clouds, audio and of course tabular data. It supports any POSIX file systems, and cloud storage like AWS S3 and Google Cloud Storage
Performance
Lance format supports fast read/writes making your training time data loading significantly faster.
## Dataset Examples
Examples on how to convert existing datasets to Lance format.
| Example | Scripts | Read The Blog! |
|-------- | ------------- | ------------- |
| [Creating text dataset for LLM pre-training](/examples/wikitext-llm-dataset/) | 
| [](https://blog.lancedb.com/custom-dataset-for-llm-training-using-lance/)|
| [Creating Instruction dataset for LLM fine-tuning](/examples/alpaca-dataset/) | |
| [Creating Image Captioning Dataset for Multi-Modal Model Training](/examples/flickr8k-dataset/) | |
## Training Examples
Practical examples showcasing how to adapt your Lance dataset to popular deep learning projects.
| Example | Notebook & Scripts |
|-------- | ------------- |
| [PEFT Supervised Fine-tuning of Gemma using Huggingface Trainer](/examples/sft-gemma-hindi/) | |
| [LLM pre-training](/examples/llm-pretraining/) | |
| [COCO Image segmentation](/examples/image-segmentation/) | |
| [FSDP LLM pre-training](/examples/fsdp-llm-pretraining/) |
| [Wikiart Diffusion Training](/examples/diffusion-training/) | |
| [CLIP Training](/examples/clip-training/) | |
| [Image Classification](/examples/image-classification/) | |
| [Training a Variational AutoEncoder from scratch with Lance file format](/examples/variational-autoencoder/) | |
## Contributing Examples
If you're working on some cool deep learning examples using Lance that you'd like to add to this repo, please open a PR! More detailed instructions on contributing can be found on the [CONTRIBUTING.md](https://github.com/lancedb/lance-deeplearning-recipes/blob/main/CONTRIBUTING.md) page.