https://github.com/lancedb/lance-research

Research papers from Lance
https://github.com/lancedb/lance-research

Last synced: 7 months ago
JSON representation

Research papers from Lance

• Host: GitHub
• URL: https://github.com/lancedb/lance-research
• Owner: lancedb
• Created: 2023-12-14T19:34:12.000Z (about 2 years ago)
• Default Branch: main
• Last Pushed: 2025-02-27T23:29:19.000Z (11 months ago)
• Last Synced: 2025-02-28T08:12:42.021Z (11 months ago)
• Language: Rust
• Size: 4.01 MB
• Stars: 0
• Watchers: 6
• Forks: 0
• Open Issues: 8
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# LanceDB Research

This repository contains papers, experiments, and other research performed by the LanceDB team.

It's intended to serve as a transparent and open source resource for the community to learn

about the work we're doing and also to reproduce our results.

## Papers

### Lance: Efficient Random Access in Columnar Storage through Adaptive Structural Encodings

In this paper we introduce the 2.1 version of the Lance file format. We explore random access

performance on NVMe storage for Lance, Parquet, as well as Arrow-style approaches (what we used

in Lance 2.0). We justify the top-level structural encoding scheme for 2.1, demonstrate that we

are able to make good use of NVMe storage, and identify areas for future work.

We also explore full scan performance and verify that 2.1 is able to achieve similar compression

performance to Parquet and meet or exceed Parquet's scan performance in most cases.

Contents for this paper are located in the `file_2_1` directory.