{"id":44288078,"url":"https://github.com/lance-format/lance-context","last_synced_at":"2026-02-10T23:07:19.783Z","repository":{"id":333098021,"uuid":"1135881873","full_name":"lance-format/lance-context","owner":"lance-format","description":"Manage Multimodal Agentic Context Lifecycle with Lance","archived":false,"fork":false,"pushed_at":"2026-01-29T02:35:26.000Z","size":413,"stargazers_count":41,"open_issues_count":1,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-01-29T13:51:37.472Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lance-format.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-16T18:16:05.000Z","updated_at":"2026-01-29T02:35:31.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/lance-format/lance-context","commit_stats":null,"previous_names":["lance-format/lance-context"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/lance-format/lance-context","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lance-format%2Flance-context","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lance-format%2Flance-context/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lance-format%2Flance-context/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lance-format%2Flance-context/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lance-format","download_url":"https://codeload.github.com/lance-format/lance-context/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lance-format%2Flance-context/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29321277,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-10T20:44:44.282Z","status":"ssl_error","status_checked_at":"2026-02-10T20:44:43.393Z","response_time":65,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-10T23:07:19.086Z","updated_at":"2026-02-10T23:07:19.769Z","avatar_url":"https://github.com/lance-format.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lance-context\n\nMultimodal, versioned context storage for agentic workflows built on top of [Lance](https://lancedb.github.io/lance/).\n\nLance Context gives AI agents a durable memory that can store text, binary payloads (images, Arrow tables, etc.), and semantic embeddings in a single columnar table. Every append produces a new Lance dataset version, so you can time-travel to prior checkpoints, branch off experiments, or reproduce conversations. The project ships with both a Rust API and a thin, Pythonic wrapper that integrates easily with orchestration frameworks.\n\n## Why another context store?\n\nKey motivations inspired by the broader Lance roadmap\u003csup\u003e[1](https://github.com/lance-format/lance/discussions/5716)\u003c/sup\u003e:\n\n- **Multimodal first** – store text, images, and structured data together, keeping the original bytes plus typed metadata.\n- **Version aware** – each append creates an immutable snapshot, enabling time-travel, branching, and auditability for long-running agents.\n- **Searchable semantics** – embeddings are managed alongside content so you can run Lance vector search without leaving the dataset.\n- **Columnar performance** – backed by the Lance file format, giving fast analytics, compaction, and cloud-friendly storage.\n\n## Features\n\n- Unified schema for agent messages (`ContextRecord`) with optional embeddings and metadata.\n- Automatic versioning via Lance manifests with `checkout(version)` support.\n- Background compaction to optimize storage and read performance.\n- Remote persistence: point the store at `s3://` URIs with either AWS environment variables or explicit credentials/endpoint overrides.\n- Python API (`lance_context.api.Context`) aligned with the Rust implementation.\n- Integration tests that exercise real persistence, image serialization, and version rollbacks.\n\n## Project layout\n\n```\ncrates/lance-context-core  # Pure Rust context engine (no Python deps)\ncrates/lance-context       # Re-export crate consumed by downstream clients/bindings\npython/                    # PyO3 bindings, wheel build, and pytest suite\npython/tests/              # High-level integration tests\n```\n\n## Getting started\n\nInstall the Python package (wheel publishing coming soon):\n\n```bash\npip install lance-context\n```\n\nThen follow the usage examples below to create a `Context`, append entries, and time-travel through versions.\n\n## Usage\n\n### Python\n\n```python\nfrom pathlib import Path\nfrom lance_context.api import Context\n\nuri = Path(\"context.lance\").as_posix()\nctx = Context.create(uri)\n\n# Add multimodal entries\nctx.add(\"user\", \"Where should I travel in spring?\")\n\nfrom PIL import Image\nimage = Image.new(\"RGB\", (2, 2), color=\"teal\")\nctx.add(\"assistant\", image)\n\nprint(\"Current version:\", ctx.version())\n\n# Time-travel to prior state\nfirst_version = ctx.version()\nctx.add(\"assistant\", \"Let me fetch suggestions…\")\nctx.checkout(first_version)\n\nprint(\"Entries after checkout:\", ctx.entries())\n\n# Store context in S3 (e.g., for MinIO/moto test endpoints)\nctx = Context.create(\n    \"s3://my-bucket/context.lance\",\n    aws_access_key_id=\"minioadmin\",\n    aws_secret_access_key=\"minioadmin\",\n    region=\"us-east-1\",\n    endpoint_url=\"http://localhost:9000\",\n    allow_http=True,\n)\n# AWS_* environment variables work too—pass overrides only when you need custom endpoints.\n\n# Background Compaction - optimize storage and read performance\nctx = Context.create(\n    \"context.lance\",\n    enable_background_compaction=True,  # Enable automatic compaction\n    compaction_interval_secs=300,       # Check every 5 minutes\n    compaction_min_fragments=10,        # Trigger when 10+ fragments exist\n    quiet_hours=[(22, 6)],              # Skip compaction 10pm-6am\n)\n\n# Manual compaction control\nfor i in range(100):\n    ctx.add(\"user\", f\"message {i}\")  # Creates many small fragments\n\n# Check compaction status\nstats = ctx.compaction_stats()\nprint(f\"Fragments: {stats['total_fragments']}\")\n\n# Manually trigger compaction\nmetrics = ctx.compact()\nprint(f\"Compaction removed {metrics['fragments_removed']} fragments\")\n```\n\n### Rust\n\n```rust\nuse lance_context::{ContextStore, ContextRecord, StateMetadata};\nuse chrono::Utc;\n\n# tokio_test::block_on(async {\nlet mut store = ContextStore::open(\"context.lance\").await?;\nlet record = ContextRecord {\n    id: \"run-1-1\".into(),\n    run_id: \"run-1\".into(),\n    created_at: Utc::now(),\n    role: \"user\".into(),\n    state_metadata: Some(StateMetadata {\n        step: Some(1),\n        active_plan_id: None,\n        tokens_used: None,\n        custom: None,\n    }),\n    content_type: \"text/plain\".into(),\n    text_payload: Some(\"hello world\".into()),\n    binary_payload: None,\n    embedding: None,\n};\nstore.add(\u0026[record]).await?;\nprintln!(\"Current version {}\", store.version());\n# Ok::\u003c(), Box\u003cdyn std::error::Error\u003e\u003e(())\n# })?;\n```\n\n## Testing\n\n- `make test` – Python pytest suite (including persistence integration tests).\n- `cargo test --manifest-path crates/lance-context-core/Cargo.toml` – Rust unit tests.\n- `python/.venv/bin/ruff check python/` and `python/.venv/bin/pyright` – linting/type checks.\n\n## Roadmap\n\nWe are tracking future enhancements as GitHub issues:\n\n- [Support S3-backed context stores](https://github.com/lance-format/lance-context/issues/14)\n- [Add relationship column for GraphRAG workflows](https://github.com/lance-format/lance-context/issues/15)\n- ~~[Background compaction for Lance fragments](https://github.com/lance-format/lance-context/issues/16)~~ ✅ **Implemented**\n\nContributions are welcome—feel free to comment on the issues above or open your own proposals.\n\n## Contributing\n\n1. Fork and clone the repository.\n2. Create a feature branch off `main`.\n3. Set up the development environment:\n   ```bash\n   make venv      # creates python/.venv using uv\n   make install   # installs the package in editable mode with test extras\n   make test      # runs pytest (python/tests/)\n   cargo test --manifest-path crates/lance-context-core/Cargo.toml\n   ```\n4. Run linting/type checks: `python/.venv/bin/ruff check python/`, `python/.venv/bin/pyright`, and `~/.cargo/bin/cargo fmt -- --check`.\n5. Open a Pull Request with a clear summary of the change.\n\n## License\n\nLicensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flance-format%2Flance-context","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flance-format%2Flance-context","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flance-format%2Flance-context/lists"}