{"id":14154119,"url":"https://github.com/bosun-ai/swiftide","last_synced_at":"2025-07-14T18:20:49.999Z","repository":{"id":243619976,"uuid":"812763402","full_name":"bosun-ai/swiftide","owner":"bosun-ai","description":"Fast, streaming indexing and query library for AI (RAG) applications, written in Rust","archived":false,"fork":false,"pushed_at":"2024-08-16T08:14:45.000Z","size":1464,"stargazers_count":74,"open_issues_count":30,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-08-16T09:25:24.679Z","etag":null,"topics":["ai","data","indexing","llm","llmops","ml","rag"],"latest_commit_sha":null,"homepage":"https://swiftide.rs","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bosun-ai.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-09T20:07:06.000Z","updated_at":"2024-08-16T08:52:30.000Z","dependencies_parsed_at":"2024-06-12T20:34:47.178Z","dependency_job_id":"f598cb15-98a4-4381-aefb-9e2a72715801","html_url":"https://github.com/bosun-ai/swiftide","commit_stats":null,"previous_names":["bosun-ai/fluyt-index"],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bosun-ai%2Fswiftide","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bosun-ai%2Fswiftide/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bosun-ai%2Fswiftide/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bosun-ai%2Fswiftide/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bosun-ai","download_url":"https://codeload.github.com/bosun-ai/swiftide/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":215735789,"owners_count":15923388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","data","indexing","llm","llmops","ml","rag"],"created_at":"2024-08-17T08:00:45.816Z","updated_at":"2025-07-14T18:20:49.979Z","avatar_url":"https://github.com/bosun-ai.png","language":"Rust","funding_links":[],"categories":["AI Orchestration \u0026 Deployment","Rust","🧰 Frameworks that Facilitate RAG","ai","Open Source Tools","RAG Tools","*Ops for AI"],"sub_categories":["Workflow Orchestration for AI","AI Orchestration \u0026 Deployment"],"readme":"\u003cdetails\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n\n\u003c!--toc:start--\u003e\n\n- [What is Swiftide?](#what-is-swiftide)\n  - [High level features](#high-level-features)\n- [Latest updates on our blog :fire:](#latest-updates-on-our-blog-fire)\n- [Examples](#examples)\n- [Vision](#vision)\n- [Features](#features)\n  - [In detail](#in-detail)\n- [Getting Started](#getting-started)\n  - [Prerequisites](#prerequisites)\n  - [Installation](#installation)\n- [Usage and concepts](#usage-and-concepts)\n  - [Indexing](#indexing)\n  - [Querying](#querying)\n- [Contributing](#contributing)\n- [Core Team Members](#core-team-members)\n- [License](#license)\n\u003c!--toc:end--\u003e\n\n\u003c/details\u003e\n\n\u003ca name=\"readme-top\"\u003e\u003c/a\u003e\n\n\u003c!-- PROJECT SHIELDS --\u003e\n\u003c!--\n*** I'm using markdown \"reference style\" links for readability.\n*** Reference links are enclosed in brackets [ ] instead of parentheses ( ).\n*** See the bottom of this document for the declaration of the reference variables\n*** for contributors-url, forks-url, etc. This is an optional, concise syntax you may use.\n*** https://www.markdownguide.org/basic-syntax/#reference-style-links\n--\u003e\n\n![CI](https://img.shields.io/github/actions/workflow/status/bosun-ai/swiftide/test.yml?style=flat-square)\n![Coverage Status](https://img.shields.io/coverallsCoverage/github/bosun-ai/swiftide?style=flat-square)\n[![Crate Badge]][Crate]\n[![Docs Badge]][API Docs]\n[![Contributors][contributors-shield]][contributors-url]\n[![Stargazers][stars-shield]][stars-url]\n![Discord](https://img.shields.io/discord/1257672801553354802?style=flat-square\u0026link=https%3A%2F%2Fdiscord.gg%2F3jjXYen9UY)\n[![MIT License][license-shield]][license-url]\n[![LinkedIn][linkedin-shield]][linkedin-url]\n\n\u003c!-- PROJECT LOGO --\u003e\n\u003cbr /\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://github.com/bosun-ai/swiftide\"\u003e\n    \u003cimg src=\"https://github.com/bosun-ai/swiftide/blob/master/images/logo.png\" alt=\"Logo\" width=\"250\" height=\"250\"\u003e\n  \u003c/a\u003e\n\n  \u003ch3 align=\"center\"\u003eSwiftide\u003c/h3\u003e\n\n  \u003cp align=\"center\"\u003e\nFast, streaming indexing, query, and agentic LLM applications in Rust\n    \u003cbr /\u003e\n    \u003ca href=\"https://swiftide.rs\"\u003e\u003cstrong\u003eRead more on swiftide.rs »\u003c/strong\u003e\u003c/a\u003e\n    \u003cbr /\u003e\n    \u003cbr /\u003e\n    \u003c!-- \u003ca href=\"https://github.com/bosun-ai/swiftide\"\u003eView Demo\u003c/a\u003e --\u003e\n    \u003ca href=\"https://docs.rs/swiftide/latest/swiftide/\"\u003eAPI Docs\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/bosun-ai/swiftide/issues/new?labels=bug\u0026template=bug_report.md\"\u003eReport Bug\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/bosun-ai/swiftide/issues/new?labels=enhancement\u0026template=feature_request.md\"\u003eRequest Feature\u003c/a\u003e\n    ·\n    \u003ca href=\"https://discord.gg/3jjXYen9UY\"\u003eDiscord\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n\u003c!-- ABOUT THE PROJECT --\u003e\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n## What is Swiftide?\n\n\u003c!-- [![Product Name Screen Shot][product-screenshot]](https://example.com) --\u003e\n\nSwiftide is a Rust library for building LLM applications, enabling fast data ingestion, transformation, and indexing for effective querying and prompt injection, known as Retrieval Augmented Generation. It provides flexible building blocks for creating various agents, allowing rapid development from concept to production with minimal code.\n\n### High level features\n\n- Build fast, streaming indexing and querying pipelines\n- Easily build agents, mix and match with previously built pipelines\n- A modular and extendable API, with minimal abstractions\n- Integrations with popular LLMs and storage providers\n- Ready to use pipeline transformations\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/bosun-ai/swiftide/blob/master/images/rag-dark.svg\" alt=\"RAG\" width=\"100%\" \u003e\n\u003c/div\u003e\n\nPart of the [bosun.ai](https://bosun.ai) project. An upcoming platform for autonomous code improvement.\n\nWe \u003c3 feedback: project ideas, suggestions, and complaints are very welcome. Feel free to open an issue or contact us on [discord](https://discord.gg/3jjXYen9UY).\n\n\u003e [!CAUTION]\n\u003e Swiftide is under heavy development and can have breaking changes. Documentation might fall short of all features, and despite our efforts be slightly outdated. We recommend to always keep an eye on our [github](https://github.com/bosun-ai/swiftide) and [api documentation](https://docs.rs/swiftide/latest/swiftide/). If you found an issue or have any kind of feedback we'd love to hear from you.\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n## Latest updates on our blog :fire:\n\n- [Releasing kwaak with kwaak](https://bosun.ai/posts/releasing-kwaak-with-kwaak/)\n- [Release - Swiftide 0.16](https://bosun.ai/posts/swiftide-0-16/)\n- [Rust in LLM based tools for performance](https://bosun.ai/posts/rust-for-genai-performance/)\n- [Evaluate Swiftide pipelines with Ragas](https://bosun.ai/posts/evaluating-swiftide-with-ragas/) (2024-09-15)\n- [Release - Swiftide 0.12](https://bosun.ai/posts/swiftide-0-12/) (2024-09-13)\n- [Local code intel with Ollama, FastEmbed and OpenTelemetry](https://bosun.ai/posts/ollama-and-telemetry/) (2024-09-04\n\n- [Release - Swiftide 0.9](https://bosun.ai/posts/swiftide-0-9/) (2024-09-02)\n- [Bring your own transformers](https://bosun.ai/posts/bring-your-own-transformers-in-swiftide/) (2024-08-13)\n- [Release - Swiftide 0.8](https://bosun.ai/posts/swiftide-0-8/) (2024-08-12)\n- [Release - Swiftide 0.7](https://bosun.ai/posts/swiftide-0-7/) (2024-07-28)\n- [Building a code question answering pipeline](https://bosun.ai/posts/indexing-and-querying-code-with-swiftide/) (2024-07-13)\n- [Release - Swiftide 0.6](https://bosun.ai/posts/swiftide-0-6/) (2024-07-12)\n- [Release - Swiftide 0.5](https://bosun.ai/posts/swiftide-0-5/) (2024-07-1)\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n## Examples\n\nIndexing a local code project, chunking into smaller pieces, enriching the nodes with metadata, and persisting into [Qdrant](https://qdrant.tech):\n\n```rust\nindexing::Pipeline::from_loader(FileLoader::new(\".\").with_extensions(\u0026[\"rs\"]))\n        .with_default_llm_client(openai_client.clone())\n        .filter_cached(Redis::try_from_url(\n            redis_url,\n            \"swiftide-examples\",\n        )?)\n        .then_chunk(ChunkCode::try_for_language_and_chunk_size(\n            \"rust\",\n            10..2048,\n        )?)\n        .then(MetadataQACode::default())\n        .then(move |node| my_own_thing(node))\n        .then_in_batch(Embed::new(openai_client.clone()))\n        .then_store_with(\n            Qdrant::builder()\n                .batch_size(50)\n                .vector_size(1536)\n                .build()?,\n        )\n        .run()\n        .await?;\n```\n\nQuerying for an example on how to use the query pipeline:\n\n```rust\nquery::Pipeline::default()\n    .then_transform_query(GenerateSubquestions::from_client(\n        openai_client.clone(),\n    ))\n    .then_transform_query(Embed::from_client(\n        openai_client.clone(),\n    ))\n    .then_retrieve(qdrant.clone())\n    .then_answer(Simple::from_client(openai_client.clone()))\n    .query(\"How can I use the query pipeline in Swiftide?\")\n    .await?;\n```\n\nRunning an agent that can search code:\n\n```rust\n#[swiftide::tool(\n    description = \"Searches code\",\n    param(name = \"code_query\", description = \"The code query\")\n)]\nasync fn search_code(\n    context: \u0026dyn AgentContext,\n    code_query: \u0026str,\n) -\u003e Result\u003cToolOutput, ToolError\u003e {\n    let command_output = context\n        .executor()\n        .exec_cmd(\u0026Command::shell(format!(\"rg '{code_query}'\")))\n        .await?;\n\n    Ok(command_output.into())\n}\n\nagents::Agent::builder()\n    .llm(\u0026openai)\n    .tools(vec![search_code()])\n    .build()?\n    .query(\"In what file can I find an example of a swiftide agent?\")\n    .await?;\n```\n\n_You can find more detailed examples in [/examples](https://github.com/bosun-ai/swiftide/tree/master/examples)_\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n## Vision\n\nOur goal is to create a fast, extendable platform for building LLM applications in Rust, to further the development of automated AI applications, with an easy-to-use and easy-to-extend api.\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n## Features\n\n- Fast, modular streaming indexing pipeline with async, parallel processing\n- Experimental query pipeline\n- Experimental agent framework\n- A variety of loaders, transformers, semantic chunkers, embedders, and more\n- Bring your own transformers by extending straightforward traits or use a closure\n- Splitting and merging pipelines\n- Jinja-like templating for prompts\n- Store into multiple backends\n- Integrations with OpenAI, Groq, Gemini, Anthropic, Redis, Qdrant, Ollama, FastEmbed-rs, Fluvio, LanceDB, and Treesitter\n- Evaluate pipelines with RAGAS\n- Sparse vector support for hybrid search\n- `tracing` supported for logging and tracing, see /examples and the `tracing` crate for more information.\n\n### In detail\n\n| **Feature**                                  | **Details**                                                                                                                                                          |\n| -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| **Supported Large Language Model providers** | OpenAI (and Azure) \u003cbr\u003e Anthropic \u003cbr\u003e Gemini \u003cbr\u003e OpenRouter \u003cbr\u003e AWS Bedrock - Anthropic and Titan \u003cbr\u003e Groq - All models \u003cbr\u003e Ollama - All models                 |\n| **Loading data**                             | Files \u003cbr\u003e Scraping \u003cbr\u003e Fluvio \u003cbr\u003e Parquet \u003cbr\u003e Other pipelines and streams                                                                                        |\n| **Transformers and metadata generation**     | Generate Question and answerers for both text and code (Hyde) \u003cbr\u003e Summaries, titles and queries via an LLM \u003cbr\u003e Extract definitions and references with tree-sitter |\n| **Splitting and chunking**                   | Markdown \u003cbr\u003e Text (text_splitter) \u003cbr\u003e Code (with tree-sitter)                                                                                                      |\n| **Storage**                                  | Qdrant \u003cbr\u003e Redis \u003cbr\u003e LanceDB                                                                                                                                       |\n| **Query pipeline**                           | Similarity and hybrid search, query and response transformations, and evaluation                                                                                     |\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\u003c!-- GETTING STARTED --\u003e\n\n## Getting Started\n\n### Prerequisites\n\nMake sure you have the rust toolchain installed. [rustup](https://rustup.rs) Is the recommended approach.\n\nTo use OpenAI, an API key is required. Note that by default `async_openai` uses the `OPENAI_API_KEY` environment variables.\n\nOther integrations might have their own requirements.\n\n### Installation\n\n1. Set up a new Rust project\n2. Add swiftide\n\n   ```sh\n   cargo add swiftide\n   ```\n\n3. Enable the features of integrations you would like to use in your `Cargo.toml`\n4. Write a pipeline (see our examples and documentation)\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\u003c!-- USAGE EXAMPLES --\u003e\n\n## Usage and concepts\n\nBefore building your streams, you need to enable and configure any integrations required. See /examples.\n\n_We have a lot of examples, please refer to /examples and the [Documentation](https://docs.rs/swiftide/latest/swiftide/)_\n\n\u003e [!NOTE]\n\u003e No integrations are enabled by default as some are code heavy. We recommend you to cherry-pick the integrations you need. By convention flags have the same name as the integration they represent.\n\n### Indexing\n\nAn indexing stream starts with a Loader that emits Nodes. For instance, with the Fileloader each file is a Node.\n\nYou can then slice and dice, augment, and filter nodes. Each different kind of step in the pipeline requires different traits. This enables extension.\n\nNodes have a path, chunk and metadata. Currently metadata is copied over when chunking and _always_ embedded when using the OpenAIEmbed transformer.\n\n- **from_loader** `(impl Loader)` starting point of the stream, creates and emits Nodes\n- **filter_cached** `(impl NodeCache)` filters cached nodes\n- **then** `(impl Transformer)` transforms the node and puts it on the stream\n- **then_in_batch** `(impl BatchTransformer)` transforms multiple nodes and puts them on the stream\n- **then_chunk** `(impl ChunkerTransformer)` transforms a single node and emits multiple nodes\n- **then_store_with** `(impl Storage)` stores the nodes in a storage backend, this can be chained\n\nAdditionally, several generic transformers are implemented. They take implementers of `SimplePrompt` and `EmbedModel` to do their things.\n\n\u003e [!WARNING]\n\u003e Due to the performance, chunking before adding metadata gives rate limit errors on OpenAI very fast, especially with faster models like 3.5-turbo. Be aware.\n\n### Querying\n\nA query stream starts with a search strategy. In the query pipeline a `Query` goes through several stages. Transformers and retrievers work together to get the right context into a prompt, before generating an answer. Transformers and Retrievers operate on different stages of the Query via a generic statemachine. Additionally, the search strategy is generic over the pipeline and Retrievers need to implement specifically for each strategy.\n\nThat sounds like a lot but, tl\u0026dr; the query pipeline is _fully and strongly typed_.\n\n- **Pending** The query has not been executed, and can be further transformed with transformers\n- **Retrieved** Documents have been retrieved, and can be further transformed to provide context for an answer\n- **Answered** The query is done\n\nAdditionally, query pipelines can also be evaluated. I.e. by [Ragas](https://ragas.io).\n\nSimilar to the indexing pipeline each step is governed by simple Traits and closures implement these traits as well.\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\u003c!-- ROADMAP --\u003e\n\n## Contributing\n\nSwiftide is in a very early stage and we are aware that we lack features for the wider community. Contributions are very welcome. :tada:\n\nIf you have a great idea, please fork the repo and create a pull request. You can also simply open an issue with the tag \"enhancement\".\nDon't forget to give the project a star! Thanks again!\n\nIf you just want to contribute (bless you!), see [our issues](https://github.com/bosun-ai/swiftide/issues) or join us on Discord.\n\n1. Fork the Project\n2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your Changes (`git commit -m 'feat: Add some AmazingFeature'`)\n4. Push to the Branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\nSee [CONTRIBUTING](https://github.com/bosun-ai/swiftide/blob/master/CONTRIBUTING.md) for more\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n## Core Team Members\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/timonv\"\u003e\n        \u003cimg\n          src=\"https://avatars.githubusercontent.com/u/49373?s=100\"\n          width=\"100px;\"\n          alt=\"\"\n        /\u003e\n        \u003cbr /\u003e\u003csub\u003e\u003cb\u003etimonv\u003c/b\u003e\u003c/sub\u003e\n        \u003cbr /\u003e\u003csub\u003eopen for swiftide consulting\u003c/sub\u003e\n      \u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/tinco\"\u003e\n        \u003cimg\n          src=\"https://avatars.githubusercontent.com/u/22532?s=100\"\n          width=\"100px;\"\n          alt=\"\"\n        /\u003e\n        \u003cbr /\u003e\u003csub\u003e\u003cb\u003etinco\u003c/b\u003e\u003c/sub\u003e\n        \u003cbr /\u003e\u003cbr /\u003e\n      \u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c!-- LICENSE --\u003e\n\n## License\n\nDistributed under the MIT License. See `LICENSE` for more information.\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\u003c!-- MARKDOWN LINKS \u0026 IMAGES --\u003e\n\u003c!-- https://www.markdownguide.org/basic-syntax/#reference-style-links --\u003e\n\n[contributors-shield]: https://img.shields.io/github/contributors/bosun-ai/swiftide.svg?style=flat-square\n[contributors-url]: https://github.com/bosun-ai/swiftide/graphs/contributors\n[stars-shield]: https://img.shields.io/github/stars/bosun-ai/swiftide.svg?style=flat-square\n[stars-url]: https://github.com/bosun-ai/swiftide/stargazers\n[license-shield]: https://img.shields.io/github/license/bosun-ai/swiftide.svg?style=flat-square\n[license-url]: https://github.com/bosun-ai/swiftide/blob/master/LICENSE.txt\n[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=flat-square\u0026logo=linkedin\u0026colorB=555\n[linkedin-url]: https://www.linkedin.com/company/bosun-ai\n[Crate Badge]: https://img.shields.io/crates/v/swiftide?logo=rust\u0026style=flat-square\u0026logoColor=E05D44\u0026color=E05D44\n[Crate]: https://crates.io/crates/swiftide\n[Docs Badge]: https://img.shields.io/docsrs/swiftide?logo=rust\u0026style=flat-square\u0026logoColor=E05D44\n[API Docs]: https://docs.rs/swiftide\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbosun-ai%2Fswiftide","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbosun-ai%2Fswiftide","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbosun-ai%2Fswiftide/lists"}