https://github.com/conduitio/conduit-ai-pipelines
Real-world examples of AI-powered data pipelines built with Conduit
https://github.com/conduitio/conduit-ai-pipelines
Last synced: 5 months ago
JSON representation
Real-world examples of AI-powered data pipelines built with Conduit
- Host: GitHub
- URL: https://github.com/conduitio/conduit-ai-pipelines
- Owner: ConduitIO
- License: apache-2.0
- Created: 2025-06-04T03:27:05.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-06-23T15:49:00.000Z (about 1 year ago)
- Last Synced: 2025-06-23T16:58:33.603Z (about 1 year ago)
- Size: 14.9 MB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# Conduit AI Pipelines
Real-world examples of AI-powered data pipelines built with [Conduit](https://conduit.io) - demonstrating how to integrate modern AI services into production data workflows.
## Overview
This repository showcases practical AI pipeline implementations using Conduit's data streaming capabilities. Each example demonstrates different aspects of building intelligent, real-time data processing systems that integrate with popular AI services like OpenAI, cloud storage, and vector databases.
## Examples
### [AI Ticket Summarizer](./examples/summarize/)
**Real-time customer support automation**
- **Source**: PostgreSQL (support tickets)
- **AI Processing**: OpenAI GPT-4 summarization
- **Destination**: Slack webhooks
- **Use Case**: Automatically summarize and notify teams of new support tickets
### [AI Sentiment Analysis](./examples/sentiment-analysis/)
**Real-time customer feedback monitoring**
- **Source**: PostgreSQL (customer reviews)
- **AI Processing**: OpenAI GPT-4 sentiment classification
- **Destination**: Slack webhooks
- **Use Case**: Automatically analyze and notify teams of customer review sentiment
### [RAG Knowledge System (OpenAI)](./examples/rag-openai/)
**Real-time ingest document processing and intelligent search**
- **Source**: AWS S3 (documents)
- **AI Processing**: Document parsing + OpenAI embeddings + vector search
- **Destination**: PostgreSQL with pgvector
- **Use Case**: Build searchable knowledge bases from document collections
- **Includes**: Document parsing service + RAG query API
### [RAG Knowledge System (Ollama)](./examples/rag-ollama/)
**Real-time ingest document processing and chatbot UI running locally**
- **Source**: AWS S3 (documents)
- **AI Processing**: Document parsing + Ollama embeddings + vector search
- **Destination**: Supabase
- **Use Case**: Build a chatbot UI with knowledge bases from document collections using local and open source software
- **Includes**: Document parsing service
## Quick Start
1. **Install Conduit**
```bash
# Download latest release
curl -sSL https://get.conduit.io | sh
# Or use Homebrew
brew install conduit
```
2. **Choose an Example**
```bash
cd examples/summarize
```
3. **Follow Setup Instructions**
Each example includes detailed setup instructions and environment configuration in the respective README
## Prerequisites
- [Conduit](https://conduit.io/docs/getting-started) (latest version)
- OpenAI API key for AI processing
- Database access (PostgreSQL recommended)
- Cloud storage credentials (AWS S3 for RAG example)
## Features Demonstrated
- **AI Integration**: OpenAI GPT models, embeddings, and text processing
- **Real-time Processing**: Stream processing with immediate AI-powered transformations
- **Vector Databases**: pgvector integration for similarity search and retrieval
- **API Orchestration**: HTTP processors for external service integration
- **Data Transformation**: Complex multi-step processing workflows
- **Custom Processing**: JavaScript processors for domain-specific logic
- **Multiple Connectors**: S3, PostgreSQL, HTTP webhooks, and more
## Getting Help
- **Documentation**: [conduit.io/docs](https://conduit.io/docs)
- **Community**: [Discord](https://discord.meroxa.com)
- **Issues**: [GitHub Issues](https://github.com/ConduitIO/conduit/issues)
- **API Reference**: [docs.conduit.io/api](https://docs.conduit.io/api)
## Related Projects
- **[Conduit](https://github.com/ConduitIO/conduit)** - The core data streaming platform
- **[Conduit Connectors](https://conduit.io/docs/using/connectors/list)** - Full list of available connectors
- **[Conduit Processors](https://conduit.io/docs/using/processors/builtin/)** - Built-in data transformation capabilities
## License
This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.