{"id":29192262,"url":"https://github.com/conduitio/conduit-ai-pipelines","last_synced_at":"2026-02-02T08:31:48.899Z","repository":{"id":300791005,"uuid":"995809222","full_name":"ConduitIO/conduit-ai-pipelines","owner":"ConduitIO","description":"Real-world examples of AI-powered data pipelines built with Conduit","archived":false,"fork":false,"pushed_at":"2025-06-23T15:49:00.000Z","size":15591,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-23T16:58:33.603Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ConduitIO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-04T03:27:05.000Z","updated_at":"2025-06-23T15:49:01.000Z","dependencies_parsed_at":"2025-06-23T17:10:18.632Z","dependency_job_id":null,"html_url":"https://github.com/ConduitIO/conduit-ai-pipelines","commit_stats":null,"previous_names":["conduitio/conduit-ai-pipelines"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ConduitIO/conduit-ai-pipelines","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-ai-pipelines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-ai-pipelines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-ai-pipelines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-ai-pipelines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ConduitIO","download_url":"https://codeload.github.com/ConduitIO/conduit-ai-pipelines/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConduitIO%2Fconduit-ai-pipelines/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263056763,"owners_count":23406818,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-02T01:05:30.409Z","updated_at":"2026-02-02T08:31:43.880Z","avatar_url":"https://github.com/ConduitIO.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Conduit AI Pipelines\n\nReal-world examples of AI-powered data pipelines built with [Conduit](https://conduit.io) - demonstrating how to integrate modern AI services into production data workflows.\n\n## Overview\n\nThis repository showcases practical AI pipeline implementations using Conduit's data streaming capabilities. Each example demonstrates different aspects of building intelligent, real-time data processing systems that integrate with popular AI services like OpenAI, cloud storage, and vector databases.\n\n## Examples\n\n### [AI Ticket Summarizer](./examples/summarize/)\n**Real-time customer support automation**\n- **Source**: PostgreSQL (support tickets)\n- **AI Processing**: OpenAI GPT-4 summarization\n- **Destination**: Slack webhooks\n- **Use Case**: Automatically summarize and notify teams of new support tickets\n\n### [AI Sentiment Analysis](./examples/sentiment-analysis/)\n**Real-time customer feedback monitoring**\n- **Source**: PostgreSQL (customer reviews)\n- **AI Processing**: OpenAI GPT-4 sentiment classification\n- **Destination**: Slack webhooks\n- **Use Case**: Automatically analyze and notify teams of customer review sentiment\n\n### [RAG Knowledge System (OpenAI)](./examples/rag-openai/)\n**Real-time ingest document processing and intelligent search**\n- **Source**: AWS S3 (documents)\n- **AI Processing**: Document parsing + OpenAI embeddings + vector search\n- **Destination**: PostgreSQL with pgvector\n- **Use Case**: Build searchable knowledge bases from document collections\n- **Includes**: Document parsing service + RAG query API\n\n### [RAG Knowledge System (Ollama)](./examples/rag-ollama/)\n**Real-time ingest document processing and chatbot UI running locally**\n- **Source**: AWS S3 (documents)\n- **AI Processing**: Document parsing + Ollama embeddings + vector search\n- **Destination**: Supabase\n- **Use Case**: Build a chatbot UI with knowledge bases from document collections using local and open source software\n- **Includes**: Document parsing service\n\n## Quick Start\n\n1. **Install Conduit**\n   ```bash\n   # Download latest release\n   curl -sSL https://get.conduit.io | sh\n   \n   # Or use Homebrew\n   brew install conduit\n   ```\n\n2. **Choose an Example**\n   ```bash\n   cd examples/summarize \n   ```\n\n3. **Follow Setup Instructions**\n   Each example includes detailed setup instructions and environment configuration in the respective README\n\n## Prerequisites\n\n- [Conduit](https://conduit.io/docs/getting-started) (latest version)\n- OpenAI API key for AI processing\n- Database access (PostgreSQL recommended)\n- Cloud storage credentials (AWS S3 for RAG example)\n\n## Features Demonstrated\n\n- **AI Integration**: OpenAI GPT models, embeddings, and text processing\n- **Real-time Processing**: Stream processing with immediate AI-powered transformations\n- **Vector Databases**: pgvector integration for similarity search and retrieval\n- **API Orchestration**: HTTP processors for external service integration\n- **Data Transformation**: Complex multi-step processing workflows\n- **Custom Processing**: JavaScript processors for domain-specific logic\n- **Multiple Connectors**: S3, PostgreSQL, HTTP webhooks, and more\n\n## Getting Help\n\n- **Documentation**: [conduit.io/docs](https://conduit.io/docs)\n- **Community**: [Discord](https://discord.meroxa.com)\n- **Issues**: [GitHub Issues](https://github.com/ConduitIO/conduit/issues)\n- **API Reference**: [docs.conduit.io/api](https://docs.conduit.io/api)\n\n## Related Projects\n\n- **[Conduit](https://github.com/ConduitIO/conduit)** - The core data streaming platform\n- **[Conduit Connectors](https://conduit.io/docs/using/connectors/list)** - Full list of available connectors\n- **[Conduit Processors](https://conduit.io/docs/using/processors/builtin/)** - Built-in data transformation capabilities\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduitio%2Fconduit-ai-pipelines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconduitio%2Fconduit-ai-pipelines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduitio%2Fconduit-ai-pipelines/lists"}