{"id":28225808,"url":"https://github.com/jisnap/rag_agent_for_streaming_data","last_synced_at":"2025-06-13T00:31:37.125Z","repository":{"id":292355846,"uuid":"980632357","full_name":"JisnaP/Rag_Agent_for_Streaming_Data","owner":"JisnaP","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-12T10:50:42.000Z","size":1227,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-18T11:09:40.373Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JisnaP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-09T12:59:16.000Z","updated_at":"2025-05-12T10:50:46.000Z","dependencies_parsed_at":"2025-05-09T14:41:58.228Z","dependency_job_id":null,"html_url":"https://github.com/JisnaP/Rag_Agent_for_Streaming_Data","commit_stats":null,"previous_names":["jisnap/rag_agent_for_streaming_data"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/JisnaP/Rag_Agent_for_Streaming_Data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JisnaP%2FRag_Agent_for_Streaming_Data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JisnaP%2FRag_Agent_for_Streaming_Data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JisnaP%2FRag_Agent_for_Streaming_Data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JisnaP%2FRag_Agent_for_Streaming_Data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JisnaP","download_url":"https://codeload.github.com/JisnaP/Rag_Agent_for_Streaming_Data/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JisnaP%2FRag_Agent_for_Streaming_Data/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259554922,"owners_count":22875896,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-18T11:09:39.611Z","updated_at":"2025-06-13T00:31:37.106Z","avatar_url":"https://github.com/JisnaP.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"**Project Title**\n## RAG Agentic System with FastAPI, LangChain \u0026 Chroma\n\n## Table of Contents\n1. Description\n\n2. Features\n\n3. Prerequisites\n\n4. Installation\n\n5. Configuration\n\n6. Data Pipeline\n\n7. Agent Server (FastAPI)\n\n8. Usage\n\n9. Project Structure\n\n10. License\n\n**Description**\nThis project implements a Retrieval-Augmented Generation (RAG) agent using:\n\nLangChain for orchestration and tool calling\n\nChroma vector store for document embeddings and similarity search\n\nSentence-Transformer embeddings (all-MiniLM-L6-v2) for semantic search\n\nFastAPI to expose a streaming chat UI via Server-Sent Events (SSE) and a JSON API\n\nIt ingests daily Federal Register documents into a vector DB and serves them via an agent that can answer user queries in real time. \n\n\n## Features\nAutomated ingestion of regulatory documents from the Federal Register\n\nChunking and embedding of text for efficient semantic search\n\nAgentic reasoning with tool calls (RAG) and memory support\n\nFastAPI UI with live streaming chat (SSE)\n\nModular codebase for easy extension\n\n## Prerequisites\nPython 3.10\n\n\n## Installation\n\n```bash\n# Clone the repository\ngit clone https://github.com/JisnaP/Rag_Agent_for_Streaming_Data.git\n\ncd Rag_Agent_for_Streaming_Data\n\n# Create and activate a virtual environment\nconda create -p ./venv python=3.10 -y\nconda activate ./venv\n\n# Install dependencies\npip install -r requirements.txt\n\n```\n\n##  Configuration\n\n\n```dotenv\nOPENAI_KEY=your_openai_api_key\n\n```\n## Data Pipeline\n\n\nThe data_pipeline.py script:\n\n1. Fetches documents from https://www.federalregister.gov/api/v1/documents.json\n\n2. Processes and splits them into 1 000-character chunks\n\n3. Embeds with SentenceTransformerEmbeddings\n\n4. Persists into a Chroma DB at ./chroma_db\n\nTo ingest streaming data run with:\n\n```python \npython scheduler.py\n```\n\nAfter successful ingestion you should see logs like:\n\n\n```cpp\nLength of all_splits :248\nIngestion complete and vector store persisted.\n```\n\n## Agent Server (FastAPI)\n\n\nThe app.py exposes:\n\n**GET / **: HTML chat UI\n\n**GET /api/query?message=… **: SSE endpoint for streaming agent replies\n\n\n\n\n```python\nuvicorn app:app --host 127.0.0.1 --port 8000\n````\n\nOpen your browser at http://127.0.0.1:8000/.\n\n## Project Structure \n\n```csharp \n├── agent.py           # LangGraph agent setup\n├── app.py             # FastAPI server\n├── data_pipeline.py   # Ingestion \u0026 embedding\n├── chroma_db/         # Persisted vector store\n├── templates/\n│   └── index.html     # Chat UI\n├── static/\n│   └── app.js         # Front-end logic\n├── requirements.txt   # Python dependencies\n└── README.md          # README file\n```\n\n## Lisence \nMIT Lisence\n\n## Example Response \n\n![RAG_AGENT_RESPONSE](https://github.com/JisnaP/Rag_Agent_for_Streaming_Data/blob/main/app_response.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjisnap%2Frag_agent_for_streaming_data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjisnap%2Frag_agent_for_streaming_data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjisnap%2Frag_agent_for_streaming_data/lists"}