{"id":21693804,"url":"https://github.com/spandan114/luminai-data-analyst","last_synced_at":"2025-07-30T23:18:49.725Z","repository":{"id":262186852,"uuid":"868552982","full_name":"spandan114/LuminAI-Data-Analyst","owner":"spandan114","description":"LUMIN: Your data analysis companion that turns natural language questions into powerful insights through AI-driven visualizations and clear explanations.","archived":false,"fork":false,"pushed_at":"2024-11-11T03:21:21.000Z","size":64925,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-26T05:41:46.264Z","etag":null,"topics":["ai-agents","ai-data-analysis","ai-tools","chatgpt","data-analytics","fastapi","groq","langchain","llm","react","sql","typescript"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spandan114.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-06T17:13:46.000Z","updated_at":"2025-03-16T05:52:22.000Z","dependencies_parsed_at":"2024-11-11T04:25:49.964Z","dependency_job_id":"4cbfd1f0-ccd1-4cbc-861c-ec094df9b4fe","html_url":"https://github.com/spandan114/LuminAI-Data-Analyst","commit_stats":null,"previous_names":["spandan114/luminai-data-analyst"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spandan114%2FLuminAI-Data-Analyst","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spandan114%2FLuminAI-Data-Analyst/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spandan114%2FLuminAI-Data-Analyst/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spandan114%2FLuminAI-Data-Analyst/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spandan114","download_url":"https://codeload.github.com/spandan114/LuminAI-Data-Analyst/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248557753,"owners_count":21124161,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","ai-data-analysis","ai-tools","chatgpt","data-analytics","fastapi","groq","langchain","llm","react","sql","typescript"],"created_at":"2024-11-25T18:22:38.372Z","updated_at":"2025-04-12T10:43:26.628Z","avatar_url":"https://github.com/spandan114.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ch1\u003e\n    \u003cb\u003eLUMIN\u003c/b\u003e\n  \u003c/h1\u003e\n\u003c/div\u003e\n\nLUMIN is an intelligent data analysis platform that transforms how you interact with your data. Using LLM, LUMIN enables you to ask analytical questions about your data in plain English and receive insights through beautiful visualizations and clear explanations.\n\n## 🎥 Demo \n\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://www.youtube.com/watch?v=jR0rGJOhxIw\"\u003e\n    \u003cimg src=\"https://img.youtube.com/vi/jR0rGJOhxIw/maxresdefault.jpg\" alt=\"Video Demo\" style=\"width:100%;\"\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n**Youtube Video URL :** https://www.youtube.com/watch?v=jR0rGJOhxIw\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- Docker \u0026 Docker Compose\n- Git\n\n### Clone Project\n\n```bash\n# Clone the repository\ngit clone https://github.com/spandan114/LuminAI-Data-Analyst.git\ncd lumin_ai\n\n```\n### 🔐 Environment Setup\n\n1. Navigate to the backend directory and create your environment file:\n```bash\ncd backend\ncp .env.example .env\n```\n\n2. Configure the following environment variables in your `.env` file:\n\n| Variable | Description | Example |\n|----------|-------------|---------|\n| `OPENAI_API_KEY` | Your OpenAI API key for ChatGPT integration | \"sk-...\" |\n| `GROQ_API_KEY` | Your Groq API key for Groq LLM integration | \"gsk-...\" |\n| `SECRET_KEY` | Secret key for JWT token encryption | \"your-secret-key\" |\n| `DATABASE_URL` | PostgreSQL connection URL | \"postgresql://lumin:root@db:5432/lumin\" |\n| `LANGCHAIN_PROJECT` | Project name for LangChain tracking (optional) | \"lumin\" |\n| `HF_TOKEN` | Hugging Face API token for model access | \"hf_...\" |\n\n### Notes:\n- For local development using Docker, keep the `DATABASE_URL` as is - Docker Compose will handle the connection\n- The project uses Groq as the primary LLM provider - a Groq API key is required for full functionality\n- `SECRET_KEY` should be a secure random string in production\n- While the codebase supports OpenAI and Hugging Face as alternative LLM providers, they are optional, you can configure the methods to use different llm provider\n- Default database credentials can be modified in the `docker-compose.yml` file\n\n### Getting API Keys:\n- OpenAI API: https://platform.openai.com/\n- Groq API: https://console.groq.com/\n- Hugging Face: https://huggingface.co/settings/tokens\n\n### Start Container\n```bash\n# Start the containers\ndocker compose up --build\n```\n\n## 🔌 Local Development URLs\n\nAfter starting the container, you can access:\n\n| Service | URL | Description |\n|---------|-----|-------------|\n| Frontend | `http://localhost:3000` | React application interface |\n| Backend | `http://localhost:8000` | FastAPI server |\n| API Docs | `http://localhost:8000/docs` | Swagger API documentation |\n\n### Remove Container\n\n```bash\n# Stop and remove containers\ndocker compose down\n```\n\n## ⚡ Features\n\n- 📂 Universal Data Connection: Seamlessly connect with multiple data sources:\n\n    - CSV and Excel files\n    - SQL Databases\n    - PDF Documents (API Not integrated yet)\n    - Text Files (API Not integrated yet)\n\n- 🧠 Multiple LLM Support: Choose your preferred AI engine:\n\n    - OpenAI (ChatGPT)\n    - Groq\n    - Hugging Face Models\n    - Ollama (Self-hosted)\n    - Easy to extend with new LLM providers\n\n- 🤖 Natural Language Processing: Ask questions in plain English about your data\n- 💾 Database Support:\n    - Full support for tabular databases\n    - NoSQL databases not currently supported\n- 📊 Smart Visualizations: Automatically generates relevant charts and graphs\n\n- 🔍 Intelligent Analysis: Provides deep insights and patterns in your data\n\n\n\n## 🛠️ Tech Stack\n\n### Frontend Modules\n\n| Module | Description |\n|--------|-------------|\n| `@tanstack/react-query` | Powerful data synchronization for React |\n| `chart.js` \u0026 `react-chartjs-2` | Rich data visualization library with React components |\n| `react-hook-form` | Performant forms with easy validation |\n| `react-router-dom` | Declarative routing for React applications |\n| `react-toastify` | Toast notifications made easy |\n| `recharts` | Composable charting library for React |\n| `zustand` | Lightweight state management solution |\n| `prismjs` | Syntax highlighting for code blocks |\n| `axios` | Promise-based HTTP client |\n\n### Backend Modules\n\n| Module | Description |\n|--------|-------------|\n| `fastapi` | Modern, fast web framework for building APIs |\n| `langchain` | Framework for developing LLM powered applications |\n| `langgraph` | State management for LLM application workflows |\n| `langchain-openai` | OpenAI integration for LangChain |\n| `sqlalchemy` | SQL toolkit and ORM |\n| `pgvector` | Vector similarity search for PostgreSQL |\n| `pydantic` | Data validation using Python type annotations |\n| `alembic` | Database migration tool |\n| `pandas` | Data manipulation and analysis library |\n| `passlib` | Password hashing library |\n| `python-multipart` | Streaming multipart parser for Python |\n\n### Development Tools\n\n| Tool | Purpose |\n|------|---------|\n| `vite` | Next generation frontend tooling |\n| `typescript` | JavaScript with syntax for types |\n| `tailwindcss` | Utility-first CSS framework |\n| `eslint` \u0026 `prettier` | Code linting and formatting |\n| `autopep8` | Python code formatter |\n\n\n## 🔄 Workflow Architecture\n\n### High level flow\n```mermaid\nflowchart TD\n    Start([Start]) --\u003e InputDoc{Document Type?}\n    \n    %% Document Processing Branch\n    InputDoc --\u003e|CSV/Excel| DB[(Database)]\n    InputDoc --\u003e|PDF/Text| VEC[(pgvector DB)]\n    InputDoc --\u003e|SQL Connection| DBTable[(DB Table)]\n    \n    %% Data Source Selection\n    DB --\u003e DataSelect{Data Source?}\n    VEC --\u003e DataSelect\n    DBTable --\u003e DataSelect\n    \n    %% Query Processing\n    DataSelect --\u003e|CSV/Excel/DB Link| QueryDB[Query Database]\n    DataSelect --\u003e|PDF/Text| QueryVec[Query Vector Database]\n    \n    QueryDB --\u003e Process[Process Data]\n    QueryVec --\u003e Process\n    \n    %% Question Processing Pipeline\n    Process --\u003e Questions[Get User Questions]\n    Questions --\u003e ParseQuestions[Parse Questions \u0026 Get Relevant Tables/Columns]\n    ParseQuestions --\u003e \n    \n    %% SQL Validation and Execution\n    GenSQL --\u003e ValidateSQL{Validate SQL Query}\n    ValidateSQL --\u003e|Need Fix| GenSQL\n    ValidateSQL --\u003e|Valid| ExecuteSQL[Execute SQL]\n    \n    %% Result Processing\n    ExecuteSQL --\u003e CheckResult{Check Results}\n    CheckResult --\u003e|No Error \u0026 Relevant| ChooseViz[Choose Visualization]\n    ChooseViz --\u003e FormatViz[Format Data for Visualization]\n    CheckResult --\u003e|Error or Not Relevant| FormatResult[Format Result]\n    \n    %% End States\n    FormatViz --\u003e End([End])\n    FormatResult --\u003e End\n    \n    %% Styling\n    classDef database fill:#f9f,stroke:#333,stroke-width:2px\n    class DB,VEC,DBTable database\n\n```\n\n### Lang graph flow\n\n```mermaid\nflowchart TD\n    Start([START]) --\u003e ParseQuestion[Parse Question]\n    \n    ParseQuestion --\u003e ShouldContinue{Should Continue?}\n    \n    ShouldContinue --\u003e|Yes| GenSQL[Generate SQL Query]\n    GenSQL --\u003e ValidateSQL[Validate and Fix SQL]\n    ValidateSQL --\u003e ExecuteSQL[Execute SQL Query]\n    \n    ExecuteSQL --\u003e FormatResults[Format Results]\n    ExecuteSQL --\u003e ChooseViz[Choose Visualization]\n    \n    ChooseViz --\u003e FormatViz[Format Data for Visualization]\n    \n    ShouldContinue --\u003e|No| ConvResponse[Conversational Response]\n    \n    FormatResults --\u003e End([END])\n    FormatViz --\u003e End\n    ConvResponse --\u003e End\n\n    classDef conditional fill:#f9f,stroke:#333,stroke-width:2px\n    classDef process fill:#bbf,stroke:#333,stroke-width:1px\n    class ShouldContinue conditional\n    class ParseQuestion,GenSQL,ValidateSQL,ExecuteSQL,FormatResults,ChooseViz,FormatViz,ConvResponse process\n```\n\n## Database Schema \n```mermaid\nerDiagram\n    users ||--o{ data_sources : creates\n    users ||--o{ conversations : has\n    data_sources ||--o{ conversations : used_in\n    conversations ||--o{ messages : contains\n\n    users {\n        int id PK\n        string name\n        string email UK\n        string hashed_password UK\n        datetime created_at\n    }\n\n    data_sources {\n        int id PK\n        int user_id FK\n        string name\n        string type\n        string table_name UK\n        string connection_url UK\n        datetime created_at\n    }\n\n    conversations {\n        int id PK\n        int user_id FK\n        int data_source_id FK\n        string title\n        datetime created_at\n        datetime updated_at\n    }\n\n    messages {\n        int id PK\n        int conversation_id FK\n        enum role\n        json content\n        datetime created_at\n        datetime updated_at\n    }\n```\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature_name`)\n3. Commit your changes (`git commit -m 'Add some comment'`)\n4. Push to the branch (`git push origin feature_name`)\n5. Open a Pull Request\n\n### Features You Can Contribute\nWe welcome contributions! Here are some exciting features you can help implement:\n\n**💭 Contextual Chat Enhancement:**\n*Status:* Needs Implementation\n\n- Implement context retrieval system\n- Integrate pgvector for similarity search\n- Add relevance scoring for context selection\n- Create context window management\n- Add context visualization for users\n\n**📑 Document Analysis Integration:**\n*Status:* Backend Ready, Needs Frontend Implementation\n- Add functionality to upload PDF or Text document\n- Integrate PDF and Text file analysis in the frontend\n\n**💾 Implement NoSQL Database Support**\n*Status:* Needs Implementation\n- Add MongoDB integration for for analysis\n- Implement schema-less data handling\n- Add support for nested JSON structures\n\n**⚙️ User Settings Dashboard:**\n*Status:* Needs Implementation\n\n- Profile management interface\n- Password change workflow with validation\n- Email update with verification\n- LLM platform selection with configuration\n- Model selection based on chosen platform\n\n## Testing Dataset\n\nIn the project Demo i used the [Brazilian E-commerce Public Dataset by Olist](https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce), available on Kaggle. The dataset includes information about:\n\n- Customer and orders\n- Order items and payments\n- Product details\n- Seller information\n- Geolocation data\n- Order reviews\n- Product category translations\n\n### Data Loading\n\nThe following code demonstrates how to load the Olist dataset into a SQLite database:\n\n1. Download the dataset from [Kaggle](https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce)\n2. Extract the files to an `/ecommerce` directory\n3. Run the data loading script to create and populate the SQLite database\n\n```python\nimport os\nimport pandas as pd\nfrom sqlalchemy import create_engine\n\ndef insert_data_to_sqlite(file_path):\n    # Extract the file name without extension to use as table name\n    file_name = os.path.splitext(os.path.basename(file_path))[0]\n\n    # Read the data (change this to pd.read_excel() for Excel files)\n    data = pd.read_csv(file_path)\n\n    # Create a SQLite database (or connect if it already exists)\n    engine = create_engine('sqlite:///lumin.db')\n\n    # Insert data into the SQLite database with the table name as the file name\n    data.to_sql(file_name, con=engine, if_exists='replace', index=False)\n    print(\n        f\"Data from {file_path} has been inserted into the '{file_name}' table in the 'lumin.db' database.\")\n\n# List of dataset files\necom_data = [\n    \"olist_customers_dataset.csv\",\n    \"olist_geolocation_dataset.csv\",\n    \"olist_order_items_dataset.csv\",\n    \"olist_order_payments_dataset.csv\",\n    \"olist_order_reviews_dataset.csv\",\n    \"olist_orders_dataset.csv\",\n    \"olist_products_dataset.csv\",\n    \"olist_sellers_dataset.csv\",\n    \"product_category_name_translation.csv\"\n]\n\n# Load each dataset\nfor data in ecom_data:\n    path = (f\"/ecommerce/{data}\")\n    file_data = os.path.abspath(path)\n    insert_data_to_sqlite(file_data)\n    print(file_data)\n```\n\n## Multiple LLM provider setup\n\nThe project supports multiple LLM providers through a flexible switching mechanism:\n\n```python\nfrom langchain_groq import ChatGroq\nfrom langchain_openai import OpenAI\nfrom langchain_ollama.llms import OllamaLLM\nfrom app.config.env import (GROQ_API_KEY, OPENAI_API_KEY)\n\nclass LLM:\n    def __init__(self):\n        self.llm = None\n        self.platform = None\n\n    def groq(self, model: str):\n        self.llm = ChatGroq(groq_api_key=GROQ_API_KEY, model=model)\n        self.platform = \"Groq\"\n        return self.llm\n\n    def openai(self, model: str):\n        self.llm = OpenAI(api_key=OPENAI_API_KEY, model=model)\n        self.platform = \"OpenAi\"\n        return self.llm\n\n    def ollama(self, model: str):\n        self.llm = OllamaLLM(model=model)\n        self.platform = \"Ollama\"\n        return self.llm\n\n    def get_llm(self):\n        return self.llm\n\n    def invoke(self, prompt: str):\n        return self.llm.invoke(prompt)\n```\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspandan114%2Fluminai-data-analyst","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspandan114%2Fluminai-data-analyst","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspandan114%2Fluminai-data-analyst/lists"}