{"id":23221046,"url":"https://github.com/spenceypantsy1/rag-sql-langchain-chat-bot","last_synced_at":"2026-02-12T07:31:24.257Z","repository":{"id":265176507,"uuid":"895322760","full_name":"spenceypantsy1/rag-sql-langchain-chat-bot","owner":"spenceypantsy1","description":null,"archived":false,"fork":false,"pushed_at":"2024-12-18T15:59:06.000Z","size":6798,"stargazers_count":1,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-07T05:45:31.496Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spenceypantsy1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-11-28T01:46:31.000Z","updated_at":"2025-03-07T05:52:59.000Z","dependencies_parsed_at":"2024-11-28T05:19:56.020Z","dependency_job_id":"51be9ee9-bf56-43fa-b9e4-bc01423cc94c","html_url":"https://github.com/spenceypantsy1/rag-sql-langchain-chat-bot","commit_stats":null,"previous_names":["spenceypantsy1/dsa-sim-group-6-internal-project","spenceypantsy1/rag-sql-langchain-chat-bot"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/spenceypantsy1/rag-sql-langchain-chat-bot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spenceypantsy1%2Frag-sql-langchain-chat-bot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spenceypantsy1%2Frag-sql-langchain-chat-bot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spenceypantsy1%2Frag-sql-langchain-chat-bot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spenceypantsy1%2Frag-sql-langchain-chat-bot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spenceypantsy1","download_url":"https://codeload.github.com/spenceypantsy1/rag-sql-langchain-chat-bot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spenceypantsy1%2Frag-sql-langchain-chat-bot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29361447,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-12T01:03:07.613Z","status":"online","status_checked_at":"2026-02-12T02:00:06.911Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-18T22:14:58.556Z","updated_at":"2026-02-12T07:31:24.243Z","avatar_url":"https://github.com/spenceypantsy1.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SQL Data Exploration with Langchain and RAG 💬\n\nThis project involves three main parts:\n1. **Database creation and EDA**: The first notebook(`eda-data-exploration.ipynb`) cleans the dataset and combines. both authors.csv \u0026 papers.csv into a single table. Exploratory Data Analysis is done here to understand the dataset better.\n2. **NLP hot topics generation**: The second notebook (`nlp-hot-topics-generation.ipynb`) creates word clouds and finds hot topics for each year with TF-IDF NLP to upload into the SQLite database.\n3. **RAG SQL Langchain Chatbot**: The third notebook (`rags-sql-langchain-chat-bot.ipynb`) leverages the database and Langchain to create an interactive chatbot for SQL-based queries.\n\nThe dataset used comes from https://www.kaggle.com/datasets/rowhitswami/nips-papers-1987-2019-updated/data\n\n## Overview\n\nThis was a group effort as part of our SIM Data Analytics Club - Data Science Academy internal projects.\n\nThe system uses **Langchain** for structured query generation, **RAG** (Retrieval-Augmented Generation) to retrieve relevant documents from the database, and **OpenAI's GPT** to generate responses based on SQL query results. The two notebooks must be run in order to set up and utilize the database and chatbot functionality.\n\n---\n\n## Prerequisites\n\n1. **Python 3.x** (Recommended: Python 3.7+ 🐍)\n2. **Install required Python packages**: You'll need to install the necessary dependencies from the `requirements.txt` file.\n\nTo install the required packages, use the following command:\n\n```bash\npip install -r requirements.txt bash\n```\n3. OpenAI API Key: You'll need to set up your OpenAI API key to interact with GPT models. You can get your API key from OpenAI's API platform 🌐.\n\nCreate a .env file in the root of the project and add your OpenAI API key as follows:\n\n```bash\nOPENAI_API_KEY=your_openai_api_key\n```\n\n4. SQLite Database: The first notebook creates the SQLite database (main.db) that will be used in the second notebook. This file must exist before running the chatbot notebook 🗃️.\n\n# Steps\n\n### Step 1: Run the Database Creation ane EDA Notebook\n\nThe first notebook, `eda-data-exploration.ipynb`, generates the SQLite database (`main.db`) and gives us an understanding on the intricacies of the data.\n\n**To run the database creation notebook:**\n\n1. Open Jupyter Notebook or JupyterLab 🖥️.\n2. Open the `eda-data-exploration.ipynb` notebook.\n3. Run the cells sequentially to:\n   - Create the SQLite database (`main.db`) 🗃️.\n   - Clean the data, removing NA fields and combining both data files into a single table.\n   - Plot the emerging trends in NIPS research papers.\n\nOnce the notebook finishes, the `main.db` database and word clouds will be ready 🎉.\n\n---\n\n### Step 2: Run the Database Creation Notebook\n\nThe second notebook, `sql-data-exploration-database-creation.ipynb`, engaging NLP techniques like TF-IDF to hot-topic trends and updates our SQLite database.\n\n**To run the database creation notebook:**\n\n1. Open Jupyter Notebook or JupyterLab 🖥️.\n2. Open the `sql-data-exploration-database-creation.ipynb` notebook.\n3. Run the cells sequentially to:\n   - Populate the database with your data (you can modify the notebook to customize the data source).\n   - Generate word clouds based on the content of the database to visualize the data.\n\n---\n\n### Step 3: Run the RAG SQL Langchain Chatbot Notebook\n\nOnce the database is created, the second notebook, `rags-sql-langchain-chat-bot.ipynb`, leverages Langchain to build a chatbot capable of interacting with the SQLite database using natural language queries 💬.\n\n**To run the chatbot notebook:**\n\n1. Open Jupyter Notebook or JupyterLab 🖥️.\n2. Open the `rags-sql-langchain-chat-bot.ipynb` notebook.\n3. Run the cells sequentially to:\n   - Use Langchain to generate SQL queries based on user input 📝.\n   - Execute the generated SQL queries on the `main.db` database 🗃️.\n   - Use OpenAI's GPT to generate human-readable answers based on the query results 💬.\n   - Present the results interactively, powered by Langchain and RAG 🔄.\n\n---\n\n# Example Usage\n\nOnce the chatbot is running, you can interact with it via the notebook interface. Example interactions:\n\n- **User**: \"How many customers purchased product X last month?\"\n  \n  **Response**: The chatbot will generate an SQL query, execute it on the `main.db`, and return a response like: \"Product X was purchased by 150 customers last month.\"\n\n- **User**: \"Show me the top 5 products sold in the last quarter.\"\n  \n  **Response**: The chatbot will execute a query to fetch the top 5 products from the database and display them.\n\n---\n\n# Future Improvements 🛠️\n\n- **Support for Additional Databases**: Enhance the tool to support other databases like MySQL, PostgreSQL, etc.\n- **Error Handling**: Improve error handling for invalid SQL queries or database issues.\n- **User Interface**: Consider adding a web interface with Gradio or a more interactive UI for ease of use.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspenceypantsy1%2Frag-sql-langchain-chat-bot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspenceypantsy1%2Frag-sql-langchain-chat-bot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspenceypantsy1%2Frag-sql-langchain-chat-bot/lists"}