{"id":24147500,"url":"https://github.com/smpy2002/csv_data-info-retrieval-by-ai","last_synced_at":"2026-07-02T09:32:45.770Z","repository":{"id":262186086,"uuid":"886058182","full_name":"SMPY2002/CSV_Data-Info-Retrieval-by-AI","owner":"SMPY2002","description":"Automated Web Search and Data Retrieval System (Visit the below URL to see)","archived":false,"fork":false,"pushed_at":"2025-01-11T06:03:48.000Z","size":28964,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-01T16:13:16.325Z","etag":null,"topics":["google-auth","groq-api","python","serp-api","streamlit-dashboard"],"latest_commit_sha":null,"homepage":"https://ai-info-retrieval.streamlit.app/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SMPY2002.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-10T04:31:19.000Z","updated_at":"2025-01-11T06:03:51.000Z","dependencies_parsed_at":"2024-11-11T04:20:50.255Z","dependency_job_id":"dfa55525-19df-4d18-b171-c92803b2f067","html_url":"https://github.com/SMPY2002/CSV_Data-Info-Retrieval-by-AI","commit_stats":null,"previous_names":["smpy2002/ai-agent-project"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SMPY2002/CSV_Data-Info-Retrieval-by-AI","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SMPY2002%2FCSV_Data-Info-Retrieval-by-AI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SMPY2002%2FCSV_Data-Info-Retrieval-by-AI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SMPY2002%2FCSV_Data-Info-Retrieval-by-AI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SMPY2002%2FCSV_Data-Info-Retrieval-by-AI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SMPY2002","download_url":"https://codeload.github.com/SMPY2002/CSV_Data-Info-Retrieval-by-AI/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SMPY2002%2FCSV_Data-Info-Retrieval-by-AI/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35041996,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-02T02:00:06.368Z","response_time":173,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["google-auth","groq-api","python","serp-api","streamlit-dashboard"],"created_at":"2025-01-12T07:15:16.022Z","updated_at":"2026-07-02T09:32:45.750Z","avatar_url":"https://github.com/SMPY2002.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# **Automated Web Search and Data Retrieval System**\n\n## **Overview**\nThis project automates information retrieval from the web for entities listed in a dataset. Designed with simplicity and efficiency in mind, it provides a user-friendly interface to upload datasets or connect to a Google Sheet, define custom prompts, and retrieve structured results dynamically.\n\nThe project is built using cutting-edge technologies like Groq API for natural language understanding, SerpAPI for web searches, and Streamlit for the user interface.\n\n---\n\n**Loom-Video:-**  A short video walkthrough of the Project.\nhttps://drive.google.com/file/d/1y00rkDAoB-2vFliS5QKaN2I_4h6CHKKT/view?usp=drivesdk\n\n---\n\n## **Features**\n- **Data Input Options**: Upload a CSV file or connect to a Google Sheet for real-time data processing.\n- **Dynamic Querying**: Define custom search prompts using placeholders (e.g., `{company}`) to extract relevant information dynamically.\n- **Seamless Information Retrieval**: Uses Groq API and SerpAPI to perform searches and return accurate, structured data.\n- **User-Friendly Dashboard**: View extracted results in a clean tabular format and download them as a CSV file.\n- **Error Handling**: Provides user-friendly feedback for invalid inputs or configuration errors.\n\n---\n\n## **Tech Stack**\n- **Frontend/UI**: Streamlit\n- **Backend**: Python\n- **APIs**: \n  - Groq API for natural language processing.\n  - SerpAPI for performing web searches.\n  - Google Sheets API for real-time spreadsheet integration.\n\n---\n\n## **Prerequisites**\nEnsure the following are installed and configured:\n1. **Python**: Version 3.8 or higher.\n2. **API Access**: \n   - Groq API Key.\n   - SerpAPI Key.\n3. **Google Cloud Project**: \n   - Enable Google Sheets API and Google Drive API.\n   - Download the OAuth credentials JSON file.\n4. **Libraries**: Listed in the `requirements.txt` file.\n\n---\n\n## **Setup Instructions**\n\n### **Step 1: Clone the Repository**\n```bash\ngit clone https://github.com/your-username/your-repo-name.git\ncd your-repo-name\n```\n---\n\n### **Step 2: Install Dependencies**\nInstall the required Python libraries using:\n```bash\npip install -r requirements.txt\n```\n\n### **Step 3: Run the Application**\nStart the Streamlit application:\n```bash\nstreamlit run app/main_app.py\n```\n\n---\n\n## API Keys and Environment Variables\n\nTo ensure the smooth functioning of the application, you need to set up your API keys and environment variables. Follow these steps:\n\n### 1. **Locating the `.env` File**\n   - The repository contains an example `.env.example` file in the root directory.\n   - Rename this file to `.env`.\n\n### 2. **Adding API Keys**\n   - Open the `.env` file in a text editor.\n   - Add the required API keys and credentials for third-party integrations.\n   - Example:\n     ```plaintext\n     GOOGLE_SHEETS_API_KEY=\u003cYour Google Sheets API Key\u003e\n     GOOGLE_OAUTH_CLIENT_ID=\u003cYour OAuth 2.0 Client ID\u003e\n     ```\n\n### 3. **NGROK Configuration (If Applicable)**\n   - For exposing your local server to the internet using ngrok, add the ngrok token in the `.env` file.\n   - Example:\n     ```plaintext\n     NGROK_AUTH_TOKEN=\u003cYour ngrok Auth Token\u003e\n     ```\n\n### 4. **Usage**\n   - The application will automatically read the environment variables from the `.env` file during runtime.\n   - Ensure the `.env` file is stored securely and not shared or uploaded publicly to maintain the confidentiality of your API keys.\n**Note:-** Google Cloud secret file is not present in the .env file you should just download your secret_file(JSON Format) from google cloud and place in your root directory and paste it path to the code just below the .env configuration.\n\n---\n\n## Usage Guide\n\nFollow these instructions to utilize the dashboard efficiently:\n\n### 1. **Uploading CSV Files**\n   - Click the **\"Upload CSV File\"** button on the dashboard.\n   - Choose a valid CSV file from your local system.\n   - Once uploaded, the file will be processed, and a preview of the data will be displayed in an interactive table.\n\n### 2. **Connecting to Google Sheets**\n   - Click the **\"Connect Google Sheet\"** button.\n   - Authenticate with your Google account to grant access to your Google Sheets.\n   - After authentication, provide the URL of the Google Sheet you want to connect.\n   - Select the required sheet from the dropdown menu to load its data into the dashboard for processing.\n   - The app maintains a real-time connection with the Google Sheet for live updates.\n\n### 3. **Setting Up Search Queries**\n   - Use the **\"Primary Column Selection\"** dropdown to choose the column containing the entities (e.g., company names).\n   - Input your custom search prompt in the **\"Custom Prompt\"** text box.\n   - Click **\"Run Query\"** to initiate the information retrieval process.\n   - The results will appear in a separate section and can be downloaded as a CSV file.\n\n### 4. **Extracted Results**\n   - Once the query completes, view the extracted results in an interactive preview table.\n   - Download the results as a CSV file for further analysis or use.\n\n---\n\n\n## **Working Example**\n\n### **Input**\n#### Uploaded CSV:\n| Company       | Country       |\n|---------------|---------------|\n| Tesla         | USA           |\n| Samsung       | South Korea   |\n| BMW           | Germany       |\n\n#### Query:\n_\"Retrieve the latest market share data of {Company}.\"_\n\n### **Output**\n| Company       | Market Share (%) |\n|---------------|------------------|\n| Tesla         |       45%        |\n| Samsung       |       38%        |\n| BMW           |       32%        |\n\n---\n\n## **Third-Party Tools**\n\n1. **Groq API**:\n   - Used for understanding and dynamically generating search queries.\n   - [Groq API Documentation](https://groq.com/docs)\n\n2. **SerpAPI**:\n   - Enables search engine queries with structured JSON responses.\n   - [SerpAPI Documentation](https://serpapi.com/docs)\n\n3. **Google Sheets API**:\n   - Integrates live data from Google Sheets into the app.\n   - [Google Sheets API Documentation](https://developers.google.com/sheets)\n\n---\n\n## **Project Structure**\n```plaintext\nProject/\n├── app/\n│   ├── main_app.py          # Streamlit application code\n├── resources/\n│   ├── ngrok.exe/zip        # Optional HTTPS support(When run on local environment)\n├── requirements.txt         # Python dependencies\n├── README.md                # Project documentation\n├── LICENSE                  # Open-source license\n├── .gitignore               # Files to ignore in Git\n```\n---\n\n## **Contributing**\nContributions are welcome! To contribute:\n  1. Fork the repository.\n  2. Create a feature branch:\n    ```bash\n    git checkout -b feature-name\n    ```\n  3. Commit your changes:\n     ```bash\n     git commit -m \"Add feature-name\"\n     ```\n  4. Push to your branch:\n     ```bash\n     git push origin feature-name\n     ```\n  5. Open a pull request.\n\n---\n\n## **License**\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n\n---\n\n## **Contact**\n\nFor any queries or feedback, reach out via email at smpy1405@gmail.com.\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmpy2002%2Fcsv_data-info-retrieval-by-ai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmpy2002%2Fcsv_data-info-retrieval-by-ai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmpy2002%2Fcsv_data-info-retrieval-by-ai/lists"}