{"id":20213746,"url":"https://github.com/lexiestleszek/scrapegpt","last_synced_at":"2026-03-11T10:32:52.046Z","repository":{"id":222303944,"uuid":"756867080","full_name":"LexiestLeszek/scrapeGPT","owner":"LexiestLeszek","description":"ScrapeGPT is a RAG-based Telegram bot designed to scrape and analyze websites, then answer questions based on the scraped content. The bot utilizes Retrieval Augmented Generation and webscraping to return natural language answers to the user's queries.","archived":false,"fork":false,"pushed_at":"2024-02-17T09:06:56.000Z","size":64,"stargazers_count":82,"open_issues_count":0,"forks_count":12,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-10T14:14:32.290Z","etag":null,"topics":["crawler","huggingface","large-language-models","llm","ollama","proxy","rag","retrieval-augmented-generation","robots-txt","scraper","telegram-bot","website-scraper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LexiestLeszek.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-02-13T13:17:26.000Z","updated_at":"2025-03-27T04:59:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"45a38f6f-2384-407a-977b-2ca25523307f","html_url":"https://github.com/LexiestLeszek/scrapeGPT","commit_stats":null,"previous_names":["lexiestleszek/scrapegpt"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/LexiestLeszek/scrapeGPT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LexiestLeszek%2FscrapeGPT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LexiestLeszek%2FscrapeGPT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LexiestLeszek%2FscrapeGPT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LexiestLeszek%2FscrapeGPT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LexiestLeszek","download_url":"https://codeload.github.com/LexiestLeszek/scrapeGPT/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LexiestLeszek%2FscrapeGPT/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30378102,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-11T06:09:32.197Z","status":"ssl_error","status_checked_at":"2026-03-11T06:09:17.086Z","response_time":84,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","huggingface","large-language-models","llm","ollama","proxy","rag","retrieval-augmented-generation","robots-txt","scraper","telegram-bot","website-scraper"],"created_at":"2024-11-14T06:11:27.996Z","updated_at":"2026-03-11T10:32:52.010Z","avatar_url":"https://github.com/LexiestLeszek.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ScrapeGPT \n## Telegram Bot for Web Content Analysis and Question Answering\n\nScrapeGPT is a Telegram bot designed to scrape and analyze websites, then answer questions based on the scraped content. The bot utilizes advanced natural language processing techniques to provide accurate responses to user queries.\n\nThe idea came from the job interview question when I was given an assignment to \"study our company website and provide new product recommendations\"\u003e I decided to automate the process of studying a whole website and made ScrapeGPT.\n\nNote:\n\n*You don't have to use it as a Telegram bot, at the very end of the scrapeGPT.py there is a commented main() function to use it as CLI.*\n*Also, there is an experimental file called scrapeGPT_gradio_app.py, where the same functionality is implemented in the form of Gradio app and using qDrant VectorDB, Ollama and GPT4ALLEmbeddings.*\n\n## Features\n\n- **Web Scraping**: Automatically scrapes text from provided URLs, including PDF files.\n- **Context Retrieval**: Utilizes embeddings and retrieval models to extract relevant context from scraped content.\n- **Question Answering**: Generates answers to user questions based on the retrieved context.\n- **Robots.txt Parsing**: Respects website's robots.txt to avoid scraping restricted areas.\n- **Database Management**: Stores scraped content in a database for future reference and quick access.\n- **Proxy Support**: Uses rotating proxies to bypass geo-restrictions and anonymize requests.\n- **LLM-based**: Supports both public and local LLMs.\n\n## Getting Started\n### Installation\n\nTo get started with the bot, follow these steps:\n\n1. Clone the repository to your local machine:\n```bash\ngit clone https://github.com/LexiestLeszek/scrapeGPT.git\ncd scrapeGPT\n```\n2. Install the required dependencies:\n```bash\npip install -r requirements.txt\n```\n3. Go to ollama.com and download it\n4. download local model:\n```bash\nollama pull qwen:1.8b\n```\n5. Set up your Telegram bot by obtaining a token from the [BotFather](https://core.telegram.org/bots#botfather).\n6. Replace `API_TOKEN` in the script with your actual token and don't forget bot's telegram nickname.\n7. Run the bot server script:\n```bash\npython scrapeGPT.py\n```\n\n### Usage\n\nOnce the bot is running, interact with it on Telegram:\n\n1. Find the bot by its name and open the chat with the bot\n2. Send `/start` command to initialize the bot.\n3. Provide a website URL to begin analysis.\n4. Ask a question regarding the analyzed content, and the bot will respond with an answer.\n\n## Contributing\n\nContributions are welcome! To contribute:\n\n- Fork the repository.\n- Make changes in your fork.\n- Submit a pull request with a clear description of your changes.\n\n## License\n\nThis project is licensed under the terms of the MIT license. See the LICENSE file for details.\n\n## Contact\n\nIf you have any questions or suggestions, feel free to open an issue on GitHub.\n\n## No liability for the Developer\n\n- Usage of this software for attacking targets without prior mutual consent is illegal.\n- It is the end user's responsibility to obey all applicable local, state and federal laws.\n- Developers of this software assume no liability and are not responsible for any misuse or damage caused by this program\nby third parties using the software in violation of laws and regulations.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flexiestleszek%2Fscrapegpt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flexiestleszek%2Fscrapegpt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flexiestleszek%2Fscrapegpt/lists"}