{"id":28863187,"url":"https://github.com/smaranjitghose/websage","last_synced_at":"2026-05-07T01:38:00.518Z","repository":{"id":299416332,"uuid":"1002959719","full_name":"smaranjitghose/WebSage","owner":"smaranjitghose","description":"AI agent that can scrape websites, crawl pages, and extract structured data in real-time to answer your complex queries","archived":false,"fork":false,"pushed_at":"2025-06-16T13:04:25.000Z","size":99,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-06-16T13:26:39.186Z","etag":null,"topics":["agent","ai-agent","firecrawl","gpt-4o-mini","gpt-o4-mini","langchain","langgraph","mcp","mcp-server","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/smaranjitghose.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-16T12:07:18.000Z","updated_at":"2025-06-16T13:25:41.000Z","dependencies_parsed_at":"2025-06-16T13:37:51.731Z","dependency_job_id":null,"html_url":"https://github.com/smaranjitghose/WebSage","commit_stats":null,"previous_names":["smaranjitghose/websage"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/smaranjitghose/WebSage","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smaranjitghose%2FWebSage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smaranjitghose%2FWebSage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smaranjitghose%2FWebSage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smaranjitghose%2FWebSage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/smaranjitghose","download_url":"https://codeload.github.com/smaranjitghose/WebSage/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smaranjitghose%2FWebSage/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260898761,"owners_count":23079263,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","ai-agent","firecrawl","gpt-4o-mini","gpt-o4-mini","langchain","langgraph","mcp","mcp-server","python"],"created_at":"2025-06-20T07:02:07.599Z","updated_at":"2026-05-07T01:38:00.512Z","avatar_url":"https://github.com/smaranjitghose.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🕷️ WebSage\n\n[![Python 3.13+](https://img.shields.io/badge/Python-3.13+-blue.svg)](https://www.python.org/downloads/)\n[![OpenAI](https://img.shields.io/badge/OpenAI-GPT--4o--mini-green.svg)](https://openai.com/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Firecrawl](https://img.shields.io/badge/Firecrawl-Web%20Scraping-orange.svg)](https://firecrawl.dev/)\n[![MCP](https://img.shields.io/badge/MCP-Model%20Context%20Protocol-purple.svg)](https://modelcontextprotocol.io/)\n\nAn intelligent conversational agent that combines OpenAI's GPT models with Firecrawl's powerful web scraping capabilities through the Model Context Protocol (MCP). Chat naturally with an AI that can scrape websites, crawl pages, and extract structured data in real-time.\n\n## 🖼️ Snapshot\n\n![working](./assets/output.png)\n\n## 🔧 Prerequisites\n\n- [Python 3.13](https://www.python.org/downloads/) or higher\n- [uv](https://docs.astral.sh/uv/#highlights) (efficient Python Package Management)\n- [Node.js](https://nodejs.org/en) (for Firecrawl MCP)\n- [OpenAI API](https://platform.openai.com) key\n- [Firecrawl](https://www.firecrawl.dev/) API key\n\n## 📥 Installation\n\n1. Clone the project\n\n```\ngit clone https://github.com/smaranjitghose/websage\n```\n\n2. Navigate inside the directory\n\n```\ncd websage\n```\n3. Create a `.env` file and store the API keys from OPENAI and FIRECRAWL as:\n\n```\nOPENAI_API_KEY=your-openai-api-key-here\nFIRECRAWL_API_KEY=your-firecrawl-api-key-here\n```\n\n4. Run the agent using uv (one the first run it will create a virtual environment and install the dependencies in it)\n\n```\nuv run main.py\n```\n\n5. Start chatting with examples:\n\n```\n🗣️  You: Scrape https://news.ycombinator.com and get the top 5 story titles\n\n🗣️  You: Extract all product prices from https://store.example.com\n\n🗣️  You: Crawl https://blog.company.com and summarize the latest 3 articles\n```\n\n6. Exit when done:\n\n```\n🗣️  You: quit\n```\n\n\n\n\n## 💡 Usage Examples\n\n### Basic Web Scraping\n```\nScrape https://example.com for main content and headings\n```\n\n### E-commerce Data Extraction\n```\nGet all product names and prices from https://shop.example.com/category/electronics\n```\n\n### News and Content Analysis\n```\nCrawl https://techblog.com and extract all article titles published this month\n```\n\n### Research and Data Collection\n```\nScrape https://research-site.edu/publications and extract paper titles, authors, and abstracts\n```\n\n### SEO and Competitive Analysis\n```\nAnalyze the main headings and meta descriptions from https://competitor.com\n```\n\n## 🔍 How It Works\n\n1. **MCP Integration** - Uses Model Context Protocol to connect GPT with Firecrawl\n2. **Tool Loading** - Dynamically loads available Firecrawl scraping tools\n3. **ReAct Agent** - Employs reasoning and acting pattern for intelligent tool use\n4. **Conversation Loop** - Maintains context across multiple scraping requests\n5. **Error Recovery** - Handles network issues and API errors gracefully\n\n## 🛡️ Best Practices\n\n### Ethical Scraping\n- Always respect `robots.txt` files\n- Be mindful of rate limiting and server load\n- Only scrape publicly available content\n- Consider reaching out to site owners for large-scale scraping\n\n### API Usage\n- Monitor your Firecrawl and OpenAI API usage\n- Set appropriate rate limits in production\n- Cache results when appropriate to reduce API calls\n\n## 🔧 Troubleshooting\n\n### Common Issues\n\n\n**\"Invalid API key\" errors**\n- Verify your `.env` file is in the correct location\n- Check that API keys are valid and have sufficient credits\n- Ensure no extra spaces or quotes in the `.env` file\n\n**Connection timeout errors**\n- Check your internet connection\n- Some websites may block automated requests\n- Try with a different website to test functionality\n\n**Python import errors**\n```bash\nuv add --upgrade mcp langchain-mcp-adapters langgraph langchain-openai python-dotenv\n```\n\n## 🤝 Contributing\n\nContributions are welcome! Here's how:\n\n1. Fork the repository\n2. Create a feature branch\n```\ngit switch -c feature/amazing-feature\n```\n3. Commit and push your changes\n```\ngit commit -m \"Add amazing feature\"\ngit push origin feature/amazing-feature\n```\n4. Open a pull request\n\n\n## 📝 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n---\n\nMade with ❤️ by [Smaranjit Ghose](https://github.com/smaranjitghose)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmaranjitghose%2Fwebsage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmaranjitghose%2Fwebsage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmaranjitghose%2Fwebsage/lists"}