{"id":50851807,"url":"https://github.com/anthonystepvoy/reddit-ideas-scrapper","last_synced_at":"2026-06-14T14:03:27.037Z","repository":{"id":311456429,"uuid":"1043737715","full_name":"anthonystepvoy/reddit-ideas-scrapper","owner":"anthonystepvoy","description":"A comprehensive intelligence platform for capturing, analyzing, and developing startup ideas using systematic data collection and AI-powered enrichment. Built with Next.js, Supabase, and AI integration.","archived":false,"fork":false,"pushed_at":"2026-02-06T15:07:53.000Z","size":1148,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-14T14:03:26.452Z","etag":null,"topics":["business","ideas","reddit","saas","scrapper"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anthonystepvoy.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-24T14:12:51.000Z","updated_at":"2026-02-06T15:09:46.000Z","dependencies_parsed_at":"2025-08-24T20:02:56.554Z","dependency_job_id":"def87ce2-5cce-4766-a413-1ebb9d7f9650","html_url":"https://github.com/anthonystepvoy/reddit-ideas-scrapper","commit_stats":null,"previous_names":["anthonystepvoy/reddit-ideas-scrapper"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/anthonystepvoy/reddit-ideas-scrapper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anthonystepvoy%2Freddit-ideas-scrapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anthonystepvoy%2Freddit-ideas-scrapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anthonystepvoy%2Freddit-ideas-scrapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anthonystepvoy%2Freddit-ideas-scrapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anthonystepvoy","download_url":"https://codeload.github.com/anthonystepvoy/reddit-ideas-scrapper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anthonystepvoy%2Freddit-ideas-scrapper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34324004,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-14T02:00:07.365Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["business","ideas","reddit","saas","scrapper"],"created_at":"2026-06-14T14:03:26.308Z","updated_at":"2026-06-14T14:03:27.032Z","avatar_url":"https://github.com/anthonystepvoy.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reddit Ideas Scrapper 🚀\n\n\u003e **Note**: This project was originally built for personal use and is now being shared publicly to contribute to the developer community.\n\nA comprehensive intelligence platform for capturing, analyzing, and developing startup ideas using systematic data collection and AI-powered enrichment. Built with Next.js, Supabase, and AI integration.\n\n## 🎯 Project Overview\n\nThis project implements the \"Pragmatic Builder's Action Plan\" - a strategic approach to building a personal idea generation and analysis system that's superior to scraping competitors. It combines:\n\n- **Patient Collector**: Manual + API-based idea capture\n- **Systematic Reddit Mining**: Automated problem discovery\n- **Enrichment Pipeline**: AI-powered idea development\n- **Remix Framework**: Strategic differentiation and innovation\n\n\u003cdiv align=\"center\"\u003e\n\n\u003cimg width=\"1210\" height=\"859\" alt=\"image\" src=\"https://github.com/user-attachments/assets/40c66a2f-248c-4ca1-a816-2696dedefc3a\" /\u003e\n\n\n\u003cimg width=\"1382\" height=\"907\" alt=\"image\" src=\"https://github.com/user-attachments/assets/47221711-43ec-4d49-9cbd-ff50a56eb44b\" /\u003e\n\n\u003c/div\u003e\n\n\n## 🛠️ Tech Stack\n\n- **Frontend**: Next.js 15 with React 19\n- **Styling**: Tailwind CSS\n- **Authentication**: Clerk\n- **Database**: Supabase\n- **AI Integration**: OpenAI API\n- **Backend Scripts**: Python for data collection\n\n## 📁 Project Structure\n\n```\nreddit-ideas-scrapper/\n├── src/                    # Next.js app router\n│   ├── app/               # Next.js app router\n│   ├── components/        # React components\n│   └── utils/            # Utility functions\n├── public/                # Static assets\n├── airtable_manager.py    # Airtable database interactions\n├── reddit_scanner.py      # Reddit idea mining\n├── requirements.txt       # Python dependencies\n└── package.json          # Node.js dependencies\n```\n\n## 🚀 Quick Start\n\n### Frontend Setup (Next.js)\n\n1. **Clone the repository**\n   ```bash\n   git clone https://github.com/anthonystepvoy/reddit-ideas-scrapper.git\n   cd reddit-ideas-scrapper\n   ```\n\n2. **Install Node.js dependencies:**\n   ```bash\n   npm install\n   ```\n\n3. **Set up environment variables:**\n   - Copy `.env.local.example` to `.env.local`\n   - Add your API keys for Clerk, Supabase, and OpenAI\n\n4. **Run the development server:**\n   ```bash\n   npm run dev\n   ```\n\n### Backend Setup (Python Scripts)\n\n1. **Set up Python virtual environment:**\n   ```bash\n   python -m venv venv\n   source venv/bin/activate  # On Windows: venv\\Scripts\\activate\n   pip install -r requirements.txt\n   ```\n\n2. **Configure Python environment:**\n   - Set up your Reddit API credentials\n   - Configure Airtable API access\n\n## 📊 Airtable Database Schema\n\n| Field Name | Data Type | Notes |\n|------------|-----------|-------|\n| `IdeaID` | Autonumber | Auto-generated |\n| `IdeaTitle` | Single line text | |\n| `ProblemStatement` | Long text | |\n| `SolutionOverview` | Long text | |\n| `OpportunityAnalysis` | Long text | |\n| `FeasibilityScore` | Rating (1-5) | |\n| `MarketInsights` | Long text | Enable rich text |\n| `CustomerPersona` | Long text | Enable rich text |\n| `DistributionStrategy` | Long text | Enable rich text |\n| `PricingStrategy` | Single line text | |\n| `DataSource` | Single select | Options: \"Reddit\", \"IdeaBrowser Daily\", \"Manual\" |\n| `Status` | Single select | Options: \"Backlog\", \"Researching\", \"Prototyping\" |\n\n## 🤖 AI Prompts Library\n\n### Prompt 1: The \"Tedious Workflow\" Generator\n*Use this to generate new raw ideas based on Greg Isenberg's methodology*\n\n```\nBased on Greg Isenberg's methodology, give me 10 tedious, manual, or repetitive workflows that a [Insert Profession, e.g., 'Paralegal', 'e-commerce store owner', 'project manager'] does that AI could automate. For each one, briefly describe the pain point.\n```\n\n### Prompt 2: The \"Idea Enrichment\" Pipeline\n*Use this when you have a raw idea and want to fill out your database fields*\n\n```\nI have a startup concept. Act as a startup analyst and help me enrich it based on the following schema.\n\n**Idea Title:** [e.g., \"AI-Powered Report Formatter for Accountants\"]\n**Problem Statement:** [e.g., \"Accountants spend hours manually copying data from various sources into a standardized weekly report format. It's time-consuming, error-prone, and soul-crushing.\"]\n\nNow, generate the following fields:\n- **Solution Overview:**\n- **Opportunity Analysis:** (Why is now the right time?)\n- **Initial Customer Persona:** (Be specific: name, role, frustrations, goals)\n- **Data-Driven Market Insights:** (Provide some hypothetical but realistic data points on market size or trends)\n- **Ranked Distribution Strategy:** (List 3-4 potential customer acquisition channels)\n- **Suggested Pricing Strategy:**\n```\n\n### Prompt 3: The \"Remix Framework\"\n*Use this on existing ideas to generate differentiated versions*\n\n```\nI want to apply the \"Remix Framework\" to an existing business concept.\n\n**Original Concept:** [e.g., A prayer reminders app called \"Prayminder\"]\n**Core Components:** [e.g., It's a free, solitary app for Christians to list prayers and get reminders]\n**Known Weaknesses:** [e.g., The Android app is buggy, it's limited to one religion, and it has no community or B2B features]\n\nNow, perform Step 3 and 4 of the framework:\n\n1. **Identify Differentiating Angles:** Brainstorm multiple vectors for differentiation based on the weaknesses (e.g., Niche, Technology, Business Model, Feature Set).\n\n2. **Rebuild and Differentiate:** Generate three distinct \"remixes\" based on these angles. For each remix, provide a new concept name, its core value proposition, its target audience, and a potential monetization model.\n```\n\n## 🔄 Complete Workflow\n\n1. **Daily/Weekly:** Run `python reddit_scanner.py` to find raw problems\n2. **Capture:** Use `python airtable_manager.py` to log promising ideas with status \"Backlog\"\n3. **Enrich:** Pick backlog ideas and use the \"Idea Enrichment\" prompt to generate detailed fields\n4. **Remix \u0026 Innovate:** Use the \"Remix Framework\" prompt on your most promising ideas\n5. **Prototype:** Start coding MVPs for your best remixed ideas\n\n## 🛠️ Usage Examples\n\n### Adding a new idea manually:\n```bash\npython airtable_manager.py\n```\n\n### Scanning Reddit for problems:\n```bash\npython reddit_scanner.py\n```\n\n## 📈 Strategic Benefits\n\nThis system provides several advantages over competitor scraping:\n\n1. **Ethical \u0026 Legal**: No copyright issues or terms of service violations\n2. **Proprietary Data**: Your insights are unique and valuable\n3. **Scalable Process**: Systematic approach that improves over time\n4. **Defensible Position**: Builds real expertise and market understanding\n5. **Innovation Focus**: Generates truly new ideas rather than copying existing ones\n\n## 🔒 Security Notes\n\n- Never commit your `.env` file to version control\n- Keep your API keys secure and rotate them regularly\n- The `.gitignore` file is configured to exclude sensitive files\n\n## 📚 Resources\n\n- [Airtable API Documentation](https://airtable.com/api)\n- [PRAW (Python Reddit API Wrapper)](https://praw.readthedocs.io/)\n- [Greg Isenberg's Content](https://twitter.com/gregisenberg)\n\n---\n\n*Built with ❤️ using Cursor's AI-assisted coding capabilities* \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanthonystepvoy%2Freddit-ideas-scrapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanthonystepvoy%2Freddit-ideas-scrapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanthonystepvoy%2Freddit-ideas-scrapper/lists"}