{"id":31752066,"url":"https://github.com/rs-labhub/secret-scraper","last_synced_at":"2025-10-09T16:55:19.096Z","repository":{"id":317737295,"uuid":"1067980058","full_name":"RS-labhub/secret-scraper","owner":"RS-labhub","description":"A powerful web scraping application that aggregates trending products from GitHub and ProductHunt.","archived":false,"fork":false,"pushed_at":"2025-10-02T17:32:34.000Z","size":652,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-10-02T19:25:20.605Z","etag":null,"topics":["firecrawl","github-api","github-scraper","producthunt-scraper","python","scraper","typescript"],"latest_commit_sha":null,"homepage":"https://secret-scraper.vercel.app","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RS-labhub.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-01T17:09:40.000Z","updated_at":"2025-10-02T17:32:39.000Z","dependencies_parsed_at":"2025-10-02T19:25:28.620Z","dependency_job_id":"0eb31520-187d-4d40-8929-f369e1d598be","html_url":"https://github.com/RS-labhub/secret-scraper","commit_stats":null,"previous_names":["rs-labhub/secret-scraper"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/RS-labhub/secret-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RS-labhub%2Fsecret-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RS-labhub%2Fsecret-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RS-labhub%2Fsecret-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RS-labhub%2Fsecret-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RS-labhub","download_url":"https://codeload.github.com/RS-labhub/secret-scraper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RS-labhub%2Fsecret-scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001778,"owners_count":26083173,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["firecrawl","github-api","github-scraper","producthunt-scraper","python","scraper","typescript"],"created_at":"2025-10-09T16:55:17.939Z","updated_at":"2025-10-09T16:55:19.089Z","avatar_url":"https://github.com/RS-labhub.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"![secret-scrapper](public/og-image.png)\n\n# Secret Scrapper\n\nA powerful web scraping application that aggregates trending products from GitHub and ProductHunt.\n\n## Features\n\n- **GitHub Scraping**: Scrape trending repositories with GitHub trending pages\n- **ProductHunt Scraping**: Scrape ProductHunt leaderboards (daily, weekly, monthly) using Firecrawl\n- **AI Enhancement**: Enhance product descriptions and categorization using various AI providers\n- **Product Management**: Approve, reject, and edit scraped products\n- **Export**: Export data to various formats (JSON, CSV, etc.)\n- **CMS Integration**: Publish to various CMS platforms (REMOVED FOR NOW)\n\n## Architecture\n\n- **Frontend**: Next.js with TypeScript and Tailwind CSS\n- **GitHub Scraping**: TypeScript-based scraper integrated into the Next.js API\n- **ProductHunt Scraping**: Python backend using Firecrawl and FastAPI\n- **AI Enhancement**: Multiple AI provider support (OpenAI, Anthropic, etc.)\n\n## Setup\n\n### Prerequisites\n\n- Node.js 18+\n- npm/yarn/pnpm\n- Python 3.8+ (for ProductHunt scraping)\n- Firecrawl API key (for ProductHunt scraping)\n\n### Frontend Setup\n\n1. Install dependencies:\n   ```bash\n   bun install\n   ```\n\n2. Start the development server:\n   ```bash\n   bun run dev\n   ```\n\n3. Open [http://localhost:3000](http://localhost:3000) in your browser.\n\n### ProductHunt Backend Setup (DEPRECATED)\n\nFor ProductHunt scraping, you need to start the Python backend:\n\n1. Navigate to the backend directory:\n   ```bash\n   cd backend\n   ```\n\n2. Install Python dependencies:\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n3. Start the backend server:\n   ```bash\n   python main.py\n   ```\n   \n   Or use the startup scripts:\n   - Windows: `start.bat`\n   - Linux/Mac: `./start.sh`\n\n4. The backend will be available at [http://localhost:8000](http://localhost:8000)\n\n### API Keys\n\n- **Firecrawl API Key**: Required for ProductHunt scraping. Get it from [firecrawl.dev](https://www.firecrawl.dev)\n- **AI Provider API Keys**: Required for AI enhancement features\n\n## Usage\n\n### Scraping Products\n\n1. Click \"Scrape Products\" button\n2. Choose your source (GitHub or ProductHunt)\n3. For ProductHunt:\n   - Enter your Firecrawl API key\n   - Select time period (daily/weekly/monthly)\n   - Enter the date/week/month you want to scrape\n   - Optionally enable \"Featured only\" mode\n4. Configure filters and click \"Start Scraping\"\n\n### Managing Products\n\n- **View**: Browse scraped products in the main interface\n- **Edit**: Click edit button to modify product details\n- **Enhance**: Use AI to improve descriptions and categorization\n- **Approve/Reject**: Set product status for workflow management\n- **Export**: Export selected products to various formats\n\n## ProductHunt URLs\n\nThe application generates ProductHunt URLs in these formats:\n\n- **Daily**: `https://www.producthunt.com/leaderboard/daily/YYYY/M/D/all`\n- **Weekly**: `https://www.producthunt.com/leaderboard/weekly/YYYY/W/all`\n- **Monthly**: `https://www.producthunt.com/leaderboard/monthly/YYYY/M/all`\n- **Featured** (remove `/all`): Shows only featured products\n\nExamples:\n- Daily: `https://www.producthunt.com/leaderboard/daily/2025/9/21/all`\n- Weekly: `https://www.producthunt.com/leaderboard/weekly/2025/38/all`\n- Monthly: `https://www.producthunt.com/leaderboard/monthly/2025/9/all`\n\n## API Endpoints\n\n### Frontend (Next.js)\n- `POST /api/scrape` - GitHub scraping (ProductHunt now redirects to Python backend)\n- `POST /api/ai/enhance` - AI enhancement\n- `POST /api/export` - Export products\n- `POST /api/cms/publish` - CMS publishing // REMOVED FOR NOW\n\n### Python Backend (DEPRECATED)\n- `GET http://localhost:8000/` - API information\n- `GET http://localhost:8000/health` - Health check\n- `POST http://localhost:8000/scrape` - ProductHunt scraping\n- `GET http://localhost:8000/docs` - Interactive API documentation\n\n\n\u0026nbsp;\n\n## Meet the Author\n\n\u003cimg  src=\"https://raw.githubusercontent.com/RS-labhub/secret-scraper/master/public/Author.jpg\" alt=\"Author\"\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n**Built for Finding Radhika by [RS-labhub](https://github.com/RS-labhub)**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frs-labhub%2Fsecret-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frs-labhub%2Fsecret-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frs-labhub%2Fsecret-scraper/lists"}