{"id":44372623,"url":"https://github.com/firecrawl/open-scouts","last_synced_at":"2026-02-11T20:25:50.865Z","repository":{"id":328744195,"uuid":"1106048549","full_name":"firecrawl/open-scouts","owner":"firecrawl","description":"  AI-powered web monitoring platform. Create automated scouts that search the web and send email alerts when they find what you're looking for. ","archived":false,"fork":false,"pushed_at":"2026-01-30T00:48:17.000Z","size":35021,"stargazers_count":1194,"open_issues_count":3,"forks_count":167,"subscribers_count":6,"default_branch":"main","last_synced_at":"2026-01-30T08:50:33.724Z","etag":null,"topics":["ai-agents","alerts","automation","email-notifications","firecrawl","nextjs","open-source","openai","posthog","react","resend","supabase","tailwindcss","typescript","vercel","web-monitoring","web-scraping"],"latest_commit_sha":null,"homepage":"https://openscouts.firecrawl.dev","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/firecrawl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-28T14:46:39.000Z","updated_at":"2026-01-30T00:48:21.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/firecrawl/open-scouts","commit_stats":null,"previous_names":["firecrawl/open-scouts"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/firecrawl/open-scouts","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firecrawl%2Fopen-scouts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firecrawl%2Fopen-scouts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firecrawl%2Fopen-scouts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firecrawl%2Fopen-scouts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/firecrawl","download_url":"https://codeload.github.com/firecrawl/open-scouts/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firecrawl%2Fopen-scouts/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29343823,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-11T20:11:40.865Z","status":"ssl_error","status_checked_at":"2026-02-11T20:10:41.637Z","response_time":97,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","alerts","automation","email-notifications","firecrawl","nextjs","open-source","openai","posthog","react","resend","supabase","tailwindcss","typescript","vercel","web-monitoring","web-scraping"],"created_at":"2026-02-11T20:25:50.181Z","updated_at":"2026-02-11T20:25:50.855Z","avatar_url":"https://github.com/firecrawl.png","language":"TypeScript","readme":"# Open Scouts\n\nCreate AI scouts that continuously search the web and notify you when they find what you're looking for.\n\n![open-scouts_4](https://github.com/user-attachments/assets/a1ff82ef-97e4-469b-9712-99d0367755a7)\n\n## About\n\nOpen Scouts is an AI-powered monitoring platform that lets you create \"scouts\" - automated tasks that run on a schedule to continuously search for and track information. Whether you're looking for new restaurants near you, monitoring AI news, or tracking any other updates, scouts work 24/7 to find what you need and notify you when they discover it.\n\n## Tech Stack\n\n- **Next.js 15**\n- **React 19**\n- **TypeScript**\n- **Tailwind CSS v4**\n- **Supabase** (Database + Auth + Edge Functions)\n- **pgvector** (Vector embeddings for semantic search)\n- **Firecrawl SDK** (@mendable/firecrawl-js)\n- **OpenAI API** (AI Agent + Embeddings)\n- **Resend** (Email Notifications)\n\n## Getting Started\n\n### Prerequisites\n\n- Node.js 18+\n- bun (default), npm, or pnpm\n- Supabase account ([supabase.com](https://supabase.com))\n- OpenAI API key ([platform.openai.com](https://platform.openai.com))\n- Firecrawl API key ([firecrawl.dev](https://firecrawl.dev))\n- Resend API key ([resend.com](https://resend.com)) - for email notifications\n- Google Cloud Console account (for Google OAuth - optional)\n\n### 1. Clone and Install\n\n```bash\ngit clone https://github.com/firecrawl/open-scouts\ncd open-scouts\nbun install  # or: npm install / pnpm install\n```\n\n### 2. Create Supabase Project\n\n1. Go to [supabase.com](https://supabase.com/dashboard)\n2. Create a new project\n3. Wait for the project to finish provisioning\n\n### 3. Enable Required Extensions\n\nIn your Supabase Dashboard:\n\n1. Go to **Database → Extensions**\n2. Search for and enable:\n   - `pg_cron` (for scheduled jobs)\n   - `pg_net` (for HTTP requests from database)\n   - `vector` (for AI-powered semantic search on execution summaries)\n   - `supabase_vault` (for secure credential storage - usually enabled by default)\n\n### 4. Set Up Environment Variables\n\nCreate a `.env` file in the root directory by copying the example file:\n\n```bash\ncp .env.example .env\n```\n\nThen fill in your actual values in the `.env` file.\n\n**The `.env.example` file contains all required environment variables with detailed instructions and direct links for where to obtain each API key.**\n\n### 5. Run Database Setup\n\nFirst, link your Supabase project (required for syncing secrets):\n\n```bash\nbunx supabase login        # Login to Supabase CLI (one-time)\nbunx supabase link --project-ref \u003cyour-project-ref\u003e  # Find ref in Supabase Dashboard URL\n```\n\nThen run the setup script:\n\n```bash\nbun run setup:db  # or: npm run setup:db / pnpm run setup:db\n```\n\nThis will:\n- Create all required tables (`scouts`, `scout_executions`, `scout_execution_steps`, etc.)\n- Add user authentication support (user_id columns, Row Level Security)\n- Enable real-time subscriptions\n- Set up vector embeddings for AI-generated execution summaries\n- Configure the **scalable dispatcher architecture** (pg_cron + pg_net + vault)\n- Automatically store your Supabase URL and service role key in the vault\n- Set up cron jobs for scout dispatching and cleanup\n- **Sync Edge Function secrets** from your `.env` file (OPENAI_API_KEY, FIRECRAWL_API_KEY, RESEND_API_KEY)\n\n**Note:** The setup script will check if the required extensions (`vector`, `pg_cron`, `pg_net`) are enabled. If not, follow the on-screen instructions to enable them in the Supabase Dashboard, then run the script again.\n\n### 6. Set Up Authentication\n\nOpen Scouts uses Supabase Auth for user authentication, supporting both email/password and Google OAuth.\n\n#### Enable Email/Password Auth (Enabled by Default)\n\n1. Go to Supabase Dashboard → **Authentication** → **Providers** → **Email**\n2. Ensure \"Enable Email Provider\" is toggled on\n3. Configure email templates as needed in **Authentication** → **Email Templates**\n\n#### Enable Google OAuth (Optional but Recommended)\n\n1. **Create Google OAuth Credentials:**\n   - Go to [Google Cloud Console](https://console.cloud.google.com/)\n   - Create a new project or select existing one\n   - Navigate to **APIs \u0026 Services** → **Credentials**\n   - Click **Create Credentials** → **OAuth client ID**\n   - Choose \"Web application\" as Application type\n   - Add authorized JavaScript origins:\n     - `http://localhost:3000` (development)\n     - `https://your-domain.com` (production)\n   - Add authorized redirect URIs:\n     - `https://\u003cyour-project-ref\u003e.supabase.co/auth/v1/callback`\n   - Copy the **Client ID** and **Client Secret**\n\n2. **Configure in Supabase:**\n   - Go to Supabase Dashboard → **Authentication** → **Providers** → **Google**\n   - Toggle \"Enable Google Provider\"\n   - Paste your Client ID and Client Secret\n   - Save\n\n### 7. Deploy Edge Functions\n\nDeploy the scout execution agent and email functions to Supabase Cloud:\n\n```bash\nbunx supabase functions deploy scout-cron\nbunx supabase functions deploy send-test-email\n```\n\n**Note:** Secrets (OPENAI_API_KEY, FIRECRAWL_API_KEY, RESEND_API_KEY) are automatically synced when you run `setup:db`. If you need to update them manually:\n\n```bash\nbunx supabase secrets set OPENAI_API_KEY=sk-proj-...\n```\n\n### 8. Set Up Resend (Email Notifications)\n\nEmail notifications are sent to your account email when scouts find results.\n\n1. **Create a Resend account** at [resend.com](https://resend.com)\n2. **Get your API key** from the Resend dashboard\n3. **Add to `.env`** and run `setup:db` again to sync, or set manually:\n   ```bash\n   bunx supabase secrets set RESEND_API_KEY=re_...\n   ```\n4. **Verify a custom domain** at [resend.com/domains](https://resend.com/domains) to send to any email\n\n**Important - Free Tier Limitations:**\n- Without a verified domain, Resend only sends to your Resend account email\n- Free tier includes 3,000 emails/month (100/day limit)\n\n**Testing Email Setup:**\n1. Go to **Settings** in the app\n2. Click **Send Test Email** to verify the configuration\n3. Check your inbox for the test email\n\n### 9. Firecrawl Configuration\n\nOpen Scouts uses [Firecrawl](https://firecrawl.dev) for web scraping and search.\n\n\u003e **📌 Important: No Environment Variable Configuration Required**\n\u003e\n\u003e **You do NOT need to configure a Firecrawl API key in your environment variables.** Each user can simply add their own custom Firecrawl API key directly in the **Settings** page within the app. This is the recommended approach for most users.\n\n#### Custom API Key (Recommended)\n\nThe simplest way to use Open Scouts:\n\n1. Sign up at [firecrawl.dev](https://firecrawl.dev)\n2. Get your API key from the [dashboard](https://www.firecrawl.dev/app/api-keys)\n3. Go to **Settings** in the Open Scouts app\n4. Enter your Firecrawl API key in the **Firecrawl Integration** section\n5. Click **Save** - you're ready to go!\n\nEach user manages their own API key, and usage is tracked to their individual Firecrawl account.\n\n#### Server-Side API Key (Optional - For Self-Hosting)\n\nIf you're self-hosting and want to provide a shared API key for all users:\n\n1. Sign up at [firecrawl.dev](https://firecrawl.dev)\n2. Get your API key from the [dashboard](https://www.firecrawl.dev/app/api-keys)\n3. Add to your `.env` file:\n   ```bash\n   FIRECRAWL_API_KEY=fc-your-key-here\n   ```\n4. Set the edge function secret:\n   ```bash\n   npx supabase secrets set FIRECRAWL_API_KEY=fc-your-key-here\n   ```\n\n**Note:** Users who add their own custom API key in Settings will use their personal key instead of the server-side shared key.\n\n#### Partner Integration (Enterprise - Closed Beta)\n\n\u003e **⚠️ The Partner Key feature is currently in closed beta and only available for enterprise customers.** This feature is not publicly available.\n\n**How Partner Integration Works (when available):**\n- When users sign up, a unique Firecrawl API key is automatically created for them\n- Each user's usage is tracked separately\n- Keys are stored securely in the `user_preferences` table\n- If a user's key fails, the system automatically falls back to the shared partner key\n- Users can view their connection status in **Settings → Firecrawl Integration**\n\n### 10. Run the Development Server\n\n```bash\nbun run dev  # or: npm run dev / pnpm run dev\n```\n\nOpen [http://localhost:3000](http://localhost:3000) to see the app.\n\n## How It Works\n\n### User Authentication Flow\n\n1. **Public Home Page**: Users can browse the landing page without signing in\n2. **Create Scout**: When a user types a query and hits Enter, they're prompted to sign in\n3. **Sign In/Sign Up**: Users can authenticate via email/password or Google OAuth\n4. **Continue Flow**: After authentication, the scout creation continues automatically\n5. **User Isolation**: Each user only sees and manages their own scouts\n\n### Scout System\n\n1. **Create a Scout**: Define what you want to monitor (e.g., \"Scout for any recent Indian restaurants near me\" or \"Scout for any AI news\")\n2. **AI Agent Setup**: The system automatically configures search queries and strategies\n3. **Set Frequency**: Choose how often to run (hourly, every 3 days, weekly)\n4. **Configure Notifications**: Add your email in Settings to receive alerts when scouts find results\n5. **Continuous Monitoring**: The dispatcher checks every minute and triggers due scouts individually\n6. **AI Summaries**: Each successful execution generates a concise one-sentence summary with semantic embeddings\n7. **Get Notified**: Receive email alerts when scouts find new results (if email is configured)\n8. **View Results**: See all findings with AI-generated summaries in real-time on the scout page\n\n### Manual Execution\n\nClick the **\"Run Now\"** button on any scout page to trigger execution immediately without waiting for the cron.\n\n### Email Notifications\n\nWhen scouts find results, you'll automatically receive email alerts at your account email:\n\n- **Automatic**: Emails are sent only when scouts successfully find results\n- **Rich Content**: Beautiful HTML emails with scout results and links\n- **Test**: Use the \"Send Test Email\" button in Settings to verify setup\n\n**Email Service**: Powered by Resend (free tier includes 3,000 emails/month)\n\n**Note:** On Resend's free tier without a verified domain, emails can only be sent to your Resend account email. Verify a custom domain at [resend.com/domains](https://resend.com/domains) to send to any email.\n\n### Architecture\n\n- **Frontend**: Next.js app with real-time updates via Supabase Realtime\n- **Database**: PostgreSQL (Supabase) with pg_cron for scheduling and pgvector for semantic search\n- **Authentication**: Supabase Auth (Email/Password + Google OAuth)\n- **AI Agent**: OpenAI GPT-4 with function calling (search \u0026 scrape tools)\n- **AI Summaries**: Auto-generated one-sentence summaries with vector embeddings for each successful execution\n- **Edge Function**: Deno-based serverless function that orchestrates agent execution\n- **Web Scraping**: Firecrawl API for search and content extraction (supports per-user API keys via partner integration)\n\n#### Scalable Dispatcher Architecture\n\nOpen Scouts uses a dispatcher pattern designed to scale to thousands of scouts:\n\n```\nEvery minute:\npg_cron → dispatch_due_scouts() → finds due scouts → pg_net HTTP POST\n                                                          ↓\n                                    ┌──────────────────────┼──────────────────────┐\n                                    ↓                      ↓                      ↓\n                              Edge Function          Edge Function          Edge Function\n                              (scout A)              (scout B)              (scout C)\n                              [isolated]             [isolated]             [isolated]\n```\n\n- **Dispatcher (SQL)**: Runs every minute via pg_cron, queries for due scouts, and fires individual HTTP requests\n- **Isolated Execution**: Each scout runs in its own edge function invocation with full resources (256MB memory, 400s timeout)\n- **Automatic Cleanup**: A separate cron job cleans up stuck executions every 5 minutes\n- **Vault Integration**: Supabase credentials are securely stored in the vault and read by the dispatcher\n\n## Security\n\n- **Row Level Security (RLS)**: All database tables have RLS policies ensuring users can only access their own data\n- **User Isolation**: Scouts, messages, and executions are all tied to authenticated users\n- **Secure Auth Flow**: OAuth tokens and sessions are managed by Supabase Auth\n- **Service Role**: Server-side operations (cron jobs, edge functions) use service role for privileged access\n- **API Key Storage**: Firecrawl API keys (when using partner integration) are stored server-side in `user_preferences` and never exposed to the client\n\n## Build for Production\n\n```bash\nbun run build  # or: npm run build / pnpm run build\nbun start      # or: npm start / pnpm start\n```\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nMIT\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirecrawl%2Fopen-scouts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffirecrawl%2Fopen-scouts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirecrawl%2Fopen-scouts/lists"}