{"id":26317219,"url":"https://github.com/alexmarqs/webscraper-greenhouse","last_synced_at":"2026-04-28T21:35:16.161Z","repository":{"id":214631114,"uuid":"641790046","full_name":"alexmarqs/webscraper-greenhouse","owner":"alexmarqs","description":"🍀 Webscraper Green House API","archived":false,"fork":false,"pushed_at":"2023-08-22T16:22:22.000Z","size":575,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-13T01:03:01.243Z","etag":null,"topics":["api","cheerio","cronjob","nodemailer","postmark","serverless","vercel","webscraping","zod"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alexmarqs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-05-17T07:10:37.000Z","updated_at":"2023-12-29T12:29:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"e68d530d-f8ab-42d1-9f90-b8fdde364d16","html_url":"https://github.com/alexmarqs/webscraper-greenhouse","commit_stats":null,"previous_names":["alexmarqs/webscraper-greenhouse"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/alexmarqs/webscraper-greenhouse","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmarqs%2Fwebscraper-greenhouse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmarqs%2Fwebscraper-greenhouse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmarqs%2Fwebscraper-greenhouse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmarqs%2Fwebscraper-greenhouse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alexmarqs","download_url":"https://codeload.github.com/alexmarqs/webscraper-greenhouse/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmarqs%2Fwebscraper-greenhouse/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32400868,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-28T19:38:08.556Z","status":"ssl_error","status_checked_at":"2026-04-28T19:37:55.688Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","cheerio","cronjob","nodemailer","postmark","serverless","vercel","webscraping","zod"],"created_at":"2025-03-15T13:18:48.083Z","updated_at":"2026-04-28T21:35:16.149Z","avatar_url":"https://github.com/alexmarqs.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Websraper Green House API\n\nMy wife is checking every week a specific website to check if certain candidatures are already available. This is a very manual and boring task, so I decided to automate it. I will be using a Cron Job to check every week if the candidatures are available, and if so, I will send an email to my wife. It will be running only for a few weeks, after that I will stop it.\n\n## API Endpoints\n\n- `/api/cron/greenhouse` - Web scraper to check if the Green House is already available, if so, it will send a notification. Pass a request parameter `?API_KEY=\u003cyour secret\u003e` to authenticate the request.\n\n## Requirements\n\n- Node.js 18.x\n- Vercel account + package `vercel` installed globally\n- Postmark account\n\n## Tech Stack\n\n- [Vercel](https://vercel.com/) - Serverless Functions\n- [Cheerio](https://cheerio.js.org/) - Web scraping (to parse HTML)\n- [Vercel Cron Jobs](https://vercel.com/blog/cron-jobs) - Cron jobs (alternatives: AWS CloudWatch Events, qstash, inngest, github actions, etc.)\n- [Zod](https://zod.dev) - Type validation for the environment variables\n- [NodeMailer](https://nodemailer.com/about/) - Send emails\n- [Postmark](https://postmarkapp.com/) - Email service (in combination with NodeMailer)\n\n## Architecture Diagram\n\n![Architecture Diagram](./docs/arch.png)\n\n## Tecnical Notes\n\nAs you can see in the code I'm including as part of the lambda/function the notification mecanism. I'm creating the promises to send the email for each user, and then I'm using `Promise.all` to wait for all the promises to be resolved. This is because I want to send all the emails at the same time, and not one by one. What are the downsides of this approach? Let's see:\n\n- If one of the emails fails, all the emails will fail. I'm not handling this case, but I could do it quickly by using `Promise.allSettled` and then check if there is any email that failed and retry it;\n- If the number of users is too big the function will timeout or the third party email service will reject requests (due to some rate limits):\n  - I could use dedicated queue system (like AWS SQS, Upstash QStash, ...) to send the emails in batches + retry the failed ones --\u003e Event Driven Architecture!\n  - I can send emails in batches (e.g. 10 emails per batch) and then wait for a few seconds before sending the next batch: I can do this by using `Promise.all` and then `setTimeout` to wait for a few seconds before sending the next batch. Or actually wait until it's the previous batch is resolved. To send emails in batch I can use some great packages like `p-queue` or `p-limit`;\n\n## Run Locally\n\n```bash\nvercel dev\n```\n\nTo expose to the API locally to the outside world,\nyou can use [ngrok](https://ngrok.com/).\n\n## Deploy to production\n\n```bash\nvercel deploy --prod\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexmarqs%2Fwebscraper-greenhouse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexmarqs%2Fwebscraper-greenhouse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexmarqs%2Fwebscraper-greenhouse/lists"}