{"id":29313716,"url":"https://github.com/franc-dev/remote-job-scraper","last_synced_at":"2025-07-07T10:12:16.437Z","repository":{"id":300027967,"uuid":"1004975613","full_name":"Franc-dev/remote-job-scraper","owner":"Franc-dev","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-19T13:21:58.000Z","size":382,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-19T14:28:06.155Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Franc-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-19T13:17:33.000Z","updated_at":"2025-06-19T13:22:02.000Z","dependencies_parsed_at":"2025-06-19T14:39:57.189Z","dependency_job_id":null,"html_url":"https://github.com/Franc-dev/remote-job-scraper","commit_stats":null,"previous_names":["franc-dev/remote-job-scraper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Franc-dev/remote-job-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Franc-dev%2Fremote-job-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Franc-dev%2Fremote-job-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Franc-dev%2Fremote-job-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Franc-dev%2Fremote-job-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Franc-dev","download_url":"https://codeload.github.com/Franc-dev/remote-job-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Franc-dev%2Fremote-job-scraper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264057650,"owners_count":23550755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-07T10:12:12.496Z","updated_at":"2025-07-07T10:12:16.411Z","avatar_url":"https://github.com/Franc-dev.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Job Board Scraper\n\nA TypeScript-powered web scraper for remote job boards, with automatic saving to both JSON files and a PostgreSQL database using Prisma. Cleans and deduplicates job data on save.\n\n---\n\n## Features\n- Scrapes multiple remote job boards (We Work Remotely, RemoteOK, Remotive, etc.)\n- Saves scraped jobs to JSON files in `job_data/`\n- Automatically saves jobs to a PostgreSQL database (with deduplication)\n- Cleans and normalizes job data before saving\n- Supports scheduling (e.g., every 2 minutes) with `node-cron`\n- CLI for scraping, scheduling, and importing jobs\n\n---\n\n## Setup\n\n### 1. Clone and Install\n```sh\nnpm install\n```\n\n### 2. Configure Database\n- Edit `.env` and set your PostgreSQL connection string:\n  ```\n  DATABASE_URL=\"postgresql://user:password@localhost:5432/yourdb\"\n  ```\n\n### 3. Prisma Setup\n```sh\nnpx prisma migrate dev --name init\nnpx prisma generate\n```\n\n---\n\n## Usage\n\n### Scrape All Boards (and save to DB automatically)\n```sh\nnpm run dev all \"developer\" \"Remote\" 1\n```\n- This scrapes all boards for \"developer\" jobs in \"Remote\" locations (1 page each).\n- Jobs are saved to `job_data/` and also to your database (cleaned and deduplicated).\n\n### Schedule Scraping (e.g., every 2 minutes)\n```sh\nnpm run dev schedule \"developer\" \"Remote\" 1 \"*/2 * * * *\"\n```\n- Uses cron syntax for flexible scheduling.\n\n### Import Existing JSON Data\n```sh\nnpm run dev import job_data/ycombinator_jobs.json\n```\n- Imports jobs from a JSON file into the database (cleaned and deduplicated).\n\n### Import All Job Data (if you have a script for this)\n```sh\nnpx ts-node import_all_job_data.ts\n```\n\n---\n\n## Project Structure\n\n- `main.ts` - Main entry point, CLI, and scheduling logic\n- `JobScraper.ts` - Scraper logic for each job board\n- `db.ts` - Prisma integration, cleaning, deduplication, and import helpers\n- `job_data/` - Folder for all scraped job JSON files\n- `prisma/schema.prisma` - Prisma schema for the Job model\n\n---\n\n## Database Model\n\nPrisma schema (`prisma/schema.prisma`):\n```prisma\nmodel Job {\n  id              Int      @id @default(autoincrement())\n  title           String\n  company         String\n  location        String\n  description     String\n  salary          String?\n  postedDate      DateTime?\n  jobType         String?\n  experienceLevel String?\n  url             String   @unique\n  source          String\n  logo            String?\n  createdAt       DateTime @default(now())\n}\n```\n- Deduplication is enforced by the unique `url` field.\n\n---\n\n## Data Cleaning \u0026 Deduplication\n- All job fields are trimmed and normalized before saving.\n- Only valid dates are saved; invalid or missing dates are set to `null`.\n- Jobs with missing required fields (`title`, `company`, `url`) are skipped.\n- Upsert is used to avoid duplicates in the database.\n\n---\n\n## Extending\n- Add new job boards by extending `JobScraper.ts`.\n- Add new fields to the Prisma schema and regenerate the client.\n\n---\n\n## License\nISC ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffranc-dev%2Fremote-job-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffranc-dev%2Fremote-job-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffranc-dev%2Fremote-job-scraper/lists"}