{"id":35699377,"url":"https://github.com/feashliaa/job-board-aggregator","last_synced_at":"2026-07-05T12:00:45.405Z","repository":{"id":332009311,"uuid":"1128608573","full_name":"Feashliaa/job-board-aggregator","owner":"Feashliaa","description":"Job board aggregator indexing 1,000,000+ active positions from 20,000+ companies across Greenhouse, Lever, Ashby, Workday, and other major ATS platforms. Multithreaded Python ETL pipeline, daily automated refreshes via GitHub Actions, filterable client-side search.","archived":false,"fork":false,"pushed_at":"2026-06-28T11:05:49.000Z","size":6313281,"stargazers_count":78,"open_issues_count":0,"forks_count":18,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-06-28T13:06:50.019Z","etag":null,"topics":["ats","data-engineering","data-pipeline","etl","github-actions","gzip","javascript","job-board","job-search","jobs","multithreading","python","web-scraping","web-workers"],"latest_commit_sha":null,"homepage":"https://feashliaa.github.io/job-board-aggregator/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Feashliaa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-05T22:31:51.000Z","updated_at":"2026-06-28T11:05:52.000Z","dependencies_parsed_at":"2026-06-17T16:04:20.985Z","dependency_job_id":null,"html_url":"https://github.com/Feashliaa/job-board-aggregator","commit_stats":null,"previous_names":["feashliaa/job-board-aggregator"],"tags_count":184,"template":false,"template_full_name":null,"purl":"pkg:github/Feashliaa/job-board-aggregator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Feashliaa%2Fjob-board-aggregator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Feashliaa%2Fjob-board-aggregator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Feashliaa%2Fjob-board-aggregator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Feashliaa%2Fjob-board-aggregator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Feashliaa","download_url":"https://codeload.github.com/Feashliaa/job-board-aggregator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Feashliaa%2Fjob-board-aggregator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35153109,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-05T02:00:06.290Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ats","data-engineering","data-pipeline","etl","github-actions","gzip","javascript","job-board","job-search","jobs","multithreading","python","web-scraping","web-workers"],"created_at":"2026-01-06T01:13:39.265Z","updated_at":"2026-07-05T12:00:45.399Z","avatar_url":"https://github.com/Feashliaa.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Job Board Aggregator\n\n![Job Board Aggregator - table view with salary and ATS data](docs/screenshot-table.png)\n\nAutomated job board aggregating 1,000,000+ positions from 20,000+ companies across seven major ATS platforms. Updated daily via GitHub Actions.\n\n## Live Site\n\n[View Job Board](https://feashliaa.github.io/job-board-aggregator)\n\n## Features\n\n- **Multi-platform scraping**: Greenhouse, Lever, Ashby, BambooHR, iCIMs, Paylocity, and Workday APIs scraped in parallel using `concurrent.futures`\n- **Progressive loading**: Chunked gzip data loaded via Web Workers for fast initial render\n- **Advanced filtering**: Filter by title, company, location, ATS platform, experience level, and exclude keywords. Toggle remote-only, hide recruiter postings, or hide already-applied jobs\n- **Job tier classification**: Automatic skill-level tagging (intern/entry/mid/senior) using weighted keyword scoring on job titles\n- **Application tracking**: Mark jobs as saved, applied, or ignored with batch update support via localStorage\n- **URL state sync**: Filter/sort/page state persisted in the URL for shareable/bookmarkable searches\n- **Responsive design**: Desktop table view with card-based mobile layout\n- **Automated pipeline**: Daily GitHub Actions workflow: fetch existing data → scrape → merge → push chunks to the data repo → create release\n- **Trend tracking + anomaly detection**: Daily per-platform counts are appended to a trend log; a check flags any platform whose volume deviates sharply from its recent baseline and opens a GitHub issue\n- **Interactive heatmap**: Map view showing job density by location\n\n![Map view with job density heatmap and filtering](docs/screenshot-map.png)\n\n## Tech Stack\n\n| Layer    | Tools                                                  |\n| -------- | ------------------------------------------------------ |\n| Frontend | Vanilla JavaScript (ES Modules), Bootstrap 5, HTML/CSS |\n| Scraping | Python 3.12, `requests`, `concurrent.futures`, `gzip`  |\n| Data     | Chunked gzip JSON, Web Workers for decompression       |\n| CI/CD    | GitHub Actions (daily cron + manual dispatch)          |\n| Hosting  | GitHub Pages (app), plus a second Pages site for data  |\n\n## Architecture\n\n```\nscripts/\n├── scraper.py          # Multi-ATS scraper with parallel fetching\n├── merge_data.py       # Deduplicates and prunes stale jobs (\u003e30 days)\n├── geolocation.py      # Location lookup/enrichment for the heatmap\n└── check_anomalies.py  # Per-platform volume anomaly detection on the trend log\n\njs/\n├── app.js              # Main app class and initialization\n├── jobs_loader.js      # Progressive chunk loading + Web Worker orchestration\n├── chunk_worker.js     # Web Worker for gzip decompression\n├── filters.js          # Filter logic with regex matching\n├── sort_logic.js       # Client-side sort with alpha/numeric handling\n├── renderer.js         # Table/card rendering with pagination\n├── storage.js          # localStorage wrapper for application tracking\n├── columns.js          # Column definitions and custom renderers\n├── events.js           # Event listener setup\n├── url_state.js        # URL query string sync\n└── ui_utils.js         # Toast notifications, HTML escaping, utilities\n\ndata/\n├── *_companies.json    # Company lists per ATS platform (tracked on main)\n├── salary/             # Salary lookup table, sharded a-z (static input)\n├── locations.json      # Geolocation lookup\n└── trends/daily.jsonl  # Append-only daily trend history (one JSON object per line)\n\n# Chunked job data (jobs_chunk_*.json.gz + jobs_manifest.json) is NOT in this repo.\n# Each run force-pushes it to a separate data repo (Feashliaa/job-board-data),\n# which serves the chunks over its own GitHub Pages site (Fastly-backed CDN).\n# This keeps the code repo small and code-only while the data repo stays flat\n# (force-pushed each run, so it never accumulates history).\n```\n\n## Data Pipeline\n\n1. **Scrape**: `scraper.py` fetches jobs from all seven ATS APIs concurrently. Worker counts are tuned per platform to respect each one's rate limits: 50 for Workday, 30 for Greenhouse/Lever/iCIMs, 10 for BambooHR, and 5 for Ashby/Paylocity (the tightest limiters).\n2. **Classify**: Each job is tagged with a skill level based on title keywords and flagged if posted by a recruiting agency\n3. **Clean**: Jobs missing titles, URLs, or company info are dropped\n4. **Chunk**: Results are split into ~25k-job gzipped chunks with a manifest file\n5. **Merge**: `merge_data.py` deduplicates against existing data and prunes jobs older than 30 days\n6. **Monitor**: A daily trend snapshot (per-platform and per-tier counts) is appended to `data/trends/daily.jsonl` and committed to main. `check_anomalies.py` compares each platform against its recent baseline and opens an issue if one drops off or spikes abnormally.\n7. **Deploy**: GitHub Actions force-pushes the regenerated chunks to the separate `job-board-data` [repo](https://github.com/Feashliaa/job-board-data) and creates a tagged release on main. The frontend fetches chunks from the data repo's GitHub Pages site, keeping the main repo code-only.\n\n## Company Discovery\n\nCompany lists are built from Common Crawl index data using a separate harvesting pipeline. The harvester scans CDX archives for URLs matching 20+ ATS domain patterns, extracts company slugs via regex, and deduplicates across multiple crawl snapshots. This currently yields give or take 95,000 unique company identifiers.\n\n## Local Development\n\n```bash\ngit clone https://github.com/Feashliaa/job-board-aggregator.git\ncd job-board-aggregator\npython -m http.server 8000\n# Visit http://localhost:8000\n```\n\nThe frontend fetches live chunk data from the data repo's Pages site over the network, so local development works against current data without needing the chunks checked out locally.\n\nTo run the scraper locally:\n\n```bash\ncd scripts\npip install -r requirements.txt\npython scraper.py --source manual\n```\n\n## License\n\nCode in this repository is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\nThe curated company datasets in `data/` are licensed under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). You're free to use, modify, and share the data for non-commercial purposes. Commercial use of the datasets requires permission - reach out via [GitHub Issues](https://github.com/Feashliaa/job-board-aggregator/issues) or email.\n\n---\n\nBuilt by [Riley Dorrington](https://github.com/Feashliaa)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffeashliaa%2Fjob-board-aggregator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffeashliaa%2Fjob-board-aggregator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffeashliaa%2Fjob-board-aggregator/lists"}