{"id":27026920,"url":"https://github.com/mazzasaverio/structured-data-jobs","last_synced_at":"2026-05-14T05:32:22.240Z","repository":{"id":282442323,"uuid":"948336526","full_name":"mazzasaverio/structured-data-jobs","owner":"mazzasaverio","description":"A data pipeline that scrapes job opportunities from company websites and uses OpenAI to structure the data. Initially focused on tech roles, but easily adaptable for any job type.","archived":false,"fork":false,"pushed_at":"2025-05-17T15:19:38.000Z","size":685,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-17T15:28:42.935Z","etag":null,"topics":["crawler","docker","llm","logfire","neon","openai","python","uv"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mazzasaverio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-14T06:37:55.000Z","updated_at":"2025-05-17T15:19:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"14a99c31-3f58-4703-851f-cf56b72479e2","html_url":"https://github.com/mazzasaverio/structured-data-jobs","commit_stats":null,"previous_names":["mazzasaverio/lean-jobs-crawler"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mazzasaverio/structured-data-jobs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mazzasaverio%2Fstructured-data-jobs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mazzasaverio%2Fstructured-data-jobs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mazzasaverio%2Fstructured-data-jobs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mazzasaverio%2Fstructured-data-jobs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mazzasaverio","download_url":"https://codeload.github.com/mazzasaverio/structured-data-jobs/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mazzasaverio%2Fstructured-data-jobs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30180672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T12:39:21.703Z","status":"ssl_error","status_checked_at":"2026-03-06T12:36:09.819Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","docker","llm","logfire","neon","openai","python","uv"],"created_at":"2025-04-04T23:16:28.500Z","updated_at":"2026-03-06T14:31:58.053Z","avatar_url":"https://github.com/mazzasaverio.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Structured Data Jobs\n\nThis project aims to build a data pipeline that can collect data about companies and job opportunities, tailored to the information you care about and the types of roles you want to track. The pipeline will pull data directly from company websites and use OpenAI tools to structure it and populate the data model.\n\nWhile the initial setup will focus on tech roles—particularly data engineering and software engineering—it’s designed to be easily adaptable for tracking any kind of role or information you’d like to analyze over time.\n\n## Data Pipeline\n\n### 0. Companies\n\n```bash\nuv run -m src.pipeline.00_companies\n```\nOnce installation and setup are complete, the first step is to prepare a seed list of companies we want to gather information on, including company details and job postings. We'll use each company's main domain as the primary key for identification. TODO: In the future, we could add a crawling and data collection layer to keep the company list up to date.\n\n### 1. Career Pages\n\n```bash\nuv run -m src.pipeline.01_career_pages\n```\n\nWe're trying to identify all the pages where job listings are typically posted. To do this, we use the following approach. The crawler operates across different depth levels:\n\n\n1.1 **Depth Level 0: Career Page Discovery**: We check for common URL patterns like /careers, /jobs, etc.\nWe'll implement several strategies, but to begin with, let's keep it simple. Most companies have a dedicated path for job postings, so we can take advantage of that. This method is fast, non-intrusive, and doesn't require interacting with the page. Second strategy is to find about buttuns in the page the can redirect to a potentially target page\n\n1.2 **Depth Level ≥ 1**: Once we find a career-related page, the system follows it, since job listings are often nested deeper in the site or split into subpages. We then use OpenAI's LLM to analyze the structured output and decide whether a URL is a seed (to explore further) or a target (that lists all job postings directly). Also in this case, let's keep it simple for now and we implement a recursive aproach that end if a target url is found\n\nThe crawler maintains state awareness, verifying target URL validity and handling broken links automatically.\n\n### 2. Job Listings\n\nThe next step is to set up an independent crawler that regularly scans each target, uses Playwright to extract all the text (including expanding any job listings if needed), and then applies an LLM to extract each job listing. We'll store the relevant details in a new table.\n\nThe next step is to choose the role (for example Data Engineer and Software Engineer) from the role in job_posts table and extrat from the related url structured information following similat approach already adopted\n\n### 3. Job Details\n\nThe next step is to configure a YAML file with a list of roles (like Data Engineer or Software Engineer), then use it to filter roles from the job_posts table and extract structured data from the related URLs, just like we did before. Go ahead and create the table, config files, and the necessary code to make this work.\n\n\n### System Requirements\n\n- Python 3.11+\n- Docker + Docker Compose\n- Neon Postgres database\n\n### Installation Steps\n\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/yourusername/lean-jobs-crawler.git\n   cd lean-jobs-crawler\n   ```\n\n2. UV-based setup:\n   ```bash\n   # Install UV package manager\n   curl -LsSf https://astral.sh/uv/install.sh | sh\n   \n   # Project initialization\n   uv init\n   \n   # Dependency installation\n   uv sync\n   ```\n\n3. Environment configuration:\n   ```bash\n   cp .env.example .env\n   # Configure database and system settings\n   ```\n\n### Database Management\n\nUsing Alembic for schema migrations:\n\n1. Initial setup:\n   ```bash\n   # Database initialization\n   uv run alembic upgrade head\n   ```\n\n2. Schema modifications:\n   ```bash\n   # Generate migration\n   uv run alembic revision --autogenerate -m \"Description of changes\"\n   \n   # Apply changes\n   uv run alembic upgrade head\n   ```\n\n3. Migration management:\n   ```bash\n   # History viewing\n   uv run alembic history\n   \n   # Version control\n   uv run alembic downgrade \u003crevision_id\u003e\n   uv run alembic downgrade -1\n   ```\n\nImportant: Always review auto-generated migrations before deployment.\n\n### Local Development\n\n```bash\n# Run the crawler locally\nuv run python -m src.main\n```\n\n### Docker Deployment\n\n#### Using Docker\n\n```bash\n# Build the Docker image\ndocker build -t lean-jobs-crawler .\n\n# Run the container\ndocker run -d --env-file .env lean-jobs-crawler\n```\n\n#### Using Docker Compose\n\n```bash\n# Start all services\ndocker-compose up -d\n\n# View logs\ndocker-compose logs -f\n\n# Stop all services\ndocker-compose down\n```\n\n## Contributing\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add some amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n\n\n## Project Status\n\n⚠️ **Early Development Phase** ⚠️\n\nThis project is in active early development. Core functionality is being implemented and architecture may evolve significantly. The codebase is under construction with ongoing major development.\n\nCurrent Focus Areas:\n- Database schema implementation\n- Job posting crawler development\n- Standardized content extraction system\n\nFuture planned enhancements include automated company discovery and advanced data analytics capabilities.\n\nContributors welcome - note that significant refactoring may occur as the project matures.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmazzasaverio%2Fstructured-data-jobs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmazzasaverio%2Fstructured-data-jobs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmazzasaverio%2Fstructured-data-jobs/lists"}