{"id":20710105,"url":"https://github.com/oxylabs/how-to-scrape-indeed","last_synced_at":"2025-04-05T14:07:32.108Z","repository":{"id":218071168,"uuid":"745397971","full_name":"oxylabs/how-to-scrape-indeed","owner":"oxylabs","description":"A tutorial for collecting job postings from Indeed using Python and Oxylabs Web Scraper API.","archived":false,"fork":false,"pushed_at":"2025-02-11T13:17:02.000Z","size":51,"stargazers_count":151,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-29T13:10:00.163Z","etag":null,"topics":["api","job-posting","python","scraper-api","web-scraper","web-scraping"],"latest_commit_sha":null,"homepage":"https://oxylabs.io/products/scraper-api/web","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-19T08:50:56.000Z","updated_at":"2025-03-06T17:45:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"9e0c454e-241e-4cae-b647-ea3d07e7c9c2","html_url":"https://github.com/oxylabs/how-to-scrape-indeed","commit_stats":null,"previous_names":["oxylabs/how-to-scrape-indeed"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-indeed","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-indeed/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-indeed/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-scrape-indeed/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/how-to-scrape-indeed/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247345853,"owners_count":20924102,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","job-posting","python","scraper-api","web-scraper","web-scraping"],"created_at":"2024-11-17T02:09:50.388Z","updated_at":"2025-04-05T14:07:32.090Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# How to Scrape Indeed\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.go2cloud.org/aff_c?offer_id=7\u0026aff_id=877\u0026url_id=112)\n\nHere's the process of extracting job postings from [Indeed](https://www.indeed.com/) with the help of Oxylabs [Web Scraper API](https://oxylabs.io/products/scraper-api/web) (**1-week free trial**) and Python.\n\nFor the complete guide with in-depth explanations and visuals, check our [blog post](https://oxylabs.io/blog/how-to-scrape-indeed).\n\n## Project setup\n\n### Creating a virtual environment\n\n```python\npython -m venv indeed_env #Windows\npython3 -m venv indeed_env #Macand Linux\n```\n\n### Activating the virtual environment\n\n```python\n.\\indeed_env\\Scripts\\Activate#Windows\nsource indeed_env/bin/activate #Macand Linux\n```\n\n### Installing libraries\n\n```python\n$ pip install requests\n```\n\n## Overview of Web Scraper API\n\nThe following is an example that shows how Web Scraper API works.\n\n```python\n# scraper_api_demo.py\nimport requests\npayload = {\n    \"source\": \"universal\",\n    \"url\": \"https://www.indeed.com\"\n}\nresponse = requests.post(\n    url=\"https://realtime.oxylabs.io/v1/queries\",\n    json=payload,\n    auth=(username,password),\n)\nprint(response.json())\n```\n\n## Web Scraper API parameters\n\n### Parsing the page title and retrieving results in JSON\n\n```python\n\"title\": {\n    \"_fns\": [\n                {\n                    \"_fn\": \"xpath_one\",\n                    \"_args\": [\"//title/text()\"]\n                }\n            ]\n        }\n},\n```\n\nIf you send this as `parsing_instructions`, the output would be the following JSON.\n\n```python\n{ \"title\": \"Job Search | Indeed\", \"parse_status_code\": 12000 }\n```\n\nNote that the `parse_status_code` means a successful response.\n\nThe following code prints the title of the Indeed page.\n\n```python\n# indeed_title.py\n\nimport requests\npayload = {\n    \"source\": \"universal\",\n    \"url\": \"https://www.indeed.com\",\n    \"parse\": True,\n    \"parsing_instructions\": {\n        \"title\": {\n            \"\\_fns\": [\n                        {\n                            \"\\_fn\": \"xpath_one\",\n                            \"\\_args\": [\n                                \"//title/text()\"\n                                ]\n                        }\n                    ]\n                }\n    },\n}\nresponse = requests.post(\n    url=\"https://realtime.oxylabs.io/v1/queries\",\n    json=payload,\n    auth=('username', 'password'),\n)\nprint(response.json()['results'][0]['content'])\n```\n\n## Scraping Indeed job postings\n\n### Selecting a job listing\n\n```python\n`.job_seen_beacon`\n```\n\n### Creating the placeholder for a job listing\n\n```\n\"job_listings\": {\n    \"_fns\": [\n        {\n            \"_fn\": \"css\",\n            \"_args\": [\".job_seen_beacon\"]\n        }\n    ],\n    \"_items\": {\n        \"job_title\": {\n            \"_fns\": [\n                {\n                \"_fn\": \"xpath_one\",\n                \"_args\": [\".//h2[contains(@class,'jobTitle')]/a/span/text()\"]\n                }\n            ]\n        },\n        \"company_name\": {\n            \"_fns\": [\n                {\n                    \"_fn\": \"xpath_one\",\n                    \"_args\": [\".//span[@data-testid='company-name']/text()\"]\n                }\n            ]\n        },\n```\n\n### Adding other selectors\n\n```json\n{\n  \"source\": \"universal\",\n  \"url\": \"https://www.indeed.com/jobs?q=work+from+home\u0026l=San+Francisco%2C+CA\",\n  \"parse\": true,\n  \"parsing_instructions\": {\n    \"job_listings\": {\n      \"_fns\": [\n        {\n          \"_fn\": \"css\",\n          \"_args\": [\".job_seen_beacon\"]\n        }\n      ],\n      \"_items\": {\n        \"job_title\": {\n          \"_fns\": [\n            {\n              \"_fn\": \"xpath_one\",\n              \"_args\": [\".//h2[contains(@class,'jobTitle')]/a/span/text()\"]\n            }\n          ]\n        },\n        \"company_name\": {\n          \"_fns\": [\n            {\n              \"_fn\": \"xpath_one\",\n              \"_args\": [\".//span[@data-testid='company-name']/text()\"]\n            }\n          ]\n        }\n      }\n    }\n  }\n}\n```\n\nFor other data points, see the file [here](src/job_search_payload.json).\n\n### Saving the payload as a separator JSON file\n\n```python\n# parse_jobs.py\n\nimport requests\nimport json\npayload = {}\nwith open(\"job_search_payload.json\") as f:\n    payload = json.load(f)\nresponse = requests.post(\n    url=\"https://realtime.oxylabs.io/v1/queries\",\n    json=payload,\n    auth=(\"username\", \"password\"),\n)\nprint(response.status_code)\nwith open(\"result.json\", \"w\") as f:\n    json.dump(response.json(), f, indent=4)\n```\n\n## Exporting to JSON and CSV\n\n```python\n# parse_jobs.py\nwith open(\"results.json\", \"w\") as f:\n    json.dump(data, f, indent=4)\ndf = pd.DataFrame(data[\"results\"][0][\"content\"][\"job_listings\"])\ndf.to_csv(\"job_search_results.csv\", index=False)\n```\n\n## Final word\n\nCheck our [documentation](https://developers.oxylabs.io/scraper-apis/web-scraper-api) for more API parameters and variables found in this tutorial.\n\nIf you have any questions, feel free to contact us at support@oxylabs.io.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fhow-to-scrape-indeed","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fhow-to-scrape-indeed","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fhow-to-scrape-indeed/lists"}