{"id":20710018,"url":"https://github.com/oxylabs/scraping-real-estate-data-with-python","last_synced_at":"2025-10-09T14:05:01.477Z","repository":{"id":216662468,"uuid":"741851517","full_name":"oxylabs/scraping-real-estate-data-with-python","owner":"oxylabs","description":"A guide for extracting public data from Redfin using Python and Oxylabs Web Scraper API. From using the API to processing data and saving the result.","archived":false,"fork":false,"pushed_at":"2025-09-24T12:58:30.000Z","size":1252,"stargazers_count":7,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-09-24T14:39:22.196Z","etag":null,"topics":["e-commerce-scraper","ecommerce-api","ecommerce-scraper","python","python-scraper","real-estate","scraper-api","web-scraping"],"latest_commit_sha":null,"homepage":"https://oxylabs.io/products/scraper-api/web","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-01-11T08:43:01.000Z","updated_at":"2025-09-24T12:58:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"2e02b15d-f122-4add-96a3-5377a3503f01","html_url":"https://github.com/oxylabs/scraping-real-estate-data-with-python","commit_stats":null,"previous_names":["oxylabs/scraping-real-estate-data-with-python"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oxylabs/scraping-real-estate-data-with-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscraping-real-estate-data-with-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscraping-real-estate-data-with-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscraping-real-estate-data-with-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscraping-real-estate-data-with-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/scraping-real-estate-data-with-python/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscraping-real-estate-data-with-python/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001522,"owners_count":26083117,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["e-commerce-scraper","ecommerce-api","ecommerce-scraper","python","python-scraper","real-estate","scraper-api","web-scraping"],"created_at":"2024-11-17T02:09:33.137Z","updated_at":"2025-10-09T14:05:01.471Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":["GitHub Projects"],"sub_categories":[],"readme":"# Scraping Real Estate Data With Python\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.io/pages/gitoxy?utm_source=877\u0026utm_medium=affiliate\u0026groupid=877\u0026utm_content=scraping-real-estate-data-with-python-github\u0026transaction_id=102f49063ab94276ae8f116d224b67)\n\n[![](https://dcbadge.limes.pink/api/server/Pds3gBmKMH?style=for-the-badge\u0026theme=discord)](https://discord.gg/Pds3gBmKMH) [![YouTube](https://img.shields.io/badge/YouTube-Oxylabs-red?style=for-the-badge\u0026logo=youtube\u0026logoColor=white)](https://www.youtube.com/@oxylabs)\n\nHere's the process of collecting public property data from [Redfin](https://www.redfin.com/) with the help of Oxylabs [Web Scraper API](https://oxylabs.io/products/scraper-api/web) (**1-week free trial**) and Python. You can scrape real estate data like prices, sizes, number of beds and baths available, and addresses, increasing the likelihood of finding a good deal or understanding the market better. \n\nFor the full guide, check our [blog post](https://oxylabs.io/blog/scraping-real-estate-data).\n\n## 1. Prepare environment\n\n```python\ntouch main.py\n```\n\n### Install dependencies\n\n```python\npip install bs4 requests pandas\n```\n\n### Import libraries\n\n```python\nimport requests\nimport pandas as pd\nfrom bs4 import BeautifulSoup\n```\n\n## 2. Prepare the API request\n\n```python\nUSERNAME = \"USERNAME\"\nPASSWORD = \"PASSWORD\"\n\npayload = {\n    \"source\": \"universal\",\n    \"url\": \"https://www.redfin.com/city/29470/IL/Chicago\",\n}\n```\n\n## 3. Send request\n\n```python\nresponse = requests.post(\n    url=\"https://realtime.oxylabs.io/v1/queries\",\n    auth=(USERNAME, PASSWORD),\n    json=payload,\n)\nresponse.raise_for_status()\nprint(response.status_code)\n```\n## 4. Extract HTML\n\n```python\nhtml = response.json()[\"results\"][0][\"content\"]\nsoup = BeautifulSoup(html, \"html.parser\")\n```\n\n## 5. Parse data from HTML\n\n![Locating elements](images/Redfin1.jpg)\n\n![Locating elements](images/Redfin2.jpg)\n\n```python\ndef extract_data_from_listing(listing):\n    price = listing.find(\"span\", {\"class\": \"homecardV2Price\"}).get_text(strip=True)\n    address = listing.find(\"span\", {\"class\": \"collapsedAddress\"}).get_text(strip=True)\n    stats = listing.find_all(\"div\", {\"class\":\"stats\"})\n    try:\n        bed_count_elem, bath_count_elem, size_elem = stats[0], stats[1], stats[2]\n    except IndexError:\n        raise Exception(\"Got less stats than expected\")\n\n    bed_count = bed_count_elem.get_text(strip=True)\n    bath_count = bath_count_elem.get_text(strip=True)\n    size = size_elem.get_text(strip=True)\n\n    return {\n        \"price\": price,\n        \"address\": address,\n        \"bed_count\": bed_count,\n        \"bath_count\": bath_count,\n        \"size\": size,\n    }\n\n\ndata = []\n\nfor listing in soup.find_all(\"div\", {\"class\": \"bottomV2\"}):\n    entry = extract_data_from_listing(listing)\n    data.append(entry)\n```\n**Note:** you can also request Markdown output using `\"markdown: true\"` flag to get results in an easy-to-read format for various AI workflows.\n\n## 6. Save to CSV\n\n```python\ndf = pd.DataFrame(data)\ndf.to_csv(\"real_estate_data.csv\")\n```\n\n## The complete code\n\n```python\nimport requests\nimport pandas as pd\nfrom bs4 import BeautifulSoup\n\n\ndef extract_data_from_listing(listing):\n    price = listing.find(\"span\", {\"class\": \"homecardV2Price\"}).get_text(strip=True)\n    address = listing.find(\"span\", {\"class\": \"collapsedAddress\"}).get_text(strip=True)\n    stats = listing.find_all(\"div\", {\"class\":\"stats\"})\n    try:\n        bed_count_elem, bath_count_elem, size_elem = stats[0], stats[1], stats[2]\n    except IndexError:\n        raise Exception(\"Got less stats than expected\")\n\n    bed_count = bed_count_elem.get_text(strip=True)\n    bath_count = bath_count_elem.get_text(strip=True)\n    size = size_elem.get_text(strip=True)\n\n    return {\n        \"price\": price,\n        \"address\": address,\n        \"bed_count\": bed_count,\n        \"bath_count\": bath_count,\n        \"size\": size,\n    }\n\n\nUSERNAME = \"USERNAME\"\nPASSWORD = \"PASSWORD\"\n\npayload = {\n    \"source\": \"universal\",\n    \"url\": \"https://www.redfin.com/city/29470/IL/Chicago\",\n}\n\nresponse = requests.post(\n    url=\"https://realtime.oxylabs.io/v1/queries\",\n    auth=(USERNAME, PASSWORD),\n    json=payload,\n)\nresponse.raise_for_status()\n\nhtml = response.json()[\"results\"][0][\"content\"]\nsoup = BeautifulSoup(html, \"html.parser\")\n\ndata = []\n\nfor listing in soup.find_all(\"div\", {\"class\": \"bottomV2\"}):\n    entry = extract_data_from_listing(listing)\n    data.append(entry)\n\n\ndf = pd.DataFrame(data)\ndf.to_csv(\"real_estate_data.csv\")\n```\n\n## Wrapping up\n\nUsing Python and [Web Scraper API](https://oxylabs.io/products/scraper-api/web) is a seamless way to automate real estate data collection processes required for insights into the real estate market. \n\nPlease refer to our [technical documentation](https://developers.oxylabs.io/scraper-apis/real-estate-scraper-api/redfin) for more on the API parameters and variables found in this tutorial.\n\nIf you have any questions, feel free to reach out by sending a message to support@oxylabs.io.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fscraping-real-estate-data-with-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fscraping-real-estate-data-with-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fscraping-real-estate-data-with-python/lists"}