{"id":13566023,"url":"https://github.com/RealEstateWebTools/property_web_scraper","last_synced_at":"2025-04-03T23:30:56.374Z","repository":{"id":26560447,"uuid":"94769726","full_name":"RealEstateWebTools/property_web_scraper","owner":"RealEstateWebTools","description":"Web based UI to make scraping data from real estate websites super simple","archived":false,"fork":false,"pushed_at":"2023-03-09T00:12:14.000Z","size":16921,"stargazers_count":92,"open_issues_count":7,"forks_count":22,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-11-04T19:42:32.232Z","etag":null,"topics":["mls","rails","real-estate","rets","ruby","scrapers"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RealEstateWebTools.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-06-19T11:29:38.000Z","updated_at":"2024-11-01T16:02:27.000Z","dependencies_parsed_at":"2024-01-19T07:02:16.126Z","dependency_job_id":"6d6c05ce-118a-47bd-90dc-018538f44117","html_url":"https://github.com/RealEstateWebTools/property_web_scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RealEstateWebTools%2Fproperty_web_scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RealEstateWebTools%2Fproperty_web_scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RealEstateWebTools%2Fproperty_web_scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RealEstateWebTools%2Fproperty_web_scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RealEstateWebTools","download_url":"https://codeload.github.com/RealEstateWebTools/property_web_scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247097644,"owners_count":20883122,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mls","rails","real-estate","rets","ruby","scrapers"],"created_at":"2024-08-01T13:02:00.302Z","updated_at":"2025-04-03T23:30:52.436Z","avatar_url":"https://github.com/RealEstateWebTools.png","language":"JavaScript","funding_links":["https://opencollective.com/property_web_builder"],"categories":["JavaScript"],"sub_categories":[],"readme":"# PWB as used by [https://homestocompare.com/](https://homestocompare.com/)\n\nGreat new for anyone interested in this project, a US appeals court has recently reaffirmed that web scraping is legal!!\n\nhttps://techcrunch.com/2022/04/18/web-scraping-legal-court/amp/\n\nI am currently a bit busy with other projects but hope to return to update it shortly.\n\nPlease get in touch if you are interested in helping me take this project forward.\n\n\n## [Demo of PropertyWebScraper ](https://homestocompare.com/)\n\nPreviously there was a demo at [https://real-estate-web-scraper.herokuapp.com](https://real-estate-web-scraper.herokuapp.com/)\n\nI have had to take that down as heroku has cancelled thier free tier.  You can see it being used in production here though: [https://homestocompare.com/](https://homestocompare.com/)\n\nTo use Property Web Scraper as a service you would make calls like:\n\n[https://real-estate-web-scraper.herokuapp.com/retriever/as_json?url=https%3A%2F%2Fwww.rightmove.co.uk%2Fproperty-for-sale%2Fproperty-68790657.html](https://real-estate-web-scraper.herokuapp.com/retriever/as_json?url=https%3A%2F%2Fwww.rightmove.co.uk%2Fproperty-for-sale%2Fproperty-68790657.html)\n\n## Support for importing properties from an MLS\n\nPropertyWebScraper also supports connecting to different MLSs (Multiple Listing Services) in the US using RETS and Web Services.  This functionality is currently experimental and has only been implemented for 3 MLSs.  If there is a particular MLS for which you need support, open an issue with the details.   \n\n## Rails Version\n\nPropertyWebScraper runs with Rails \u003e= 5.0\n\n## Ruby Version\n\nPropertyWebScraper runs with Ruby \u003e= 2.0.0.\n\n\n## Installation\n\nInstall into an existing Rails project by adding these lines in your applications's Gemfile:\n\n```ruby\ngem 'property_web_scraper', git: 'https://github.com/RealEstateWebTools/property_web_scraper', branch: 'master'\n```\n\nAlso, be sure to use Postgres as your database (by having the \"pg\" gem and Postgres installed locally \nAnd then execute:\n```bash\n$ bundle\n```\n\nMount PropertyWebScraper by adding the following to your routes.rb file:\n```ruby\nmount PropertyWebScraper::Engine =\u003e '/'\n```\n\nand run the ff commands from the console:\n```bash\nrails property_web_scraper:install:migrations\nrails db:create\nrails db:migrate\nrails property_web_scraper:db:seed\n```\n\n\n## Contribute and spread the love\nWe encourage you to contribute to this project and file issues for any problems you encounter.\n\nIf you like it, please star it and spread the word on [Twitter](https://twitter.com/prptywebbuilder), [LinkedIn](https://www.linkedin.com/company/propertywebbuilder) and [Facebook](https://www.facebook.com/propertywebbuilder).  You can also subscribe to github notifications on this project.  \n\nPlease consider making a contribution to the development of PropertyWebScraper.  If you wish to pay for specific enhancements, please email me directly (opensource at propertywebbuilder.com).\n\nOr make a contribution to PropertyWebBuilder here:\n[https://opencollective.com/property_web_builder](https://opencollective.com/property_web_builder)\n\n---\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).\n\n## Disclaimer\n\nWhile scraping can sometimes be used as a legitimate way to access all kinds of data on the internet, it’s also important to consider the legal implications. There are cases where scraping data may be considered illegal, or open you to the possibility of being sued. \n\nI created this tool in part as a learning exercise and am sharing it in case others find it useful. If you do decide to use this tool to scrape a website it is your responsibilty to ensure that what you are doing is legal.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRealEstateWebTools%2Fproperty_web_scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FRealEstateWebTools%2Fproperty_web_scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRealEstateWebTools%2Fproperty_web_scraper/lists"}