{"id":16463370,"url":"https://github.com/geshan/domain-scraper","last_synced_at":"2025-03-23T10:34:17.796Z","repository":{"id":52064317,"uuid":"145258518","full_name":"geshan/domain-scraper","owner":"geshan","description":"A very simple domain.com.au rental listing scraper to compare listings as there is no compare option in domain. Written with node js.","archived":false,"fork":false,"pushed_at":"2024-03-17T03:37:06.000Z","size":542,"stargazers_count":12,"open_issues_count":2,"forks_count":3,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-10-12T11:14:13.679Z","etag":null,"topics":["australia","csv","docker","dockerized","domain-com-au","domain-rentals","javascript","melbourne","nodejs","rental","rental-listing-scraper","scraper","sydney"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/geshan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-08-18T23:11:42.000Z","updated_at":"2024-08-18T05:09:56.000Z","dependencies_parsed_at":"2024-02-21T06:30:37.614Z","dependency_job_id":"f4760905-d6f2-44fa-87b6-d4a10b5c1336","html_url":"https://github.com/geshan/domain-scraper","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geshan%2Fdomain-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geshan%2Fdomain-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geshan%2Fdomain-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geshan%2Fdomain-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/geshan","download_url":"https://codeload.github.com/geshan/domain-scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221849589,"owners_count":16891495,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["australia","csv","docker","dockerized","domain-com-au","domain-rentals","javascript","melbourne","nodejs","rental","rental-listing-scraper","scraper","sydney"],"created_at":"2024-10-11T11:14:17.068Z","updated_at":"2024-10-28T15:37:37.171Z","avatar_url":"https://github.com/geshan.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Simple Domain.com.au scraper\n\nA very simple domain.com.au rental listing scraper to compare listings as there is no compare\noption in domain. So this script pulls in `X` number of listing that helps you compare domain's\nrental listing as coming to a new city it is a very handy thing to have. This will be very helpful\nto people trying to move and even more helpful to people who just landed in Australian cities.\n\n![From domain shortlist listing](https://github.com/geshan/domain-scraper/raw/master/readme-images/shortlist-listing.png)\n![to](https://github.com/geshan/domain-scraper/raw/master/readme-images/down-arrow.png)\n![to a csv file](https://github.com/geshan/domain-scraper/raw/master/readme-images/csv-listing.png)\n\n### Disclaimer\n\nVery irresponsible and hacky software ahead that works and does the job :). It might break randomly.\n\n### Packages\n\nIt is only using Axios, Cheerio and Json2Csv for now.\n\n### Steps\n\nEach domain rental listing has a unique Id like `https://www.domain.com.au/12500140` redirects to\n\n`https://www.domain.com.au/27a-arcadia-road-chester-hill-nsw-2162-12500140` but the id `12500140`\nstays the same. So this script uses this unique id to visit that page and scrape 10 fields out of it then save it as a csv which is a lot easy to compare than visiting 10-20 pages. For an example I took 5 listing/IDs randomly. You can do it following the steps below:\n\n* Add all your liked listing on `domain.com.au` in your shortlist. Yes you need to register and login.\n* Then run [this](https://gist.github.com/geshan/378be819646682c715e38a653c680401) super small script on `each` of your shortlist page to get the IDs you want to compare. (only tested on chrome)\n\n![script run on chrome](https://github.com/geshan/domain-scraper/raw/master/readme-images/shortlist-to-ids-js.png)\n* Copy/replace the IDs to line no. [22-27](https://github.com/geshan/domain-scraper/blob/master/index.js#L22-L27), basically fill up the `listingIds` array correctly.\n* Then run the script as shown below to get the csv of your listing Ids.\n\n### Run\n\n`node index.js` gives out domain-rentals.csv in the same directory with some fields for comparision like rent, no. of bedrooms, no. of bathrooms, no. of parking, available from etc. Hopefully you have a recent verison of node installed locally or try docker.\n\n### Running with docker\n\nJust do `docker-compose up`. On each save it will run the scirpt as it is running with `nodemon`.\n\n### Tests\n\nWould have been good to have them, it is just a useful hack. \n\n### Gotcha\n\nIt will not run for buying listings and it will have wierd characters here and there. Don't like this, send a PR! ;)\n\n### Contributions\n\nYes of course, there are lots of things to improve. Any contribution/PR is really welcome.\n\n### To Do\n\n* ~~add docker and docker-compose~~\n* add tests may be :D\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeshan%2Fdomain-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeshan%2Fdomain-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeshan%2Fdomain-scraper/lists"}