{"id":20166494,"url":"https://github.com/longzheng/southeastwater-usage-scraper","last_synced_at":"2026-02-06T15:05:37.678Z","repository":{"id":242016700,"uuid":"808460067","full_name":"longzheng/southeastwater-usage-scraper","owner":"longzheng","description":"Extract hourly water usage data from South East Water portal website for digital water meters","archived":false,"fork":false,"pushed_at":"2024-11-17T22:15:26.000Z","size":141,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-11T07:06:13.752Z","etag":null,"topics":["australia","data","iot","playwright","southeastwater","victoria","water"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/longzheng.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-31T05:49:15.000Z","updated_at":"2024-11-17T22:16:39.000Z","dependencies_parsed_at":"2024-06-16T05:05:38.160Z","dependency_job_id":null,"html_url":"https://github.com/longzheng/southeastwater-usage-scraper","commit_stats":null,"previous_names":["longzheng/southeastwater-usage-scraper"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/longzheng%2Fsoutheastwater-usage-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/longzheng%2Fsoutheastwater-usage-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/longzheng%2Fsoutheastwater-usage-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/longzheng%2Fsoutheastwater-usage-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/longzheng","download_url":"https://codeload.github.com/longzheng/southeastwater-usage-scraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233758873,"owners_count":18725815,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["australia","data","iot","playwright","southeastwater","victoria","water"],"created_at":"2024-11-14T00:44:51.591Z","updated_at":"2026-02-06T15:05:37.625Z","avatar_url":"https://github.com/longzheng.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Automatically scrape/extract South East Water water usage data\n\nSouth East Water introduced [digital water meters](https://southeastwater.com.au/residential/upgrades-and-projects/projects/digital-water-meters/) which record water usage data every hour. This data is available to customers through the South East Water website however the website only displays hourly data on a day-by-day basis and does not provide any export or APIs to access the data across a long time range.\n\nThis script will automatically scrape the water usage data from the South East Water website and save it to a CSV file (and optionally InfluxDB).\n\nThe script uses [Playwright](https://playwright.dev/) and a headless browser to\n- log in to the https://my.southeastwater.com.au/ customer portal\n- navigate to the usage page\n- click on the \"daily\" tab\n- go to the latest day (usually data is available up to yesterday)\n- go back day by day over the last 30 days scrape/extract the data from the network responses\n- save the data to a `usage.csv` CSV file\n\nExample\n```csv\nTimestamp,Date,Hour,MeasurementLitres\n2024-06-05T00:00:00,2024-06-04,0,4\n2024-06-05T01:00:00,2024-06-04,1,3\n2024-06-05T02:00:00,2024-06-04,2,4\n2024-06-05T03:00:00,2024-06-04,3,4\n```\n\n### How to use\n\n\u003e [!CAUTION]\n\u003e This is a proof of concept, is not thoroughly tested and may break if the South East Water website changes. Use at your own risk.\n\n1. Install dependencies `npm install`\n1. Copy `.env.example` to `.env` and fill in South East Water account email and password (and optionally InfluxDB connection details)\n1. Run `npm run start`\n\n### InfluxDB\n\nIf configured, the script will write records with\n- measurement: `water`\n- timestamp: the hour of the reading\n- field: `litres` with the water usage in litres\n\nYou can query this data with Flux, for example to get a report of the average water usage by hour of the day and the days of the week\n\n```flux\nimport \"date\"\n\ndays_of_week = [\"Sunday\", \"Monday\", \"Tuesday\", \"Wednesday\", \"Thursday\", \"Friday\", \"Saturday\"]\n\nfrom(bucket: \"bucket\")\n  |\u003e range(start: v.timeRangeStart, stop: v.timeRangeStop)\n  |\u003e filter(fn: (r) =\u003e r._measurement == \"water\")\n  |\u003e map(fn: (r) =\u003e ({\n    r with \n    day_of_week: date.weekDay(t: r._time), \n    hour_of_day: date.hour(t: r._time)\n  }))\n  |\u003e map(fn: (r) =\u003e ({\n    r with \n    day: days_of_week[r.day_of_week]\n  }))\n  |\u003e group(columns: [\"day\", \"hour_of_day\"])\n  |\u003e mean(column: \"_value\")\n  |\u003e group()\n  |\u003e pivot(\n    rowKey: [\"hour_of_day\"],\n    columnKey: [\"day\"],\n    valueColumn: \"_value\"\n  )\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flongzheng%2Fsoutheastwater-usage-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flongzheng%2Fsoutheastwater-usage-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flongzheng%2Fsoutheastwater-usage-scraper/lists"}