{"id":17531002,"url":"https://github.com/danp/nspoweroutages","last_synced_at":"2026-03-14T22:12:13.192Z","repository":{"id":37206079,"uuid":"324650197","full_name":"danp/nspoweroutages","owner":"danp","description":"Git scraping of the Nova Scotia Power Outage Map","archived":false,"fork":false,"pushed_at":"2026-03-10T06:24:43.000Z","size":240406,"stargazers_count":10,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-03-10T07:56:48.674Z","etag":null,"topics":["git-scraping"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-12-26T23:09:45.000Z","updated_at":"2026-03-10T06:24:46.000Z","dependencies_parsed_at":"2023-12-27T04:46:51.043Z","dependency_job_id":"28148d3f-11d4-4b40-bfbc-3b28c4e3a04f","html_url":"https://github.com/danp/nspoweroutages","commit_stats":{"total_commits":110507,"total_committers":2,"mean_commits":55253.5,"dds":"0.00014478720805022327","last_synced_commit":"b618316a6545f5df0f95392c66017d4949a0c11a"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/danp/nspoweroutages","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danp%2Fnspoweroutages","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danp%2Fnspoweroutages/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danp%2Fnspoweroutages/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danp%2Fnspoweroutages/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danp","download_url":"https://codeload.github.com/danp/nspoweroutages/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danp%2Fnspoweroutages/sbom","scorecard":{"id":321568,"data":{"date":"2025-08-11","repo":{"name":"github.com/danp/nspoweroutages","commit":"cf49a833bf6a66edd3ae9c3a46e2f9fdf7e223d2"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.6,"checks":[{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Maintained","score":10,"reason":"30 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":0,"reason":"license file not detected","details":["Warn: project does not have a license file"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":0,"reason":"11 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2022-0288","Warn: Project is vulnerable to: GO-2022-0969 / GHSA-69cg-p879-7622","Warn: Project is vulnerable to: GO-2022-1144 / GHSA-xrjj-mj9h-534m","Warn: Project is vulnerable to: GO-2023-1571 / GHSA-vvpx-j8f3-3w6h","Warn: Project is vulnerable to: GO-2023-1988 / GHSA-2wrh-6pvc-2jm9","Warn: Project is vulnerable to: GO-2023-2102 / GHSA-4374-p667-p6c8","Warn: Project is vulnerable to: GHSA-qppj-fm5r-hxr3","Warn: Project is vulnerable to: GO-2024-2687 / GHSA-4v7x-pqxf-cx7m","Warn: Project is vulnerable to: GO-2024-3333","Warn: Project is vulnerable to: GO-2025-3503 / GHSA-qxp5-gwg8-xv66","Warn: Project is vulnerable to: GO-2025-3595 / GHSA-vvgc-356p-c3xw"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-18T01:29:29.667Z","repository_id":37206079,"created_at":"2025-08-18T01:29:29.667Z","updated_at":"2025-08-18T01:29:29.667Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30519346,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-14T19:51:21.629Z","status":"ssl_error","status_checked_at":"2026-03-14T19:51:12.959Z","response_time":57,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["git-scraping"],"created_at":"2024-10-20T17:14:22.893Z","updated_at":"2026-03-14T22:12:13.165Z","avatar_url":"https://github.com/danp.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nspoweroutages\n\n[Git scraping](https://simonwillison.net/2020/Oct/9/git-scraping/) of the data behind the [Nova Scotia Power Outage Map](http://outagemap.nspower.ca/).\n\nMainly viewable at https://nsp.datasette.danp.net/ by way of https://github.com/danp/outages-to-sqlite.\n\n## Why?\n\nThe outage map shows what's happening _right now_ but gives no way to see information about past outages, what has changed, etc. Regularly fetching the data that powers it and saving it in this way will let us see changes over time. That could let us answer questions like:\n\n* What are the most commonly labeled causes of outages?\n* What areas have frequent outages?\n* How long do outages typically last?\n\nAnd more!\n\n## How it works\n\nEvery 10 minutes, a system of mine runs [bin/scrape.sh](bin/scrape.sh).\n\nbin/scrape.sh:\n\n1. fetches and combines outage data using [cmd/scrape-outages](cmd/scrape-outages), saving it to [data/outages.json](data/outages.json)\n2. fetches the service area summary data, saving it to [data/report_servicearea.json](data/report_servicearea.json)\n\nIf there any changes from the current data, they're committed and pushed, like\n[this](https://github.com/danp/nspoweroutages/commit/8183bda3b32f572e541caa6cd839b0d60b36bfba).\n\n## Data format\n\n`data/outages.json` has roughly this format:\n\n```\n[\n  {\n    \"desc\": {\n      \"cause\": \"\u003coutage cause, such as Under Investigation, Damage to Equipment, High Winds\u003e\",\n      \"cluster\": true|false, # whether this entry represents a number of outages or not\n      \"cust_a\": { # customer affected info\n        \"masked\": true|false, # when true, map UI would show \"fewer than \u003cval + 1\u003e affected customers\"\n        \"val\": 5 # number of customers affected\n      },\n      \"n_out\": 3, # for cluster=true, how many outages this cluster covers\n      \"outages\": [ # for cluster=true, individual outages in this cluster\n        { \"cause\": \"...\", ... } # same structure as under `desc` here\n      ],\n      \"etr\": \"2020-12-26T22:15:00-0400\", # estimated time of recovery\n      \"start\": \"2020-12-26T14:07:00-0400\" # start of outage\n    },\n    \"geom\": { # affected areas and/or points\n      \"a\": [ # affected area(s)\n        \"en`nG~nbrKqp@qIoxBe{AoG}FaLs^nzDtpAbDr{@\" # polyline encoding\n      ],\n      \"p\": [ # affected point(s)\n        \"gowuGjud~J\" # polyline encoding\n      ]\n    },\n    \"id\": \"5\",\n    \"title\": \"\u003cOutage Information or Area Outage\u003e\"\n  },\n  ...\n]\n```\n\nUnfortunately, `id` is not maintained across updates. This means when going from 5 outages to 4, ids will likely be shuffled, making it difficult to determine which outage was removed.\n\n`geom.a` and `geom.p` are in [polyline encoding](https://developers.google.com/maps/documentation/utilities/polylinealgorithm) and can be decoded with eg [polyline.DecodeCoords](https://pkg.go.dev/github.com/twpayne/go-polyline#DecodeCoords) or interactively using the [Interactive Polyline Encoder Utility](https://developers.google.com/maps/documentation/utilities/polylineutility).\n\n## How data fetching works\n\ncmd/scrape-outages starts by fetching [metadata.json](http://outagemap.nspower.ca/resources/data/external/interval_generation_data/metadata.json) which contains a single `directory` key pointing to the current data directory. At the time of this writing, it looks like this:\n\n```json\n{ \"directory\": \"2020_12_27_19_18_00\" }\n```\n\nThis means the current data can be found under http://outagemap.nspower.ca/resources/data/external/interval_generation_data/2020_12_27_19_18_00/.\n\ncmd/scrape-outages then fetches what I assume are the six data files for the map tiles visible when all of Nova Scotia is visible in the outage map, eg `2020_12_27_19_23_00/outages/030322.json` which contains data near HRM. Files are named based on their zoom level, the longer the filename the more zoomed in the data is.\n\nThese files have mostly the same format as the combined `outages.json` describe above.\n\ncmd/scrape-outages zooms in to the areas covered by the initial six files by appending `0`, `1`, `2`, and `3` to the filename, then fetching that. For example, `030322.json` would zoom in to:\n\n* `0303220.json`\n* `0303221.json`\n* `0303222.json`\n* `0303223.json`\n\nIt continues to zoom in until either a deeper level returns a 404 (no data) or there only outages (and no clusters) in the returned data.\n\nFinally, it combines all the data from the deepest possible levels into the single `outages.json`.\nThe resulting file may still have clusters in it if the deepest-found level contained them (example [here](https://github.com/danp/nspoweroutages/blob/d0cbdac3e322e22cc2b9f8b4ab213f10edff6e98/data/outages.json#L25-L30)).\n\n(This is all based on observing what happens in the [Firefox Network Monitor](https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor))\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanp%2Fnspoweroutages","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanp%2Fnspoweroutages","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanp%2Fnspoweroutages/lists"}