https://github.com/danp/outages-to-sqlite

Convert scraped NS Power outage data to a SQLite database
https://github.com/danp/outages-to-sqlite

geojson git-scraping golang sqlite

Last synced: 26 days ago
JSON representation

Convert scraped NS Power outage data to a SQLite database

• Host: GitHub
• URL: https://github.com/danp/outages-to-sqlite
• Owner: danp
• License: bsd-3-clause
• Created: 2021-01-24T18:03:29.000Z (almost 4 years ago)
• Default Branch: main
• Last Pushed: 2022-03-06T19:40:45.000Z (over 2 years ago)
• Last Synced: 2024-06-21T16:54:55.989Z (5 months ago)
• Topics: geojson, git-scraping, golang, sqlite
• Language: Go
• Homepage:
• Size: 2.24 MB
• Stars: 2
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# outages-to-sqlite

A very work-in-progress program to convert

[scraped Nova Scotia Power outage data](https://github.com/danp/nspoweroutages)

to a [SQLite](https://sqlite.org) database.

The result is currently visible at https://nsp.datasette.danp.net/outages

(powered by [Datasette](https://datasette.io)).

In short, this program:

1. Reads each commit of https://github.com/danp/nspoweroutages, cloning it in memory by default, can be configured with `-repo-remote ` or `-repo-path `

1. Parses the `data/outages.json` file

1. Uses the `geom.p` value (which every outage seems to have) as a key to determine which outages are new, ongoing, or gone

1. Uses data in the [places](places) directory to map the `geom.p` value to a place (currently at the "county" level)

1. Emits events to a sqlite database, `outages.db` by default but can be specified with `-database-file `

Once the database exists, subsequent runs will fetch the last observed time from the database and

read commits from then on, picking up where it left off.

Everything about this is subject to change!