{"id":19572647,"url":"https://github.com/const-void/covid-19-spreadsheet","last_synced_at":"2026-05-15T00:11:46.081Z","repository":{"id":211434991,"uuid":"322711773","full_name":"const-void/covid-19-spreadsheet","owner":"const-void","description":"Send Outlook email w/covid-19 XLSX attachments, using New York Times as data source.","archived":false,"fork":false,"pushed_at":"2021-01-30T13:51:02.000Z","size":562,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-09T03:41:55.634Z","etag":null,"topics":["covid-19","covid-19-data","nytimes"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/const-void.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-18T21:47:16.000Z","updated_at":"2021-01-30T13:51:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"4a068dbd-ec9a-4831-a74e-d3bfe3825b79","html_url":"https://github.com/const-void/covid-19-spreadsheet","commit_stats":null,"previous_names":["const-void/covid-19-spreadsheet"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/const-void%2Fcovid-19-spreadsheet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/const-void%2Fcovid-19-spreadsheet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/const-void%2Fcovid-19-spreadsheet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/const-void%2Fcovid-19-spreadsheet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/const-void","download_url":"https://codeload.github.com/const-void/covid-19-spreadsheet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240840103,"owners_count":19866167,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["covid-19","covid-19-data","nytimes"],"created_at":"2024-11-11T06:27:54.212Z","updated_at":"2026-05-15T00:11:46.047Z","avatar_url":"https://github.com/const-void.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About\nSend Outlook email w/covid-19 XLSX attachments, using New York Times as data source.\n* Configurable spreadsheets complete with graphs \n    * Mix/match states \u0026 counties via json file\n    * US summary by state\n    * State summary by county \n\n# Quick install \u0026 execution\n```\n$ git clone https://github.com/const-void/covid-19-spreadsheet\n$ cd covid19-19-spreadsheet\n$ pip install -r requirements.txt\n$ python ./covid19_data_gather.py\n$ python ./covid19_data_gather.py \u003c/path/to/conf.json\u003e\n```\n\nStarting from a blank slate, everything should just work - cloning NYT data, creating a sample conf file for you to edit.  It should just work, with any luck.\n\n# Goal\nI was pretty frusterated by how hard it was to answer simple questions--how many people HAVE covid, right now?  How does where I live, compare to where others live?  Are we doing ok, as a county, state, country? When I found out about the NYT data store, I jumped on it IMMEDIATELY.\n\nI also wanted to facilitate  analysis - so while emails are created automatically, they are not sent; the idea is, I, as a sender review, covid-19 data and makes some sort of conclusion.\n\nOtherwise...why generate spreadsheets at all?  If we aren't looking at data, thinking about what we see, sharing our thoughts with others, it becomes a mindless data generation task; there are too many spreadsheet generators, dumping data in a directory, never to be looked at, \"just in case\".\n\n## Objectives\n1. Align data by *days* not *date*.  If one state catches covid in March, and another in June, I want to see how those states are doing compared to each other...days since contagion aligns peaks / valleys, and is more important than the *dates* of infection.\n\n2. Graphs.  Simple graphs of all the key measures.\n\n3. Educated guesses when the data is incomplete.  We can't really know for *sure* how many active cases are walking around. But, we can infer, a conservative estimate that the average case is done within 28 days - including days *prior* to a positive test, and lingering effects afterwards.\n\n4. Trends...how are we doing?  No rosy picture...if we are getting better, and continue to get better at that rate, when does it end?  If we are doing worse, how much worse?\n\n# Dependencies\n* Git - https://git-scm.com/downloads  (for New York Times sync) \n* Python 3 -  https://www.python.org/downloads/ or https://www.microsoft.com/en-us/p/python-38/\n* pip c/o [requirement.txt](https://pip.pypa.io/en/stable/user_guide/#requirements-files):\n    * openpyxl - native spreadsheet generation\n    * gitpython - nyt covid-19-data git repo clone / pull\n    * fastjsonschema - json configuration file  \n    * pywin32 - Outlook email\n\n# Optional Packages\n* Microsoft Visual Studio Code - IDE ( https://code.visualstudio.com/Download )\n* Microsoft Outlook - Send emails\n* Microsoft Excel - Viewing generated spreadsheets\n\n# Windows Install \u0026 Execution (VS Code)Execution\n1. Download \u0026 Install git \n2. Download \u0026 Install Python 3 from Windows App Store\n3. Start Microsoft Visual Studio Code, [install git / python extensions](https://code.visualstudio.com/docs/python/python-tutorial#_prerequisites)\n4. [Clone repo](https://code.visualstudio.com/docs/editor/github#_setting-up-a-repository) https://github.com/const-void/covid-19-spreadsheet: -- \n5. Menu: Terminal, [New Terminal](https://code.visualstudio.com/docs/editor/integrated-terminal)\n6. In terminal, `~\\AppData\\Local\\Microsoft\\WindowsApps\\pip.exe install -r requirements.txt`\n7. File, Open `covid19_data_gather.py`\n8. Menu: Run, Run without Debugging\n9. File, Open `covid19_data_gather_conf.json`\n10. Edit! Go to step 8. \n\n# Windows Install \u0026 Execution (Powershell)\n1. Download \u0026 Install git \n2. Download \u0026 Install Python 3 from Windows App Store\n3. Start Powershell\n*One time install*\n```\nPS \u003e cd Documents  (or any location)\nPS \u003e git clone https://github.com/const-void/covid-19-spreadsheet\nPS \u003e cd covid-19-spreadsheet\nPS \u003e ~\\AppData\\Local\\Microsoft\\WindowsApps\\pip.exe install -r requirements.txt\n```\n\n*Execution*\n```\nPS \u003e python .\\covid19_data_gather.py\nPS \u003e python .\\covid19_data_gather.py \u003cpath\\to\\conf.json\u003e\nPS \u003e python .\\covid19_data_gather.py sample_coivd19_data_gether_conf.json\n```\n\n# JSON Configuration\nCase sensitive!\n\n`cov19_data_gather.py` without a command line parameter will look for a `covid19_data_gather_conf.json` file. \n\nIf one isn't found, one will be made for you! [`sample_covid19_data_gather_conf.json`](https://github.com/const-void/covid-19-spreadsheet/blob/master/sample_covid19_data_gather_conf.json) is used as starting place. \n\nNote that `covid19_data_gather_conf.json` is in [`.gitignore`](https://github.com/const-void/covid-19-spreadsheet/blob/master/.gitignore) -- this way you can have your own setup without having to worry about git collisions etc.  Alternatively, a given `/path/to/conf.json` can specified on the command line.\n\nAll json configurations are validated -- once at a schema level via [`covid19_data_gather_conf.schema.json`](https://github.com/const-void/covid-19-spreadsheet/blob/master/covid19_data_gather_conf.schema.json), and again, to make sure the intended geographies are accurate - counties have to match both US \u0026 NYT data *exactly*, including case sensitivity.  \n\n`los angeles, ca` will *fail* validation.  `Los Angeles County, CA` will pass.\n\nThere are two blocks, both required:\n* `\"spreadsheets\": {}`\n* `\"settings\": {}`\n\n## `\"spreadsheets\": {}`\nThis block controls spreadsheet generation. Fiddle with these settings to hone in on geographies of interest. If an email will be sent, each spreadsheet generated is attached.\n\nFor a while, I was generating spreadsheets and not sending...but...what's the point?  The data isn't going anywhere.  When we want to look at a region -- add it in! When we tire of it...take it out!\n\nkey | type | required | desc | example\n--- | ---- | -------- | ---- | -------\nus  | Boolean | Yes | Controls generation of summary (US) level spreadsheet, consisting of all 50 states! | `\"us\": true`\nstate-detail | Array of states | Yes | Controls generation of summary (state) level spreadsheets, consisting of each county in a given state.  One spreadsheet per state is generated. | `\"state-detail\": ['ND', 'SD']`\ncustom | List | Yes | List of spreadsheets to generate. each property is a  spreadsheet name; property value is a state or a county, state.  One spreadsheet per property is generated. | `\"custom\": { \"north_dakota\": [ \"Burleigh County, ND\", \"ND\" ] }`\n\n## \"settings\": {}\nThis configuration block controls the script itself.  Fiddle with these settings to change the data we see.  Don't like my take of a 28 day average case...lower it.  Or, raise it.  what happens?   Want to change the comparative per scales -- is 100k too big? Too small?  Or do we want to exclude vast swathes of the country?  The below settings allow you to do *just* that--manipulate the data as *you* see fit.\n\n*Data Controls*\nkey | type | required | desc | example\n--- | ---- | -------- | ---- | -------\ncase-min-benchmark | Number | Yes | Minimum number of cases; acts a a reporting gate. If we want to eliminate low caseload geographies, we set this property to filter to just the caseloads we are interested in--say those at 100,000 or more, or even--minimum of 10, 1000.  | `\"case-min-benchmark\": 1`\ncase-days-duration | Number | Yes | Average case duration - used in active vs recovered calculations. | `\"case-days-duration\": 28`\ngeography-per-county | Number | Yes | Scaling factor for counties. Per capita is a value of one; cdc uses 100k. | `\"geography-per-county\": 100000`\ngeography-per-state | Number | Yes |  Scaling factor for states. Per capita is a value of one; cdc uses 100k. | `\"geography-per-state\": 100000`\n\n*Email Settings*\nkey | type | required | desc | example\n--- | ---- | -------- | ---- | -------\nsend-email | Boolean | Yes | Send email if true | `\"send-email\": true`\nsend-email-client | Enumeration | No | One of `Outlook` or `N/A`, sadly. | `\"send-email-client\": \"Outlook\"`\nsend-email-to | Array of emails | No | List of email addresses to send to. | `\"send-email-to\": [ \"a@bc.com\" ]`\nsend-email-style | String | No | HTML styling for a swank email. | `\"send-email-style\": \"font-family: Trebuchet MS; color:#25253b; font-size:14pt\"`\nsend-email-greeting | String | No | HTML email greeting | `\"send-email-greeting\": \"Hello!\u003cbr\u003e\"`\nsend-email-signature | String | No | HTML signaure | `\"send-email-signature\": \"xoxo\u003cbr\u003eYours Truly!\"`\n\n# Spreadsheet Notes\nTo do\n\n# [Covid-19 Data](https://github.com/nytimes/covid-19-data/blob/master/us-counties.csv)\nc/o The New York Times. (2020). Coronavirus (Covid-19) Data in the United States. Retrieved from https://github.com/nytimes/covid-19-data.\n\n*Usage*\n* The function `update_data` sync's Covid-19 via `git pull` into a sibiling directory c/o https://github.com/nytimes/covid-19-data :\n```\ncovid-19-data/\n  us-counties.csv\ncovid-19-spreadsheet/\n  xlsx/\n     \u003cgenerated ouutputs\u003e\n```\n* If the sibiling directory/repository is missing, it is created via a `git clone`\n* The function `set_county_covid19_cases` loads covid-19 data into Counties, joining via `fips`. \n\nCol #  | Field Name  | Desc | Sample \n------ | ----------- | ---- | ------\n1 | Date | | `2020-01-21`\n2 | County | | `Snohomish`\n3 | State | | `Washington`\n4 | Fips | State FIPS + County FIPS | `53061`\n5 | Cases | | `1`\n6 | Deaths | | `0`\n\n# Geography Data\nWhen NYT started reporting on covid-19, each day would introduce a slew of new US geographies.  Initially, I thought maybe I could pull in geography data as it was found--but, as data came in, the performance impact just got to be too great. It is important to cache geograpy in adavnce - so that as new geography data comes pouring in, it has a place to go.\n\nAnd that meant...source US geography. \n\nFrom [Census.gov](https://www.census.gov/geographies/reference-files/2018/demo/popest/2018-fips.html):\n## [all-geocodes-v2018.csv](https://github.com/const-void/covid-19-spreadsheet/blob/master/all-geocodes-v2018.csv)\n[spreadsheet](https://www2.census.gov/programs-surveys/popest/geographies/2018/all-geocodes-v2018.xlsx) =\u003e csv.\n\n**Country Data**\n* Estimates Geography File: Vintage 2018\n* Source: U.S. Census Bureau, Population Division\n* Internet Release Date: May 2019\n\n*Usage*\n* Loaded by `Counties` constructor to create individual `County` objects.\n* Used to join County to State **( State Code (FIPS) )**\n* Used to join NYT Covid-19 data to County **( NYT FIPS Code = State Code (FIPS) + County Code (FIPS) )**\n\nCol #  | Field Name  | Desc | Sample \n------ | ----------- | ---- | ------\n1 | Summary Level | | `050`\n2 | State Code (FIPS) | | `01`\n3 | County Code (FIPS) | | `001`\n4 | County Subdivsion Code (FIPS) | | `00000`\n5 | Place Code (FIPS) | | `00000`\n6 | Consolidated City Code (FIPS) | | `00000`\n7 | Area Name | | `Autauga County`\n\n## [state-geocodes-v2018.csv](https://github.com/const-void/covid-19-spreadsheet/blob/master/state-geocodes-v2018.csv)\n[spreadsheet](https://www2.census.gov/programs-surveys/popest/geographies/2018/state-geocodes-v2018.xlsx) =\u003e csv\n\n**State Data**\n* Source: U.S. Census Bureau, Population Division\n* Internet Release Date: May 2019\n\n*Usage*\n* Loaded by `States` constructor to create individual `State` objects.\n* Used to join State to County  **( State Code (FIPS) )**\n\nCol # | Field Name | Desc | Sample\n----- | ---------- | ---- | ------\n1 | Region  | | `1`\n2 | Division | | `1`\n3 | State (FIPS ) | | `09`\n4 | Name | | `Connecticut`\n\n# Population\nSimply knowing geography and case wasn't enough--I wanted to know the sense of scale, and not \njust within a geography...but also *across* geographies.  How do counties compare to other counties? And states?\n\n## [co-est2019-annres.csv](https://github.com/const-void/covid-19-spreadsheet/blob/master/co-est2019-annres.csv)\n[src](https://www.census.gov/newsroom/press-kits/2020/pop-estimates-county-metro.html) [spreadsheet](https://www2.census.gov/programs-surveys/popest/tables/2010-2019/counties/totals/co-est2019-annres.xlsx) =\u003e csv\n\n**County Population Estimates**\n* Annual Estimates of the Resident Population for Counties in the United States: April 1, 2010 to July 1, 2019 (CO-EST2019-ANNRES)\n* Source: U.S. Census Bureau, Population Division\t\n* Release Date: March 2020\t\n\n*Usage*\n* Loaded by `set_county_population` function\n* Joins to county \u0026 state via name\n\t\t\t\t\t\t\t\nCol # | Field Name | Desc | Sample\n----- | ---------- | ---- | ------\n1 | Geographic Area | | `\".Autauga County, Alabama\"`\n2 | Census | | `54571`\n3 | Estimates Base | | `54597`\n4 | 2010 | | `54773`\n5 | 2011 | | `55227`\n6 | 2012 | | `54954`\n7 | 2013 | | `54727`\n8 | 2014 | | `54893`\n9 | 2015 | | `54864`\n10 | 2016 | | `55243`\n11 | 2017 | | `55390`\n12 | 2018 | | `55533`\n13 | 2019 | | `55869`\n\n## [nst-est2019-01.csv](https://github.com/const-void/covid-19-spreadsheet/blob/master/nst-est2019-01.csv)\n[src](https://www.census.gov/data/tables/time-series/demo/popest/2010s-national-total.html\n) [spreadsheet](https://www2.census.gov/programs-surveys/popest/tables/2010-2019/state/totals/nst-est2019-01.xlsx) =\u003e csv\n\n**State Population Estimates**\n* Table 1. Annual Estimates of the Resident Population for the United States, Regions, States, and Puerto Rico: April 1, 2010 to July 1, 2019 (NST-EST2019-01)\t\t\t\t\t\t\t\t\t\t\t\t\n* Source: U.S. Census Bureau, Population Division\t\t\t\t\t\t\t\t\t\t\t\t\n* Release Date: December 2019\t\n\n*Usage* \t\t\t\t\t\n* Loaded by `States` contstructor to assign population to each U.S state\n* State name joins population to state\n\nCol # | Field Name | Desc | Sample\n----- | ---------- | ---- | ------\n1 | Geographic Area | | `Alabama`\n2 | Census | | `4779736`\n3 | Estimates Base | | `4780125`\n4 | 2010 | | `4785437`\n5 | 2011 | | `4799069`\n6 | 2012 | | `4815588`\n7 | 2013 | | `4830081`\n8 | 2014 | | `4841799`\n9 | 2015 | | `4852347`\n10 | 2016 | | `4863525`\n11 | 2017 | | `48744861`\n12 | 2018 | | `4887681`\n13 | 2019 | | `4903185`\n\n# Email Notes\nEmail is a bit limited.  Currently - no osx/linux email, and Windows email is via Outlook.\n\nOutlook is standard in the enterprise world, but not at home.  I haven't figured out Win10 mail yet, and I don't have a Mac/Linux box to use.  I am unwilling to smtp - so clients TBR.\n\n# TO DO \n* US Territories (Guam, Puerto Rico, etc)\n* Win10 Mail\n* Apple Mail\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconst-void%2Fcovid-19-spreadsheet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconst-void%2Fcovid-19-spreadsheet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconst-void%2Fcovid-19-spreadsheet/lists"}