{"id":13411696,"url":"https://github.com/knudmoeller/berlin_corona_cases","last_synced_at":"2026-01-20T05:41:25.131Z","repository":{"id":39863424,"uuid":"269634867","full_name":"knudmoeller/berlin_corona_cases","owner":"knudmoeller","description":"Scraper for the official dashboard with current Corona case numbers, traffic light indicators (\"Corona-Ampel\") and vaccination situation for Berlin.","archived":false,"fork":false,"pushed_at":"2023-08-24T08:21:41.000Z","size":2382,"stargazers_count":19,"open_issues_count":1,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-07-31T20:48:37.748Z","etag":null,"topics":["berlin","corona","covid19","dataset","git-scraping","nokogiri","opendata","ruby","scraper","vaccination"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/knudmoeller.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-05T12:52:31.000Z","updated_at":"2024-01-12T18:13:36.000Z","dependencies_parsed_at":"2024-10-26T07:58:13.711Z","dependency_job_id":"f8f73844-6887-4f26-a746-132118e0e059","html_url":"https://github.com/knudmoeller/berlin_corona_cases","commit_stats":null,"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knudmoeller%2Fberlin_corona_cases","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knudmoeller%2Fberlin_corona_cases/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knudmoeller%2Fberlin_corona_cases/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knudmoeller%2Fberlin_corona_cases/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/knudmoeller","download_url":"https://codeload.github.com/knudmoeller/berlin_corona_cases/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243618656,"owners_count":20320272,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["berlin","corona","covid19","dataset","git-scraping","nokogiri","opendata","ruby","scraper","vaccination"],"created_at":"2024-07-30T20:01:15.884Z","updated_at":"2026-01-20T05:41:25.125Z","avatar_url":"https://github.com/knudmoeller.png","language":"Ruby","funding_links":[],"categories":["Ruby","ruby"],"sub_categories":[],"readme":"# Coronavirus Cases Scraper for Berlin\n\n![logo for \"Coronavirus Cases Scraper for Berlin\" project](logo/corona_cases_berlin_small.png)\n\n### Quick Links\n\n* [Timeline of case numbers](data/target/berlin_corona_cases.json)\n* [Timeline of traffic light indicators (\"Corona Ampel\")](data/target/berlin_corona_traffic_light.json)\n\nThis is a scraper for the [Corona/COVID-19 dashboard for Berlin](https://www.berlin.de/corona/lagebericht/desktop/corona.html), as issued by the [Senatsverwaltung für Gesundheit, Pflege und Gleichstellung](https://www.berlin.de/sen/gpg/) (Senate Department for Health, Care and Equality) and the [Landesamt für Gesundheit und Soziales](https://www.berlin.de/lageso/) (Regional Office for Health and Social Affairs).\nThe dashboard includes daily case numbers by district and age groups, as well as the \"Corona traffic light\"-indicators (incidence of new infections per week, ICU occupancy rate, relative change in incidence).\n\n* Change in the 7-day incidence was dropped again on September 2nd, 2021.\nInstead, the 7-day indicence for hospitalisation was introduced as a new traffic light indicator on that day.\nAlso, the color scheme for the incidence traffic light was adjusted: where before it was yellow at \u003e=20% and red at \u003e=40%, it is now yellow at \u003e=35% and red at \u003e=100%.\nThis of course makes it pointless to compare the color values before and after 2021-09-02.\n* The basic reproduction number R was included until July 22nd, 2021.\nAfter that, [R was dropped because it was no longer deemed a useful indicator](https://data.lageso.de/lageso/corona/archiv/berlin-website-2021-07-23.html \"Berliner Corona-Lagebericht vom 23.07.2021\").\nInstead, the relative change in incidence is now the third indicator in the corona traffic light.\nR is still included in the output data (as `0.0`), since some apps might expect to find it there.\n* Starting with February 15, 2021, the absolute numbers and percentages for administered vaccinations are included in the dashboard.\n* Starting with November 11, 2020, the change in the 7-day incidence is also included in the traffic light indicators.\n* As of August 31, 2020, the dashboard [replaces the previously used daily press releases](https://www.berlin.de/sen/gpg/service/presse/2020/pressemitteilung.983232.php \"SenGPG press release announcing the new Corona/COVID-19 dashboard from 2020-08-31\") containing the same data.\nThere were two separate press releases each day, one with the case numbers (see [here](https://www.berlin.de/sen/gpg/service/presse/2020/pressemitteilung.982684.php \"Last SenGPG press release with Corona/COVID-19 case numbers for Berlin from 2020-08-30\") for the last one) and one with the traffic light indicators (see [here](https://www.berlin.de/sen/gpg/service/presse/2020/pressemitteilung.982682.php \"Last SenGPG press release with Corona/COVID-19 traffic light indicators for Berlin from 2020-08-30\") for the last one of those).\n\n\nThe output of the scraper are timelines of data extracted from the individual press releases.\n\n## Output Data\n\n_Note that there is no data for 2021-05-06, because on that day, the reporting was changed from same-day-evening to next-day-noon.\nSee [this note](note20210506.md) for more information._\n\n### Corona Case Numbers\n\nThe timeline data generated by the case number scraper is a JSON file in [data/target/berlin_corona_cases.json](data/target/berlin_corona_cases.json), structured as follows:\n\n```json\n[\n  {\n    \"date\": \"2020-09-23\",\n    \"source\": \"https://www.berlin.de/corona/lagebericht/desktop/corona.html\",\n    \"counts_per_district\": {\n      \"lor_01\": {\n        \"case_count\": 2141,\n        \"indicence\": 555.0,\n        \"recovered\": 1894\n      },\n      \"lor_02\": {\n        \"case_count\": 1359,\n        \"indicence\": 468.0,\n        \"recovered\": 1176\n      },\n      ...\n      \"lor_12\": {\n        \"case_count\": 974,\n        \"indicence\": 365.6,\n        \"recovered\": 897\n      }\n    },\n    \"counts_per_age_group\": {\n      \"0-4\": {\n        \"case_count\": 358,\n        \"incidence\": 188.7\n      },\n      \"5-9\": {\n        \"case_count\": 406,\n        \"incidence\": 236.5\n      },\n      ...\n      \"90+\": {\n        \"case_count\": 156,\n        \"incidence\": 500.6\n      },\n      \"unknown\": {\n        \"case_count\": 15,\n        \"indidence\": \"n.a.\"\n      }\n    }\n  },\n  {\n    \"date\": \"2020-09-22\",\n    \"source\": \"https://www.berlin.de/corona/lagebericht/desktop/corona.html\",\n    \"counts_per_district\": {\n      \"lor_01\": {\n        \"case_count\": 2116,\n        \"indicence\": 548.5,\n        \"recovered\": 1877\n      },\n      \"lor_02\": {\n        \"case_count\": 1330,\n        \"indicence\": 458.0,\n        \"recovered\": 1154\n      },\n      ...\n      \"lor_12\": {\n        \"case_count\": 967,\n        \"indicence\": 363.0,\n        \"recovered\": 890\n      }\n    },\n    \"counts_per_age_group\": {\n      \"0-4\": {\n        \"case_count\": 356,\n        \"incidence\": 187.6\n      },\n      \"5-9\": {\n        \"case_count\": 399,\n        \"incidence\": 232.4\n      },\n      ...\n      \"90+\": {\n        \"case_count\": 155,\n        \"incidence\": 497.4\n      },\n      \"unknown\": {\n        \"case_count\": 15,\n        \"indidence\": \"n.a.\"\n      }\n    }\n  },\n  ...\n]\n```\n\nThe structure of the data is a JSON array with objects for each day.\nFor each day, the `source` is specified (where was the data scraped from – this used to be a particular press release, now it is always the dashboard), the `date` (of the data), the `counts_per_district` and the `counts_per_age_group`.\nFor the days scraped from individual press releases, there is also a `pr_date` attribute (when was the press release issued), because `pr_date` and `date` are not always the same day.\n\n#### Counts per District\n\nThe `counts_per_district` objects are structured with a key for each district, which in turn contain the actual numbers for the total `case_count`, `incidence` and number of `recovered` cases.\n\nThe district keys are their [LOR codes](data/manual/lor_district_codes.json) (see the dataset [Lebensweltlich orientierte Räume (LOR) in Berlin](https://daten.berlin.de/datensaetze/lebensweltlich-orientierte-räume-lor-berlin \"The dataset 'Lebensweltlich orientierte Räume (LOR) in Berlin' on the Berlin Open Data Portal\") for a complete definition of each LOR code):\n\n```json\n{\n    \"lor_01\": \"Mitte\",\n    \"lor_02\": \"Friedrichshain-Kreuzberg\",\n    \"lor_03\": \"Pankow\",\n    \"lor_04\": \"Charlottenburg-Wilmersdorf\",\n    \"lor_05\": \"Spandau\",\n    \"lor_06\": \"Steglitz-Zehlendorf\",\n    \"lor_07\": \"Tempelhof-Schöneberg\",\n    \"lor_08\": \"Neukölln\",\n    \"lor_09\": \"Treptow-Köpenick\",\n    \"lor_10\": \"Marzahn-Hellersdorf\",\n    \"lor_11\": \"Lichtenberg\",\n    \"lor_12\": \"Reinickendorf\"\n}\n```\n\n#### Counts per Age Group\n\nThe `counts_per_age_group` objects are similarly structured, with a key for each age group, which in turn contain the `case_count` and `incidence` (no `recovered`).\nThe age group `80+` was split into `80-89` and `90+` beginning May 11th (2020-05-11).\n\nThere is a special `unknown` age group for which the `incidence` is always `n.a.`\n\n#### Manually Extracted Data\n\nSome of the earlier press releases had a slightly different format, or were only available as screen shots (true story), so the [old press release scraper](https://github.com/knudmoeller/berlin_corona_cases/tree/0.2.4 \"Release 0.2.4 of the 'Coronavirus Cases Scraper for Berlin'\") did not work for them.\nRather than writing special code for extracting these one- or two-off cases, I manually extracted them and put them in [data/manual/manually_extracted.json](data/manual/manually_extracted.json).\nWhen creating the complete timeline, this manually extracted data was then merged with the newly scraped data.\n\n### Corona Traffic Light Indicators and Vaccination\n\nThe traffic light and vaccination data generated by the scraper is a JSON file located in [data/target/berlin_corona_traffic_light.json](data/target/berlin_corona_traffic_light.json). \nThis data contains a timeline of how the traffic light indicators changed over time.\n\nStarting February 15, 2021, the Corona dashboard includes data on vaccination. This has been added to the traffic light JSON file.\nWhile it is not strictly speaking one of the traffic light indicators, it is still an important indicator for the overall situation regarding the pandemic in Berlin.\n\nThere is a second JSON file in [data/target/berlin_corona_traffic_light.latest.json](data/target/berlin_corona_traffic_light.latest.json) which always contains the latest traffic light indicators.\n\nThe structure is as follows:\n\n```json\n[\n  {\n    \"source\": \"https://www.berlin.de/corona/lagebericht/desktop/corona.html\",\n    \"pr_date\": \"2021-09-02\",\n    \"indicators\": {\n      \"basic_reproduction_number\": {\n        \"color\": \"\",\n        \"value\": 0.0\n      },\n      \"incidence_new_infections\": {\n        \"color\": \"yellow\",\n        \"value\": 83.2\n      },\n      \"icu_occupancy_rate\": {\n        \"color\": \"yellow\",\n        \"value\": 5.4\n      },\n      \"change_incidence\": {\n        \"color\": \"\",\n        \"value\": 0.0\n      },\n      \"incidence_hospitalisation\": {\n        \"color\": \"green\",\n        \"value\": 1.3\n      }\n    },\n    \"vaccination\": {\n      \"total_administered\": 4503235,\n      \"percentage_one_dose\": 65.2,\n      \"percentage_two_doses\": 60.4\n    }\n  },\n  ...\n  {\n    \"source\": \"https://www.berlin.de/sen/gpg/service/presse/2020/pressemitteilung.982682.php\",\n    \"pr_date\": \"2020-08-30\",\n    \"indicators\": {\n      \"basic_reproduction_number\": {\n        \"value\": 1.1,\n        \"color\": \"green\"\n      },\n      \"incidence_new_infections\": {\n        \"value\": 12.5,\n        \"color\": \"green\"\n      },\n      \"icu_occupancy_rate\": {\n        \"value\": 1.5,\n        \"color\": \"green\"\n      }\n    }\n  },\n  ...\n]\n```\n\nThe structure of the data is a JSON array with objects for day.\nEach day specifies the `source` (where was the data scraped from – this used to be a particular press release, now it is always the dashboard), the `pr_date` (date when this particular set of indicators was announced – this used to be the date of the press release), an `indicators` object and a `vaccination` object (starting 2021-02-15).\n`indicators` in turn contains the indicators `incidence_new_infections` (incidence of new infections per 100,000 inhabitants per week) and `icu_occupancy_rate` (the ICU occupancy rate in %: which percentage of the available ICU capacity is currently being used).\nOn 2020-11-11 another indicator was introduced: the change in 7-day incidence (_\"Veränderung der 7-Tage-Inzidenz\"_).\nThis indicator is included as `change_incidence` (the number shows the change in percent).\n`change_incidence` was dropped again on 2021-09-02 (still included as `0.0` for backwards-compatibility) and replaced with a new indicator `incidence_hospitalisation`.\n`basic_reproduction_number` (basic reproduction number R) was recorded until 2021-07-22.\nAfter that, it is only included as `0.0`, in case applications rely on it to be there.\n\nEach indicator has a numeric `value` and a traffic light `color`-code (one of [`green`, `yellow`, `red`]).\nFor the exact meaning of color codes please refer to the corona dashboard.\nNote that the meaning of the color codes was adjusted on 2021-09-02.\n\n`vaccination` shows the total number of administered doses of COVID-19 vaccinations, as well as the percentage of the population that has received one or two doses, respectively.\n\n## Running the Scraper\n\n### Running Automatically with GitHub Actions\n\nThe scraper runs automatically every day, several times starting midday (around the time when the dashboard is usually updated).\nPreviously, the scraper ran in the afternoon.\nThis changed on 2021-05-07, when the update times for the dashboard moved to noon the following day.\n\nTo run the scraper at the specified times, I have defined a workflow in [.github/workflows/scraper.yml](.github/workflows/scraper.yml) for [GitHub Actions](https://github.com/features/actions), GitHub's continuous integration framework.\nThe workflow \n\n* sets up a virtual machine,\n* checks out the repository,\n* runs the scraper and\n* commits and pushes the updated data if there are changes.\n\nSetting up the workflow was surprisingly easy, so I'd definitely recommend this if you want to regularly run a scraper and don't want to push the button yourself every day!\n\n[@jaimergp](https://github.com/jaimergp) and later also [@graste](https://github.com/graste) recommended doing this, and I'm very grateful for the inspiration!\nI had no idea gh-actions included a cron-based trigger that makes this possible ...\n\n[@simonw](https://github.com/simonw)'s blog post at https://simonwillison.net/2020/Oct/9/git-scraping/ is a very good starting point if you want to learn how to do [git-scraping](https://github.com/topics/git-scraping).\n\n### Running Manually\n\nIf you want to run the scraper yourself manually (maybe you want to improve it), you can. Here is how:\n\n#### Requirements\n\n* Ruby 2.7.1 (most likely works with other versions too, I haven't tried)\n* the [Nokogiri](https://nokogiri.org) gem\n* the [jq](https://stedolan.github.io/jq/) JSON processor (jq is not a hard requirement anymore – it's only used to extract the current day for [data/target/berlin_corona_traffic_light.latest.json](data/target/berlin_corona_traffic_light.latest.json)).\n\n#### Installation\n\n- First, make sure you have Ruby installed.\n- If you have the [Bundler](https://bundler.io) tool, you can install the gem (Ruby library) dependencies like this:\n\n```bash\n$ bundler install\n# ... output from bundler ...\n```\n\n- If you don't have bundler (you should get it), just install the dependencies individually with `gem`.\nSince there is currently really only one non-standard gem dependency, this is no less convenient:\n\n```bash\n$ gem install nokogiri\n# ... output from gem ...\n```\n\n- Finally, download this repository to a place of your choosing, or use `git clone`.\n\n#### Make Targets\n\nThere is a [Makefile](Makefile) that orchestrates the scraping.\nThe targets should be easy to understand (each one has an `echo` statement that verbosely says what it does).\n\nTo create the complete timeline for both case numbers and traffic light indicators, use the `all` target.\nYou should get output like this:\n\n```\n$ make all\ndeleting temp folder ...\ncreating temp directory ...\nrunning corona dashboard parser ...\nI, [2020-09-24T22:17:27.647621 #99403]  INFO -- : reading current case number file data/target/berlin_corona_cases.json ...\nI, [2020-09-24T22:17:27.664712 #99403]  INFO -- : reading current traffic light file data/target/berlin_corona_traffic_light.json ...\nI, [2020-09-24T22:17:27.666040 #99403]  INFO -- : loading and parsing dashboard from https://www.berlin.de/corona/lagebericht/desktop/corona.html ...\nI, [2020-09-24T22:17:30.703220 #99403]  INFO -- : getting publication date ...\nI, [2020-09-24T22:17:30.712117 #99403]  INFO -- : publication date is 2020-09-24 ...\nI, [2020-09-24T22:17:30.712160 #99403]  INFO -- : extracting case number data ...\nI, [2020-09-24T22:17:30.733268 #99403]  INFO -- : extracting traffic light data ...\ncopying data from data/temp/berlin_corona_cases.json to data/target/berlin_corona_cases.json ...\ncopying data from data/temp/berlin_corona_traffic_light.json to data/target/berlin_corona_traffic_light.json ...\nextracting latest set of traffic light indicators from data/target/berlin_corona_traffic_light.json ...\nwriting to data/target/berlin_corona_traffic_light.latest.json ...\nwrite current date ...\nupdate README.md with current date\n```\n\n## What Happened to the Old Scraper?\n\nInitially, I had written a Python-based scraper for the daily press releases, using the [Scrapy](https://scrapy.org) Web-scraping framework.\nWhen SenGPG stopped publishing those press releases, the scraper was no longer functional.\nFor a few days I \"scraped\" manually, until I got a new scraper for the dashboard running.\nThe new scraper is Ruby-based and uses [Nokogiri](https://nokogiri.org).\nThis doesn't mean I now think Scrapy is bad – far from it, it's great!\nIt's just that for the task of scraping all individual Corona press releases, starting from a paged index of press releases, Scrapy has some pretty handy functionality.\nAlso, I just wanted to try it out.\nFor simply scraping the same dashboard page every day, Scrapy seemed overkill, and I have more experience using Ruby and Nokogiri.\nIt was just easier for me personally.\n\nIf you're looking for the old Scrapy-based scraper, you can still find it in [release 0.2.4](https://github.com/knudmoeller/berlin_corona_cases/tree/0.2.4 \"Release 0.2.4 of the 'Coronavirus Cases Scraper for Berlin'\").\n\n## Logo\n\n- [virus](https://fontawesome.com/icons/virus) logo by [FontAwesome](https://fontawesome.com) under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).\n\n## License\n\nAll software in this repository is published under the [MIT License](LICENSE).\nAll data in this repository (in particular the `.json` files) is published under [CC BY 3.0 DE](https://creativecommons.org/licenses/by/3.0/de/).\n\n## Disclaimer\n\nI do not make any claims that the data in [data/target/berlin_corona_cases.json](data/target/berlin_corona_cases.json) is correct!\nIf you find bugs in the code or in the data, please let me know by opening an issue [here](https://github.com/knudmoeller/berlin_corona_cases/issues).\n\n---\n\n2020, Knud Möller\n\nRepository: [https://github.com/knudmoeller/berlin_corona_cases](https://github.com/knudmoeller/berlin_corona_cases)\n\nLast changed: 2023-08-24\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknudmoeller%2Fberlin_corona_cases","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fknudmoeller%2Fberlin_corona_cases","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknudmoeller%2Fberlin_corona_cases/lists"}