{"id":20183001,"url":"https://github.com/emibcn/covid-data","last_synced_at":"2026-05-02T08:33:08.721Z","repository":{"id":38316852,"uuid":"295212141","full_name":"emibcn/covid-data","owner":"emibcn","description":"Store and serve daily collected data from https://dadescovid.org for sibling app at https://emibcn.github.io/covid/","archived":false,"fork":false,"pushed_at":"2026-03-20T08:51:10.000Z","size":16686,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-03-21T01:38:20.282Z","etag":null,"topics":["backend","cache","charts","covid-data","generalitat-de-catalunya","github-actions","github-page","github-pages","hacktoberfest","json","json-objects","scraped-data","scraper","storage","workflow"],"latest_commit_sha":null,"homepage":"https://emibcn.github.io/covid/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/emibcn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-09-13T18:26:34.000Z","updated_at":"2026-03-20T08:51:10.000Z","dependencies_parsed_at":"2026-01-11T07:03:31.641Z","dependency_job_id":null,"html_url":"https://github.com/emibcn/covid-data","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/emibcn/covid-data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emibcn%2Fcovid-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emibcn%2Fcovid-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emibcn%2Fcovid-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emibcn%2Fcovid-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/emibcn","download_url":"https://codeload.github.com/emibcn/covid-data/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emibcn%2Fcovid-data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32528223,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-02T01:12:54.858Z","status":"online","status_checked_at":"2026-05-02T02:00:05.923Z","response_time":132,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend","cache","charts","covid-data","generalitat-de-catalunya","github-actions","github-page","github-pages","hacktoberfest","json","json-objects","scraped-data","scraper","storage","workflow"],"created_at":"2024-11-14T02:43:40.679Z","updated_at":"2026-05-02T08:33:08.704Z","avatar_url":"https://github.com/emibcn.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![DeepSource](https://deepsource.io/gh/emibcn/covid-data.svg/?label=active+issues\u0026show_trend=true\u0026token=qFtT_T_AhcT4Azdo-cIAjfiW)](https://deepsource.io/gh/emibcn/covid-data/?ref=repository-badge)\n![Download maps and charts from backend and publish with GitHub Pages](https://github.com/emibcn/covid-data/workflows/Download%20maps%20and%20charts%20from%20backend%20and%20publish%20with%20GitHub%20Pages/badge.svg)\n\n# Covid Data\n\nThis repo is used to store and serve daily collected data from https://dadescovid.org (institutional data published by the **Generalitat de Catalunya** ) and [Seguiment Covid19 BCN](https://dades.ajuntament.barcelona.cat/seguiment-covid19-bcn) (institutional data collected and published by the **Ajuntament de Barcelona**) into GitHub pages. The reasons for collecting this data are:\n\n- The data will be used from apps which don't need more than daily updates, like [Covid Data `Refactored`](https://emibcn.github.io/covid/), an [open source serverless Progressive Web Application](https://github.com/emibcn/covid)\n- Original data requests might be blocked by CORS or other technologies\n- Original servers and data might not be efficient enough\n- When applicable, normalize data from various servers\n\nThe collected data from https://dadescovid.cat is minimally adapted before publishing it:\n\nMaps:\n\n- Transform JS statements into JSON objects\n- Reduce some non-visible `id`'s to reduce users and servers resource consumptions\n\nCharts:\n\n- Transform JS statements into JSON objects\n- Transform HTML tags attributes and content into JSON structured data\n\nThe collected data from [Seguiment Covid19 BCN](https://dades.ajuntament.barcelona.cat/seguiment-covid19-bcn) is deeply reshaped, throwing away the unneded/repetitive data.\n\nThis repo might collect other data in the future, from the same server, it's [backend server](https://analisi.transparenciacatalunya.cat/) or from 3rd party servers (EU statistics servers? Data collection from other regions?).\n\n# Techie\n\nThis process is executed from a [GitHub Workflow](./.github/workflows/get-maps-and-charts.yml) (`cron` scheduled some minuts after official data publication at 10am CEST). Once the data is obtained, deploy it to this repo' GitHub Pages in the `gh-pages` branch.\n\n## Maps\n\nThe data is collected by an ugly [BASH script](./bin/download-map-data.sh). This script collects the interesting parts (maps SVG source, JS code with data on it) and saves them into files. The SVG files are saved transparently. The JS files are executed with NODE to ouput the collected data as JSON.\n\n## Charts\n\nThe data is collected by a [NodeJS package](./charts/). This script scrapes data from HTML tags and JS code. It generates individual JSON files for each region/population selectors, and a global JSON index file with the regions recursive structure and all the download links. Deep use of `async`/`await`.\n\n## Barcelona\n\nThe data is collected by a [nice NodeJS package](./bcn/). This script uses a self made version of [SockJS](./bcn/src/Socket.js) to scrape data from a [RStudio/Shiny](https://github.com/rstudio/shiny) server. It generates individual JSON files for each datasource or datasource section, and a global JSON index file with the data and all the download links. Deep use of `async`/`await`. Some use of Streams and Iterator Generator and deep use of `async`/`await`. Very funny stuff!\n\n# License\n\nThe application, scripts and documentation in this project are released under the [GNU General Public License v3.0](./LICENSE).\n\nThe license of the data scraped from https://dadescovid.cat and saved into the directories `Charts` and `Maps` is the same as the original: [Open Data Commons Attribution License](http://opendatacommons.org/licenses/by/1.0/), as stated in [the backend API page](https://analisi.transparenciacatalunya.cat/Salut/Dades-setmanals-de-COVID-19-per-comarca/jvut-jxu8) owned by the _Generalitat de Catalunya_.\n\nThe license of the data scraped from https://dades.ajuntament.barcelona.cat/seguiment-covid19-bcn is the same as the original: [Creative Commons CC-BY](https://creativecommons.org/licenses/by/2.0/). The owner is the _Ajuntament de Barcelona_.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femibcn%2Fcovid-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Femibcn%2Fcovid-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femibcn%2Fcovid-data/lists"}