{"id":51141991,"url":"https://github.com/eDatos/siemactk","last_synced_at":"2026-07-14T07:01:25.747Z","repository":{"id":51057573,"uuid":"348285514","full_name":"eDatos/siemactk","owner":"eDatos","description":"Toolkit to support operations on sie-mac.org","archived":false,"fork":false,"pushed_at":"2021-11-08T08:38:02.000Z","size":155,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-17T00:50:28.697Z","etag":null,"topics":["azores","canary-islands","data","data-science","eurostat","eurostat-data","madeira","official-statistics","open-data","scraper","scraping-websites","statistics"],"latest_commit_sha":null,"homepage":"https://sie-mac.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eDatos.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-16T09:25:17.000Z","updated_at":"2021-11-08T08:38:05.000Z","dependencies_parsed_at":"2022-08-31T18:50:14.179Z","dependency_job_id":null,"html_url":"https://github.com/eDatos/siemactk","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/eDatos/siemactk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eDatos%2Fsiemactk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eDatos%2Fsiemactk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eDatos%2Fsiemactk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eDatos%2Fsiemactk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eDatos","download_url":"https://codeload.github.com/eDatos/siemactk/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eDatos%2Fsiemactk/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35450066,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-14T02:00:06.603Z","response_time":114,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azores","canary-islands","data","data-science","eurostat","eurostat-data","madeira","official-statistics","open-data","scraper","scraping-websites","statistics"],"created_at":"2026-06-26T00:00:39.395Z","updated_at":"2026-07-14T07:01:25.741Z","avatar_url":"https://github.com/eDatos.png","language":"Python","funding_links":[],"categories":["Datos Abiertos y Estadísticas"],"sub_categories":[],"readme":"![siemactk logo](img/siemactk-logo.png)\n\n**Toolkit to support operations on [sie-mac.org](https://sie-mac.org)**\n\n# Workflow\n\n![siemactk-workflow](img/siemactk-workflow.png)\n\n1. Get datasets urls and codelists from Google Spreadsheets.\n2. Scrap these datasets from https://ec.europa.eu/eurostat.\n3. Filter, clean \u0026 recode datasets.\n4. Generate translated output files in both `.json` and `.tsv` formats.\n5. Upload output files to a bucket in Google Cloud Storage (GCS).\n6. Notify via email the result of the automation process.\n\nThus, site sie-mac.org can report tables with [wpDataTables](https://wpdatatables.com/) plugin, sourcing from data on GCS.\n\n# Auth\n\nThis project uses [Google Drive API](https://developers.google.com/drive) and [Google Cloud Storage](hhttps://cloud.google.com/storage) behind the scenes. Both services need [authentication](https://cloud.google.com/docs/authentication).\n\n## Google Drive API\n\nAs siemactk uses [yagdrive](https://github.com/sdelquin/yagdrive) as a wrapper for the Google Drive API, you should follow [these instructions](https://github.com/sdelquin/yagdrive#auth) in order to make it properly work.\n\nAt the end of the authentication process, two files must be saved in the present work directory:\n\n- `gdrive-credentials.json`\n- `gdrive-secrets.json`\n\n## Google Cloud Storage\n\nFollow these instructions in order to get authentication for the Google Cloud Storage:\n\n1. Go to [APIs Console](https://console.cloud.google.com/) and create a new project (or reuse an existing one).\n2. Go to [IAM Administration](https://console.cloud.google.com/iam-admin/) inside your project and click on [Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) on the left hand side.\n3. Click on [Create Service Account](https://console.cloud.google.com/iam-admin/serviceaccounts/create) on the header of the page.\n4. Three steps must be completed:\n   - Step 1: Give a name for this service account.\n   - Step 2: Assign function \"Owner\" to completely access the project.\n   - Step 3: You can leave it blank.\n5. Click on the newly created service account.\n6. Go to the \"Keys\" menu at the top.\n7. Click on the \"Add Key\" button and then select \"Create new key\".\n8. Leave \"JSON\" as the type of key and click on \"Create\".\n9. A file called `\u003cproject\u003e-\u003clong_id\u003e.json` is downloaded.\n10. Rename this file to `gcs-credentials.json` and place it on the present work directory.\n\nAt the end of the authentication process, one file must be saved in the present work directory:\n\n- `gcs-credentials.json`\n\n# Setup\n\n1. Install dependencies:\n\n   ```console\n   $ pip install -r requirements.txt\n   ```\n\n2. Generate [credentials for authentication](#Auth).\n3. Create a `.env` file containing parameters with no default values on [settings.py](settings.py).\n\n# Usage\n\n```console\n$ python main.py\n```\n\n## GitHub Workflow\n\nIn order to automate the launching of this scraping, a GitHub workflow has been developed and scheduled every month.\n\n# Changelog\n\nConsult the [Changelog](CHANGELOG.md) page for bugfixes and features in each version.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FeDatos%2Fsiemactk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FeDatos%2Fsiemactk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FeDatos%2Fsiemactk/lists"}