{"id":18174352,"url":"https://github.com/data-for-good-concepts/serverless-scraping-service","last_synced_at":"2025-04-01T15:31:16.773Z","repository":{"id":62394002,"uuid":"559665463","full_name":"data-for-good-concepts/serverless-scraping-service","owner":"data-for-good-concepts","description":"A concept implementation of a serverless scraping web service","archived":false,"fork":false,"pushed_at":"2022-11-20T03:06:28.000Z","size":925,"stargazers_count":7,"open_issues_count":7,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-03-28T14:12:18.293Z","etag":null,"topics":["cloudbuild","cloudrun","docker","github-actions","plumber-api","rselenium"],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/data-for-good-concepts.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":".github/SECURITY.md","support":null}},"created_at":"2022-10-30T20:02:45.000Z","updated_at":"2024-10-16T21:49:54.000Z","dependencies_parsed_at":"2023-01-22T17:45:18.898Z","dependency_job_id":null,"html_url":"https://github.com/data-for-good-concepts/serverless-scraping-service","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/data-for-good-concepts%2Fserverless-scraping-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/data-for-good-concepts%2Fserverless-scraping-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/data-for-good-concepts%2Fserverless-scraping-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/data-for-good-concepts%2Fserverless-scraping-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/data-for-good-concepts","download_url":"https://codeload.github.com/data-for-good-concepts/serverless-scraping-service/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246662327,"owners_count":20813728,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudbuild","cloudrun","docker","github-actions","plumber-api","rselenium"],"created_at":"2024-11-02T16:03:04.615Z","updated_at":"2025-04-01T15:31:16.340Z","avatar_url":"https://github.com/data-for-good-concepts.png","language":"R","funding_links":[],"categories":["R"],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r setup, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\"\n)\n```\n\n# Serverless Scraping Service\u003cimg src=\"man/figures/logo.png\" align=\"right\" height=\"139\"/\u003e\n\n\u003c!-- badges: start --\u003e\n\n[![Lifecycle: experimental](https://lifecycle.r-lib.org/articles/figures/lifecycle-experimental.svg)](https://lifecycle.r-lib.org/articles/stages.html#experimental) [![deploy-cloudrun](https://github.com/data-for-good-concepts/eurostat-scraper/actions/workflows/deploy-cloudrun.yaml/badge.svg)](https://github.com/data-for-good-concepts/eurostat-scraper/actions/workflows/deploy-cloudrun.yaml)\n[![build](https://github.com/data-for-good-concepts/eurostat-scraper/actions/workflows/dry-run-docker-build.yaml/badge.svg)](https://github.com/data-for-good-concepts/eurostat-scraper/actions/workflows/dry-run-docker-build.yaml)\n\n\u003c!-- badges: end --\u003e\n\n## Overview\n\n\u003e ⚠️ _Please note, that the documentation of this project is still \n**Work In Progress**. The description might be incomplete and is subject to \nconstant change._  \n\nThis example scrapes the _'Asylum applicants by type of applicant, citizenship, age and sex - monthly data'_ \nfrom the [**Eurostat Data Browser**](https://ec.europa.eu/eurostat/databrowser/view/MIGR_ASYAPPCTZM/default/table?lang=en). \nThe scraper can be triggered with a `POST` request to an API endpoint, which \nreturns the scraped data as the `response`. Even though this example has been \ndeveloped to scrape one particular dataset, any other dataset that is available\nas part of the **Eurostat Data Browser** can be scraped with little to no \nadjustments.\n\nThe **Eurostat Scraper** is a working example on how to wrap a web scraper in a \nweb service and deploy it on the cloud using R. This project aims to showcase \ndifferent aspects when it comes to web scraping in general, as well as different\ntechnologies when deploying your work in production.\n\nAll this might sound good to you and you probably still wonder, why wrap a \nscraper in a web service? The reason is, having your application wrapped in a \nweb service makes it accessible, easy to interact and easy to intergrate into\nother applications. You can communicate with this application through \na `REST` API, which is possible using any modern programming language.\n\n## Architecture\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"man/figures/architecture.png\" width=\"85%\" alt=\"scraper web service architecture\"/\u003e\n\u003c/p\u003e\n\n## Prerequisites\n\n- In order to run this service locally, you need to have a [Firefox](https://www.mozilla.org/) client installed on your machine.\n- In order to deploy this image on Google Cloud, you will need a [Google Cloud Account](https://cloud.google.com/) and a project.\n- You then also must [enable the Cloud Build API](https://console.cloud.google.com/apis/library/cloudbuild.googleapis.com)\nas well and [enable the Cloud Run API](https://console.cloud.google.com/apis/library/run.googleapis.com).\n- To deploy this service from the command line you need to install [gcloud CLI](https://cloud.google.com/sdk/docs/install).\n- To deploy this service using `GitHub Actions` you need to export `Service \nAccount` credentials. See [Using GitHub Actions](#using-github-actions) for more information.\n\nWhen using a depolyed version of this project, you don't need to worry about the\n`Firefox` client, as it will be installed as part of the `Docker` image. \nFurther, following tools and technologies will be highlighted or are heavily in\nuse as part of this concept - `RSelenium`, `plumber`, `Docker`,\n`Google Cloud Build`, `Google Cloud Run` and `GitHub Actions`.\n\n## Usage\n\nIn order to launch the scraping service locally, install the necessary\ndependencies and start the web service by executing following commands in your \nR console. \n\n```{r, eval=FALSE}\nrenv::restore()\nsource('src/server.R')\n```\n\n### Trigger scraping job\n\nStart a scraping job by making a `POST` request to the trigger endpoint. \nWhen you are working locally, replace `\u003cURL\u003e` with `localhost:8080`. When you \nhave the web service deployed on Google Cloud Run, use your Cloud Run service \nURL.\n\n```{bash, eval=FALSE}\n# bash\ncurl --location --request POST '\u003cURL\u003e/api/v1/scraper/job' \\\n--header 'Content-Type: application/json' \\\n--data-raw '{\n   \"url\": \"https://ec.europa.eu/eurostat/databrowser/view/MIGR_ASYAPPCTZM/default/table?lang=en\",\n   \"dataset\":[\n      {\n         \"dimension\":\"Age class\",\n         \"filter\":[\n            \"[Y_LT14]\",\n            \"[Y14-17]\"\n         ]\n      },\n      {\n         \"dimension\":\"Country of citizenship\",\n         \"filter\":[\n            \"[UA]\"\n         ]\n      }\n   ]\n}'\n```\n\n```{r, eval=FALSE}\n# R\nhttr::POST(\n  '\u003cURL\u003e/api/v1/scraper/job', \n  body = list(\n           url = \"https://ec.europa.eu/eurostat/databrowser/view/MIGR_ASYAPPCTZM/default/table?lang=en\",\n           dataset = list(\n                       list(\n                         dimension = 'Age class', \n                         filter    = list('[Y_LT14]', '[Y14-17]')\n                       ),\n                       list(\n                         dimension = 'Country of citizenship',\n                         filter    = list('[UA]')\n                       )\n                     )\n         ),\n  encode = \"json\"\n)\n\n```\n\nAs a response the scraping service will return the requested dataset as `JSON`.\n\n```{js, eval=FALSE}\n[\n    {\n        \"DATAFLOW\": \"ESTAT:MIGR_ASYAPPCTZM(1.0)\",\n        \"LAST UPDATE\": \"11/11/22 23:00:00\",\n        \"freq\": \"M\",\n        \"unit\": \"PER\",\n        \"citizen\": \"UA\",\n        \"sex\": true,\n        \"age\": \"Y14-17\",\n        \"asyl_app\": \"ASY_APP\",\n        \"geo\": \"AT\",\n        \"TIME_PERIOD\": \"2021-12\",\n        \"OBS_VALUE\": 0\n    },\n    {\n        \"DATAFLOW\": \"ESTAT:MIGR_ASYAPPCTZM(1.0)\",\n        \"LAST UPDATE\": \"11/11/22 23:00:00\",\n        \"freq\": \"M\",\n        \"unit\": \"PER\",\n        \"citizen\": \"UA\",\n        \"sex\": true,\n        \"age\": \"Y14-17\",\n        \"asyl_app\": \"ASY_APP\",\n        \"geo\": \"AT\",\n        \"TIME_PERIOD\": \"2022-01\",\n        \"OBS_VALUE\": 0\n    },\n    ...\n]\n```\n\n## Deployment\n\n\u003e ℹ️️️️️ _Documentation on IAM permissions need to be added. Further documentation \non Service Account usage and permissions is currently missing._  \n\nThe following steps will walk you through the steps to create a `build` of this \nproject using [`Google Cloud Build`](https://cloud.google.com/build/docs/overview#:~:text=Cloud%20Build%20is%20a%20service,Docker%20containers%20or%20Java%20archives.) and then deploy it on [`Google Cloud Run`](https://cloud.google.com/run/docs/overview/what-is-cloud-run).\n\nIn case you want to modify deployment parameters, feel free to adjust the \n`cloudbuild.yaml` according to your needs.\n\n### Using the terminal\n\nOnce `gcloud CLI` is installed, execute the following command in the terminal \nin order to login to your Googgle Account and to authenticate subsequent \ncommands to deploy your service on Google Cloud. This command will open a \nbrowser prompt that will ask you to `Sign in with Google` and to grant\n`Google Cloud SDK` access to your Google account.\n\n```{bash, eval=FALSE}\ngcloud auth login\n```\n\nSet the project in which the web service is deployed to. Make sure to replace \n`PROJECT_ID` with your own project id.\n\n```{bash, eval=FALSE}\ngcloud config set project \u003cPROJECT_ID\u003e\n```\n\nYou can then deploy this service on Google Cloud by executing the following \ncommand. Make sure to replace `\u003cREGION\u003e` with the [Google Cloud region](https://cloud.google.com/compute/docs/regions-zones), \nyou want to use for the deployment of your web service.\n\n```{bash, eval=FALSE}\ngcloud builds submit --region='\u003cREGION\u003e'\n```\n\nYou can see the logs of the build in the [Cloud Build History](https://console.cloud.google.com/cloud-build).\nIn case you don't see your `build `, make sure you have selected your region in \nthe dropdown. When deploying for the first time, the build process can take \nup-to 25min. Subsequent deployments should be done in less than 5min, as we \nthen make use of `cached` Docker images. Once the build is done, you will see \nyour deployment on the [Cloud Run Overview](https://console.cloud.google.com/run).\n\n### Using GitHub Actions\n\nThe `GitHub Actions` workflow in this project is setup in a way, that with every \n`push` to the `main` branch the `deploy-cloudrun` workflow will be triggered.\nThat workflow will automatically build the image in `Cloud Build` and deploy it \non `Cloud Run`.\n\n```{yaml, eval=FALSE}\nname: deploy-cloudrun\non:\n  push:\n    branches:\n      - 'main'\n...\n```\n\nThe `deploy-cloudrun` workflow uses the `google-github-actions/auth@v1` with a \n`Service Account JSON key` to authenticate with Google Cloud. The `JSON`  key \nis stored as a Github secret and is named `GC_SA_ADMNIN_KEY`.\n\n```{yaml, eval=FALSE}\n- name: 'Authenticate GCP 🔐'\n  id: 'auth'\n  uses: 'google-github-actions/auth@v1'\n  with:\n    credentials_json: '${{ secrets.GC_SA_ADMNIN_KEY }}'\n```\n\nOnce the workflow is done, you will see your deployment on the [Cloud Run Overview](https://console.cloud.google.com/run).\n\n---\nPlease note that this project is released with a [Contributor Code of Conduct](CODE_OF_CONDUCT.md).\nBy participating in this project you agree to abide by its terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdata-for-good-concepts%2Fserverless-scraping-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdata-for-good-concepts%2Fserverless-scraping-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdata-for-good-concepts%2Fserverless-scraping-service/lists"}