{"id":29072698,"url":"https://github.com/openmost/matomo-bigquery-importer","last_synced_at":"2025-06-27T14:08:26.971Z","repository":{"id":300854852,"uuid":"1007365191","full_name":"openmost/matomo-bigquery-importer","owner":"openmost","description":"Import Matomo API Live.getLastVisitsDetails data to Google BigQuery","archived":false,"fork":false,"pushed_at":"2025-06-23T23:37:55.000Z","size":34,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-24T00:26:41.126Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openmost.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-23T22:07:45.000Z","updated_at":"2025-06-23T23:37:58.000Z","dependencies_parsed_at":"2025-06-24T00:37:20.849Z","dependency_job_id":null,"html_url":"https://github.com/openmost/matomo-bigquery-importer","commit_stats":null,"previous_names":["openmost/matomo-bigquery-importer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/openmost/matomo-bigquery-importer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openmost%2Fmatomo-bigquery-importer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openmost%2Fmatomo-bigquery-importer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openmost%2Fmatomo-bigquery-importer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openmost%2Fmatomo-bigquery-importer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openmost","download_url":"https://codeload.github.com/openmost/matomo-bigquery-importer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openmost%2Fmatomo-bigquery-importer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262270427,"owners_count":23285165,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-27T14:08:26.258Z","updated_at":"2025-06-27T14:08:26.961Z","avatar_url":"https://github.com/openmost.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Google BigQuery\n\nCreate a BigQuery account associated with a Google Cloud Console project.\n\n\u003e Make sure you have an up-to-date billing account in Google Cloud Console.\n\n### Enable the BigQuery API\n\nIf this is your first time using BigQuery, you’ll need to enable the **BigQuery API** from your Google Cloud account.\n\n### Create a Dataset\n\nIn your BigQuery project, click on the three vertical dots to the right of the project name and click **\"Create dataset\"**.\n\nName your dataset: `matomo_extract`.\n\nSet the **Region** to: `europe-west1 (Belgium)`.\n\nClick **\"Create dataset\"**.\n\n### Create a Table in the Dataset\n\nOn the newly created dataset, click on the three vertical dots next to the dataset and then on **\"Create table\"**.\n\nIn the **\"Table*\"** field, enter the value `matomo_visits`.\n\nEnable the **\"Edit as text\"** switch and paste the contents of the file [`GoogleBigQuery/bigquery-table-schema.json`](./GoogleBigQuery/bigquery-table-schema.json) from this repo.\n\nThen click the blue **\"Create table\"** button.\n\n# Google Cloud Run\n\nNow that our BigQuery table is created, we need to populate it with data from the Matomo API.\n\nTo do this, we’ll need to run a script on a regular basis to extract the data, format it, and inject it into BigQuery.\n\nGo to [Google Cloud Run](https://console.cloud.google.com/run/) (enable the API if this is your first time).\n\nOnce on the Cloud Run dashboard, click **\"Write a function\"**.\n\n### Configure a Service\n\nName your service: `matomo-bigquery-importer`.\n\nSet the region that corresponds to your legislation: `europe-west1`.\n\nChoose a runtime of `Node.js 22` or later.\n\nCheck **\"Use IAM to authenticate incoming requests\"**.\n\nCheck **\"Require authentication\"**.\n\nScroll all the way down and click the blue **\"Create\"** button.\n\n### Add Source Files\n\nNow that your service is created, you need to add source files (the code it will execute).\n\n- In the file `index.js`, paste the contents of [`GoogleCloudRun/index.js`](./GoogleCloudRun/index.js) (set `importDataFromAPI` as entry point function name)\n- In the file `package.json`, paste the contents of [`GoogleCloudRun/package.json`](./GoogleCloudRun/package.json)\n\n\u003e In the `index.js` file, the first few lines need to be updated:\n\n```javascript\n// START CONFIGURATION //\nconst matomo_host = 'https://matomo.my-company.com';\nconst matomo_site_id = 1;\nconst matomo_token_auth = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx';\nconst matomo_period = 'day';\nconst matomo_date = 'yesterday';\nconst matomo_filter_offset = 0;\nconst matomo_filter_limit = 1000;\nconst big_query_dataset_id = 'matomo_extract';\nconst big_query_table_id = 'matomo_visits';\n// END CONFIGURATION //\n```\nOnce both files are modified, click **\"Save and deploy\"**.\n\n### Create a Job for the Cloud Function\n\nIn the side menu, select **\"Jobs\"**.\n\nClick **\"Create job\"**.\n\nSelect the container image URL by choosing from the list the image with the `latest` badge (feel free to expand the options to find the correct container image).\n\nOnce the image is selected, click **\"Save\"**.\n\n### Define a Trigger for the Job with Google Scheduler\n\nIn your **\"Job\"**, go to the **\"Triggers\"** tab.\n\nClick **\"Add scheduler trigger\"** and set the frequency to `0 2 * * *` (which means 2:00 AM every day).\n\nIn the next field, select your **time zone**.\n\nThen click **\"Create\"**.\n\nYour task is now created and available in [Google Cloud Scheduler](https://console.cloud.google.com/cloudscheduler).  \nYou can test it by clicking the three vertical dots under **\"Action\"**, then selecting **\"Force run\"**.\n\n## Visualise data using BigQuery\n\nWhen you look at your data, click on the \"Request\" button and execute any request you want.\n\nYou can try to run queries, here are some [demo queries](./GoogleBigQuery/demo-queries.md).\n\n\n## Disclaimer\n\n\u003e These scripts may evolve over time.  \n\u003e Some values in the JSON object returned by the Matomo API may change and/or may not yet be supported by this script.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenmost%2Fmatomo-bigquery-importer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenmost%2Fmatomo-bigquery-importer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenmost%2Fmatomo-bigquery-importer/lists"}