{"id":13419321,"url":"https://github.com/MartinHeinz/ga-extractor","last_synced_at":"2025-03-15T05:30:48.704Z","repository":{"id":54632851,"uuid":"471688604","full_name":"MartinHeinz/ga-extractor","owner":"MartinHeinz","description":"Tool for extracting Google Analytics data suitable for migrating to other platforms/databases","archived":false,"fork":false,"pushed_at":"2024-04-23T17:45:18.000Z","size":240,"stargazers_count":47,"open_issues_count":1,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-04-23T19:33:30.418Z","etag":null,"topics":["analytics","cli","google-analytics","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MartinHeinz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-19T12:26:19.000Z","updated_at":"2024-07-31T00:47:01.984Z","dependencies_parsed_at":"2024-07-31T00:46:58.348Z","dependency_job_id":"937dba54-8432-4ca2-8b41-c842f679ebd8","html_url":"https://github.com/MartinHeinz/ga-extractor","commit_stats":{"total_commits":41,"total_committers":1,"mean_commits":41.0,"dds":0.0,"last_synced_commit":"077687b11f8f3d248e1e80fd38aa2d5ea8d8b6ab"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MartinHeinz%2Fga-extractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MartinHeinz%2Fga-extractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MartinHeinz%2Fga-extractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MartinHeinz%2Fga-extractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MartinHeinz","download_url":"https://codeload.github.com/MartinHeinz/ga-extractor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243690113,"owners_count":20331726,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","cli","google-analytics","python3"],"created_at":"2024-07-30T22:01:14.329Z","updated_at":"2025-03-15T05:30:48.404Z","avatar_url":"https://github.com/MartinHeinz.png","language":"Python","funding_links":["https://ko-fi.com/K3K6F4XN6"],"categories":["Python"],"sub_categories":[],"readme":"# Google Analytics Extractor\n\n[![PyPI version](https://badge.fury.io/py/ga-extractor.svg)](https://badge.fury.io/py/ga-extractor)\n\nA CLI tool for extracting Google Analytics data using Google Reporting API. Can be also used to transform data to various formats suitable for migration to other analytics platforms.\n\nAlso see - [Goodbye, Google Analytics - Why and How You Should Leave The Platform](https://martinheinz.dev/blog/71) for more context.\n\n-----\n\nIf you find this useful, you can support me on Ko-Fi (Donations are always appreciated, but never required):\n\n[![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/K3K6F4XN6)\n\n## Setup\n\nYou will need Google Cloud API access for run the CLI:\n\n- Navigate to [Cloud Resource Manager](https://console.cloud.google.com/cloud-resource-manager) and click _Create Project_\n    - alternatively create project with `gcloud projects create $PROJECT_ID`\n- Navigate to [Reporting API](https://console.cloud.google.com/apis/library/analyticsreporting.googleapis.com) and click _Enable_\n- Create credentials:\n    - Go to [credentials page](https://console.cloud.google.com/apis/credentials)\n    - Click _Create credentials_, select _Service account_\n    - Give it a name and make note of service account email. Click _Create and Continue_\n\n    - Open [Service account page](https://console.cloud.google.com/iam-admin/serviceaccounts)\n    - Select previously created service account, Open _Keys_ tab\n    - Click _Add Key_ and _Create New Key_. Choose JSON format and download it. (store this **securely**)\n\n- Give SA permissions to GA - [guide](https://support.google.com/analytics/answer/1009702#Add)\n    - email: SA email from earlier\n    - role: _Viewer_\n  \nAlternatively see following [setup](https://martinheinz.dev/blog/62).\n\nTo install and run:\n\n```bash\npip install ga-extractor\nga-extractor --help\n```\n  \n## Running\n\n```bash\nga-extractor --help\n# Usage: ga-extractor [OPTIONS] COMMAND [ARGS]...\n# ...\n\n# Create config file:\nga-extractor setup \\\n  --sa-key-path=\"analytics-api-24102021-4edf0b7270c0.json\" \\\n  --table-id=\"123456789\" \\\n  --metrics=\"ga:sessions\" \\\n  --dimensions=\"ga:browser\" \\\n  --start-date=\"2022-03-15\" \\\n  --end-date=\"2022-03-19\"\n  \ncat ~/.config/ga-extractor/config.yaml  # Optionally, check config\n\nga-extractor auth  # Test authentication\n# Successfully authenticated with user: ...\n\nga-extractor setup --help  # For options and flags\n```\n\n- Value for `--table-id` can be found in GA web console - Click on _Admin_ section, _View Settings_ and see _View ID_ field\n- All configurations and generated extracts/reports are stored in `~/.config/ga-extractor/...`\n- You can also use metrics and dimensions presets using `--preset` with `FULL` or `BASIC`, if you're not sure which data to extract\n\n### Extract\n\n```bash\nga-extractor extract\n# Report written to /home/some-user/.config/ga-extractor/report.json\n```\n\n`extract` perform raw extraction of dimensions and metrics using the provided configs\n\n### Migrate\n\nYou can directly extract and transform data to various formats. Available options are:\n\n- JSON (Default option; Default API output)\n- CSV\n- SQL (compatible with _Umami_ Analytics PostgreSQL backend)\n\n```bash\nga-extractor migrate --format=CSV\n# Report written to /home/user/.config/ga-extractor/02c2db1a-1ff0-47af-bad3-9c8bc51c1d13_extract.csv\n\nhead /home/user/.config/ga-extractor/02c2db1a-1ff0-47af-bad3-9c8bc51c1d13_extract.csv\n# path,browser,os,device,screen,language,country,referral_path,count,date\n# /,Chrome,Android,mobile,1370x1370,zh-cn,China,(direct),1,2022-03-18\n# /,Chrome,Android,mobile,340x620,en-gb,United Kingdom,t.co/,1,2022-03-18\n\nga-extractor migrate --format=UMAMI\n# Report written to /home/user/.config/ga-extractor/cee9e1d0-3b87-4052-a295-1b7224c5ba78_extract.sql\n\n# IMPORTANT: Verify the data and check test database before inserting into production instance \n# To insert into DB (This should be run against clean database):\ncat cee9e1d0-3b87-4052-a295-1b7224c5ba78_extract.sql | psql -Upostgres -a some-db\n```\n\nYou can verify the data is correct in Umami web console and GA web console:\n\n- [Umami extract](./assets/umami-migration.png)\n- [GA Pageviews](./assets/ga-pageviews.png)\n\n_Note: Some data in GA and Umami web console might be little off, because GA displays many metrics based on sessions (e.g. Sessions by device), but data is extracted/migrated based on page views. You can however confirm that percentage breakdown of browser or OS usage does match._\n\n## Development\n\n### Setup\n\nRequirements:\n\n- Poetry (+ virtual environment)\n\n```bash\npoetry install\npython -m ga_extractor --help\n```\n\n### Testing\n\n```bash\npytest\n```\n\n### Building Package\n\n```bash\npoetry install\nga-extractor --help\n\n# Usage: ga-extractor [OPTIONS] COMMAND [ARGS]...\n# ...\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMartinHeinz%2Fga-extractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMartinHeinz%2Fga-extractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMartinHeinz%2Fga-extractor/lists"}