{"id":19913391,"url":"https://github.com/morphl-ai/morphl-model-publishers-churning-users","last_synced_at":"2025-05-03T05:30:24.417Z","repository":{"id":113457110,"uuid":"141470573","full_name":"Morphl-AI/MorphL-Model-Publishers-Churning-Users","owner":"Morphl-AI","description":"Google Analytics connector, pre-processor and model for predicting churning users for digital publishers.","archived":false,"fork":false,"pushed_at":"2019-05-27T09:11:59.000Z","size":217,"stargazers_count":10,"open_issues_count":3,"forks_count":6,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-02-24T17:35:07.755Z","etag":null,"topics":["google-analytics","ingestion-pipeline","machine-learning","morphl-platform","prediction","preprocessor","pyspark"],"latest_commit_sha":null,"homepage":"https://morphl.io/","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Morphl-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-07-18T17:52:17.000Z","updated_at":"2022-06-05T16:33:24.000Z","dependencies_parsed_at":"2023-07-26T10:36:46.520Z","dependency_job_id":null,"html_url":"https://github.com/Morphl-AI/MorphL-Model-Publishers-Churning-Users","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Morphl-AI%2FMorphL-Model-Publishers-Churning-Users","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Morphl-AI%2FMorphL-Model-Publishers-Churning-Users/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Morphl-AI%2FMorphL-Model-Publishers-Churning-Users/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Morphl-AI%2FMorphL-Model-Publishers-Churning-Users/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Morphl-AI","download_url":"https://codeload.github.com/Morphl-AI/MorphL-Model-Publishers-Churning-Users/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224352933,"owners_count":17297171,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["google-analytics","ingestion-pipeline","machine-learning","morphl-platform","prediction","preprocessor","pyspark"],"created_at":"2024-11-12T21:32:55.466Z","updated_at":"2024-11-12T21:32:55.528Z","avatar_url":"https://github.com/Morphl-AI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MorphL Model for Predicting Churning Users for Publishers\n\n## Introduction\n\nA lot of websites from the publishing industry use Google Analytics to track their users. Google Analytics reports are useful for analyzing trends in the overall traffic and optimizing conversion rates. At the same time, the abundance of aggregated data makes it difficult to identify patterns in user behaviour, even by experienced marketers.\n\nBy default, Google Analytics includes a series of reports, for example viewing a total of users and sessions from a particular date interval.\n\nThe free version of the Google Analytics Reporting API v4 doesn't export any client ids from the **User Explorer report**. However, it is possible to make these available by creating a custom dimension with the same value as a Client ID, a process we have [documented on our Github account](https://github.com/Morphl-AI/MorphL-Collectors-Requirements/tree/master/google-analytics). This allows the analytics API to export data at the Client ID, Session or Hit level, instead of returning only aggregated data.\n\nWe should clarify that the **Client ID refers to a browser**, not to a user account, thus it doesn't contain any personal data. It is possible to associate the Client ID with a user account (across devices), however in this particular use case, all client ids refer to browsers.\n\n\u003ca name=\"orchestrator-setup\"\u003e\u003c/a\u003e\n\n## Using Model on the MorphL Orchestrator\n\nConnecting to **Google Analytics API v4** requires creating a service account and retrieving a view ID from your Google Analytics dashboard. The orchestrator assumes that your Google Analytics dashboard has already been configured to allow exporting of granular data (at the browser \u0026 session level). You can read [here](https://github.com/Morphl-Project/MorphL-Collectors/tree/master/google-analytics) about the required setup and **creating a service account**.\n\nOnce the [MorphL Orchestrator](https://github.com/Morphl-AI/MorphL-Orchestrator) has been set up, SSH to the VM and from the root prompt, log into `airflow`:\n\n```\nsu - airflow\n```\n\nPaste your key file into `/opt/secrets/keyfile.json` and your view ID into `/opt/secrets/viewid.txt`, possibly using syntax like this:\n\n```\ncat \u003e /opt/secrets/keyfile.json \u003c\u003c EOF\n{\n...supersecretkeyfilecontents...\n}\nEOF\n\ncat \u003e /opt/secrets/viewid.txt \u003c\u003c EOF\n123123456456123123\nEOF\n```\n\n## Problem Setting\n\nHaving access to granular data, **we can predict when a user is going to churn**. We have defined churned users as previously retained users that do not return to the website before a time interval (threshold) has passed. **By retained users**, we mean users that have visited the website at least twice in the past (they have at least 2 sessions).\n\nOur training sets are going to aggregate session and hit data at the user level.\n\n## Features and Data Labeling\n\nThe most relevant data related to a users history we can obtain from the [Google Analytics API v4](https://developers.google.com/analytics/devguides/reporting/core/dimsmets) includes:\n\n- Sessions (total sessions for each user, in a time interval);\n- Session duration (total sessions duration for each user, in a time interval);\n- Avg. session duration\n- Entrances\n- Bounces\n- Pageviews\n- Unique pageviews\n- Screen Views\n- Page value\n- Exits\n- Time on Page\n- Avg. Time on Page\n- Page Load Time (ms)\n- Avg. Page Load Time (sec)\n- Days since last session;\n- Count of sessions (total number of sessions for the user, independent of the selected time interval)\n- Hits (total hits for each user, in a time interval);\n- Device Category (mobile, desktop or tablet)\n\nFor predicting churn, we have labeled the users as churned / not churned by:\n\n- Calculating the average time between sessions of retained users (`Avg. days between sessions`).\n- Label the data. If a user has a value of `Days Since Last Session \u003e mean(Avg. days between sessions)`, he is labeled as churned (`Churned` = 0 or 1).\n- `Days since last session` and `Avg. days between sessions` will not be included as features in the training set, as they are heavily correlated with the label `Churned.`\n\nThe model can be improved by predicting future churned users (users that are currently not churned, but will churn in the future).\n\n## Pipelines Architecture\n\nThis repository contains the code for the churned users pipelines, including model training and predictions. The code runs on the [MorphL Platform Orchestrator](https://github.com/Morphl-AI/MorphL-Orchestrator) which creates 3 pipelines: **Ingestion Pipeline**, **Training Pipeline** and **Prediction Pipeline**.\n\n### Ingestion Pipeline\n\n#### 1. Google Analytics Connector\n\nIt is responsible for authenticating to the Google Analytics API v4 using a service account and retrieving data. See the **Features and Data Labeling** section for a complete list of Google Analytics dimensions and metrics. The Google Analytics data is saved in Cassandra tables.\n\nThe connector runs daily and it can also be used to retrieve historical data (for backfilling).\n\nYou can read about integrating the MorphL data science project with Cassandra [here](https://github.com/Morphl-AI/MorphL-Community-Edition/wiki/Integrating-the-MorphL-data-science-project-with-Cassandra).\n\n### Training Pipeline\n\nAll components from this pipeline are run on a weekly basis.\n\n#### 1. Pre-processor for formatting data\n\nIt is implemented using PySpark and it is responsible for processing the data retrieved from the Google Analytics API. It reads the data (in JSON format) and transforms it into SQL-like Cassandra tables. It also labels the data.\n\n#### 2. Pre-processor for transforming data\n\nApplies data transformations such as power transforms and feature scaling. This pre-processor is also used by the prediction pipeline.\n\nIt returns a Dask dataframe.\n\n#### 3. Model generator\n\nTakes a Dask dataframe on initialization. It will train and save the model as a .h5 file, together with a json file which includes the model scores.\n\nFor training the model we have used Keras / TensorFlow.\n\n### Prediction Pipeline\n\n#### 1. Pre-processors for formatting and transforming data\n\nUses the same pre-processors (PySpark and Dask) as the training pipeline, but in \"prediction\" mode. The same process is applied: formatting the data, followed by power transforms and feature scaling. As a difference, in \"prediction\" mode, the data is not labeled.\n\n#### 2. Batch inference\n\nIt is used for making predictions and saving them in the Cassandra database.\n\n#### 3. Endpoint\n\nAfter the prediction pipeline is triggered, predictions can be accessed at an endpoint. See the MorphL Platform Orchestrator for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmorphl-ai%2Fmorphl-model-publishers-churning-users","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmorphl-ai%2Fmorphl-model-publishers-churning-users","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmorphl-ai%2Fmorphl-model-publishers-churning-users/lists"}