{"id":29416302,"url":"https://github.com/sangnandar/realtime-gmail-listener","last_synced_at":"2026-04-18T09:33:35.886Z","repository":{"id":301035007,"uuid":"1007026871","full_name":"sangnandar/Realtime-Gmail-Listener","owner":"sangnandar","description":"Realtime Gmail listener using push notifications via Pub/Sub. Cloud Run acts as a webhook proxy to Apps Script for processing new emails into Sheets. No polling. Apps Script quota-safe. Serverless. Fully automated with PowerShell deployment.","archived":false,"fork":false,"pushed_at":"2025-06-24T20:03:49.000Z","size":18,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-24T21:22:51.003Z","etag":null,"topics":["automation","cloud-run","design-pattern","gmail","gmail-api","google-apps-script","powershell","pubsub"],"latest_commit_sha":null,"homepage":"https://script.google.com/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sangnandar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-23T10:57:03.000Z","updated_at":"2025-06-24T20:03:53.000Z","dependencies_parsed_at":"2025-06-24T21:22:53.438Z","dependency_job_id":"290d00b5-1b83-4018-a561-391ea0d88249","html_url":"https://github.com/sangnandar/Realtime-Gmail-Listener","commit_stats":null,"previous_names":["sangnandar/realtime-gmail-listener"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sangnandar/Realtime-Gmail-Listener","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sangnandar%2FRealtime-Gmail-Listener","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sangnandar%2FRealtime-Gmail-Listener/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sangnandar%2FRealtime-Gmail-Listener/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sangnandar%2FRealtime-Gmail-Listener/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sangnandar","download_url":"https://codeload.github.com/sangnandar/Realtime-Gmail-Listener/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sangnandar%2FRealtime-Gmail-Listener/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264878580,"owners_count":23677451,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","cloud-run","design-pattern","gmail","gmail-api","google-apps-script","powershell","pubsub"],"created_at":"2025-07-11T19:02:44.438Z","updated_at":"2026-04-18T09:33:35.878Z","avatar_url":"https://github.com/sangnandar.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Realtime Gmail Listener with Cloud Run and Apps Script\n\nThis project provides a **realtime, serverless solution** to monitor Gmail for new messages using **push notifications** instead of traditional polling. It leverages **Google Cloud Pub/Sub** to receive events instantly when a new email arrives, and routes them through **Cloud Run** to **Apps Script**, where your custom processing logic (e.g., updating Google Sheets) runs.\n\nMany Gmail integrations rely on **polling** (e.g., every 5–10 minutes) to check for new emails.\nThis approach not only introduces **delay**, but also **exhausts Apps Script’s daily quota**—specifically the **“Triggers total runtime” limit**, which is capped at **90 minutes per day** for consumer accounts. Even moderate polling usage can hit this limit quickly, leading to silent failures or missed updates.\n\nThis project avoids polling entirely and achieves **instant responsiveness** using Gmail’s `watch()` API with Pub/Sub and Cloud Run.\n\n### 🔧 Key Components\n\n* **Gmail API** – Watches for new messages and sends push notifications.\n* **Pub/Sub** – Receives events and buffers them reliably.\n* **Cloud Run** – Acts as a webhook endpoint to receive Pub/Sub pushes and forward to Apps Script.\n* **Apps Script** – Handles the business logic (e.g., parsing emails, writing to Sheets).\n* **SheetLayout Class** – Manages dynamic spreadsheet layouts.\n* **Secret Manager** – Stores sensitive configuration (tokens, script URLs, etc.) securely.\n\nThis setup offers a **scalable, secure, and truly real-time** email listener inside the Google ecosystem.\n\n---\n\n## ✨ Features\n\n* **📬 Realtime Email Notifications**\n  Listens for new Gmail messages using the Gmail `watch()` API and Pub/Sub, avoiding delay from polling.\n\n* **⚡ Serverless \u0026 Scalable**\n  Built entirely with Google Cloud services (Cloud Run, Pub/Sub, Secret Manager, Apps Script), requiring zero server maintenance.\n\n* **🔐 Secure Secret Management**\n  API URLs and keys are stored in Secret Manager and injected securely at runtime—no hardcoded secrets.\n\n* **🔁 Automatic Watch Renewal**\n  Time-based Apps Script triggers automatically renew Gmail watch subscriptions before expiration.\n\n* **📄 Smart Spreadsheet Layout**\n  Uses the `SheetLayout` class to dynamically manage sheet headers and structure—reducing code duplication.\n\n* **🧪 Clean Resource Management**\n  Includes deployment and teardown scripts for easy setup and full cleanup.\n\n* **🧩 Modular Design**\n  Clear separation of concerns: Cloud Run handles the webhook, Apps Script processes logic, and scripts automate the setup.\n\n* **✅ No Polling Required**\n  Entirely event-driven—avoids exceeding Apps Script's time-based trigger quotas and ensures higher reliability.\n\n---\n\n## 🧭 Workflow\n\n```mermaid\ngraph TD;\n  A[Gmail Inbox] --\u003e |Gmail API watch| B[Gmail Server] --\u003e |Publishes| C[Pub/Sub Topic] --\u003e |Triggers| D[Cloud Run] --\u003e |Call Web App| E[\"Apps Script \u003cbr\u003e(Processes new email)\"]\n```\n\n---\n\n## 🚀 Usage\n\n### 📥 Start Listening\n\nFrom the spreadsheet’s **Custom Menu**, choose **Start listening**. This will:\n\n* Send a `watch()` command to the Gmail API to start pushing notifications to the configured Pub/Sub topic.\n* Schedule a time-driven trigger to re-initiate the `watch()` **1 hour before expiration** (Gmail watches expire after 7 days).\n\n### 🛑 Stop Listening\n\nChoose **Stop listening** from the custom menu. This will:\n\n* Call the Gmail API to **stop** the current `watch()`.\n* **Delete the scheduled trigger** associated with the listener.\n\n---\n\n## 📁 Project Structure\n\n```\n/src\n│── cloud_run/\n│   ├── index.js              # Cloud Run webhook handler\n│   └── package.json\n│\n│── apps_script/\n│   ├── config.gs             # Sheet configuration and constants\n│   ├── triggers.gs           # Installable triggers for scheduling\n│   ├── utils.gs              # Helper functions\n│   ├── event-handlers.gs     # Gmail and Pub/Sub event processing\n│   └── package.json\n│\n└── scripts/\n    ├── config.ps1            # PowerShell setup for config and secrets\n    ├── deploy-cloud-run.ps1  # Deploy Cloud Run service and supporting infra\n    └── teardown-cloud-run.ps1# Full teardown of resources\n```\n\n---\n\n## ⚙️ Apps Script Setup\n\nBefore deploying the backend infrastructure, you must first set up and deploy your Apps Script project:\n\n### 1. Register Apps Script with a GCP Project\n\n1. Open your Apps Script project.\n2. Go to **Project Settings** → click **\"Change project\"** under Google Cloud Platform (GCP) Project.\n3. Link it to a new or existing GCP project.\n4. Enable the following APIs in the linked GCP project:\n   - Gmail API – obviously\n   - Cloud Build API – used to build and deploy containers\n   - Artifact Registry API – used to store Docker images\n   - Cloud Pub/Sub API – used to receive Gmail push notifications\n   - Cloud Run API – used to deploy and manage the Cloud Run service\n   - Eventarc API – used to create triggers from Pub/Sub to Cloud Run\n   - Secret Manager API – used to securely store and access sensitive information\n\n### 2. Enable Gmail API in Apps Script Services\n1. In the Apps Script editor, click + Services (left sidebar).\n2. Find and add Gmail API.\n3. This enables Gmail.Users.watch() functionality within your script.\n\n### 3. Deploy Apps Script as Web App\n\n1. Click **Deploy** → **New deployment**\n2. Choose **Web App** as the deployment type\n3. Set:\n   - **Execute as:** Me\n   - **Who has access:** Anyone\n4. Click **Deploy** and copy the **Web App URL**.\n5. Set this value in Secret Manager as `PUSH_PROXY_GAS_API_URL`.\n\n\u003e 🔐 You must also create a custom API key and store it in Secret Manager as `PUSH_PROXY_GAS_API_KEY`.\n\n---\n\n## 🔧 Setup \u0026 Deployment\n\nAll of the setup and deployment steps are handled by the provided PowerShell script.\n\nThis includes:\n\n* Pub/Sub topic setup\n* Service account creation and role bindings\n* Cloud Run deployment\n* Eventarc trigger setup\n* Permission bindings for Gmail and Cloud Run\n* Secret Manager integration\n\n\n### 🛠️ Before Running the Script\n\nYou must manually set up the following **Secret Manager entries**, as they contain sensitive data used during runtime:\n\n#### Required Secrets:\n\n| Secret Name              | Description                                                   |\n| ------------------------ | ------------------------------------------------------------- |\n| `PUSH_PROXY_GAS_API_URL` | The **Apps Script Web App URL** that Cloud Run will call      |\n| `PUSH_PROXY_GAS_API_KEY` | The **API key** to authenticate requests                      |\n\n#### To add these secrets:\n\n```bash\ngcloud secrets create PUSH_PROXY_GAS_API_URL --data-file=- \u003c\u003c\u003c \"https://script.google.com/macros/s/your-script-deployment-id/exec\"\ngcloud secrets create PUSH_PROXY_GAS_API_KEY --data-file=- \u003c\u003c\u003c \"your-secret-api-key\"\n```\n\n\u003e 🔐 Replace the placeholders with your actual values.\n\n\n### ▶️ To deploy everything:\n\n1. Open **PowerShell**.\n2. Navigate to the project root directory.\n3. Run:\n\n```powershell\npowershell -ExecutionPolicy ByPass -File \"path\\to\\scripts\\deploy-cloud-run.ps1\"\n```\n\n\u003e ⚠️ The `-ExecutionPolicy ByPass` flag is used to temporarily allow the script to run, even if your system's execution policy restricts unsigned scripts. This avoids permission errors without changing your system-wide settings.\n\n\n### 🔍 What the deployment script does\n\n```text\n# --- 1. Artifact Registry Setup ---\n# Creates Docker repository if not exists.\n\n# --- 2. Pub/Sub Topic Setup ---\n# Creates Pub/Sub topic and grants Gmail API permission to publish.\n\n# --- 3. Service Account Setup ---\n# Creates a dedicated service account and assigns necessary roles.\n\n# --- 4. Cloud Run Service Deployment ---\n# Builds Docker image, deploys Cloud Run, and assigns run.invoker role.\n\n# --- 5. Eventarc Trigger Creation ---\n# Connects Pub/Sub topic to Cloud Run with the correct trigger filter.\n```\n\nAfter deployment, the script will output the **Cloud Run URL**—which is automatically subscribed to Gmail push events and forwards them to your Apps Script Web App.\n\n---\n\n## 🧹 Teardown Instructions\n\nTo clean up all deployed resources, you can run the provided PowerShell script:\n\n```powershell\npowershell -ExecutionPolicy ByPass -File \"path\\to\\scripts\\teardown-cloud-run.ps1\"\n```\n\nThis script will reverse everything created during deployment.\n\n\n### 🔽 What the teardown script does\n\n```text\n# --- 1. Delete Eventarc Trigger ---\n# Removes the Eventarc trigger that connects the Pub/Sub topic to Cloud Run.\n\n# --- 2. Delete Cloud Run Service ---\n# Deletes the deployed service. The run.invoker IAM binding is deleted along with it.\n\n# --- 3. Remove IAM Bindings from Project ---\n# Cleans up project-level role bindings assigned to the service account.\n\n# --- 4. Delete Service Account ---\n# Deletes the custom service account created for this workflow.\n\n# --- 5. Delete Pub/Sub Topic ---\n# First removes the specific IAM policy binding for the Gmail service account,\n# then deletes the topic.\n\n# --- 6. Delete Artifact Registry Repository ---\n# Removes the Docker image repository used for Cloud Run builds.\n\n# --- 7. Delete GCS Bucket for Cloud Build Artifacts ---\n# Deletes the temporary Cloud Build bucket created during deployment.\n```\n\n\u003e ❗ **Note:** Secret Manager entries (`PUSH_PROXY_GAS_API_URL`, `PUSH_PROXY_GAS_API_KEY`) are **not deleted automatically**. You can delete them manually if no longer needed:\n\n```bash\ngcloud secrets delete PUSH_PROXY_GAS_API_URL\ngcloud secrets delete PUSH_PROXY_GAS_API_KEY\n```\n\n---\n\n## 🔐 Security Practices\n\nThis project follows several best practices to ensure secure handling of credentials, API keys, and execution environments:\n\n### ✅ Secrets Are Never Hardcoded\n\n* Sensitive data such as the Apps Script Web App URL and API key are stored in **Secret Manager**, not in source files.\n* These secrets are injected into the Cloud Run service at deploy time using the --set-secrets flag, ensuring a clean separation between code and credentials.\n\n### ✅ Principle of Least Privilege\n\n* A **dedicated service account** is created specifically for this project.\n* This service account is granted **only the required roles**, such as Pub/Sub Subscriber and Cloud Run Invoker, limiting the blast radius if compromised.\n\n### ✅ No Public Exposure of Apps Script\n\n* The Apps Script Web App is protected using a custom **API key**.\n* Only Cloud Run (authorized via the API key) is allowed to call the Apps Script endpoint.\n\n### ✅ Webhook Secured via API Key\n\n* Cloud Run includes the API key in the request header when forwarding events to Apps Script.\n* Apps Script validates the API key before processing any event.\n\n### ✅ Cloud Run HTTPS-Only by Default\n\n* Cloud Run only accepts **HTTPS traffic**.\n* All webhook communications are **TLS-encrypted** end to end.\n\n### ✅ Controlled Gmail Watch Access\n\n* For consumer Gmail accounts, only the account that owns the Apps Script project can initiate or stop the Gmail `watch()` subscription.\n* For Google Workspace accounts, the `watch()` is managed via the Gmail API using a service account with **domain-wide delegation**, so the account that initiates or stops the watch does not need to be the same account that owns the Apps Script project.\n* Pub/Sub topics are configured to only accept messages from Gmail’s **verified service account** (`gmail-api-push@system.gserviceaccount.com`).\n\n---\n\n## 📌 Real-World Use Cases\n\nThis architecture supports a wide range of realtime Gmail-driven workflows. Here are some examples:\n\n### ✅ Sample Use Case (in this project)\n\n* **Log new emails into a Google Sheet**\n  For every new email received, extract `datetime`, `from`, and `subject`, and append them as a new row in a tracking sheet.\n\n\n### 📦 Additional Real-World Use Cases\n\n* **📤 Auto-reply to Specific Senders**\n  Trigger an automated, templated response (e.g., for job applications or support emails) based on sender address or subject keywords.\n\n* **📅 Auto-create Calendar Events**\n  Parse structured emails (e.g., booking confirmations, meeting invites) and auto-add them to Google Calendar with proper details.\n\n* **📁 Label \u0026 Archive Emails Based on Rules**\n  Automatically apply Gmail labels or move emails to folders based on predefined rules or spreadsheet configurations.\n\n* **🚨 Alert \u0026 Escalation System**\n  Forward emails from high-priority senders or subjects (e.g., “urgent”, “failure”, “downtime”) to Slack, Telegram, or SMS using webhooks.\n\n* **📊 Real-time Dashboarding**\n  Populate Google Sheets with incoming customer inquiries or orders for live dashboards and reporting.\n\n* **🔄 Sync Gmail with External Systems**\n  On each new email, push metadata to a CRM, ticketing system, or internal API to keep systems in sync.\n\n* **🧹 Auto-cleanup of Inbox**\n  Automatically delete or archive promotional/notification emails after recording them elsewhere (e.g., in a Sheet or database).\n\n---\n\n## ⚠️ Why Not Use Apps Script as a Webhook Endpoint?\n\nApps Script **cannot act as a webhook endpoint** for services like Pub/Sub due to how it handles URL redirects:\n\n* Apps Script Web Apps are hosted at `script.google.com`, which **redirects** to `script.googleusercontent.com`.\n* The original endpoint (`script.google.com`) returns **HTTP 302**, not **200 OK**.\n* **Most webhook providers (Pub/Sub, FCM, etc.) do not follow redirects**, and treat this as a failure.\n* The push itself **does go through successfully**, but since Pub/Sub doesn't receive a 200 OK, it **retries the push repeatedly**.\n\n\u003e 💡 A workaround can be made so that Pub/Sub only sends **one push and never retries**, but this makes the workflow **unreliable**—you’d lose guaranteed delivery in case of failures.\n\nThis redirect behavior is **intentional for security reasons**. Platforms like Zapier, GoHighLevel, etc., do follow redirects and can work with Apps Script endpoints, but for **Pub/Sub**, a proxy (such as **Cloud Run**) is required for reliable delivery.\n\n\n## 💰 Cost Analysis\n\nFor typical usage—such as receiving **1,000 emails per day**—the cost is minimal and often stays within GCP’s free tier. Services like **Cloud Run**, **Pub/Sub**, **Eventarc**, and **Secret Manager** all have generous free quotas. Gmail push notifications are lightweight, so Pub/Sub and Eventarc costs remain near zero. Occasional Cloud Build usage for deployment and small artifact storage adds negligible cost.\n\nIn most cases, total monthly cost will be **under \\$1–2 USD**, or even free if you remain within free tier limits.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsangnandar%2Frealtime-gmail-listener","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsangnandar%2Frealtime-gmail-listener","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsangnandar%2Frealtime-gmail-listener/lists"}