{"id":51595376,"url":"https://github.com/monkey-mode/log-explorer-parser","last_synced_at":"2026-07-11T18:30:30.169Z","repository":{"id":355816868,"uuid":"1207820242","full_name":"monkey-mode/log-explorer-parser","owner":"monkey-mode","description":"A GCP Logs Explorer-style viewer for OpenSearch CSV exports. Built to make it easy to investigate logs when your DevOps team migrates GCP structured logs into OpenSearch and exports them with json_payload grouped into a single CSV column.","archived":false,"fork":false,"pushed_at":"2026-06-27T12:43:20.000Z","size":19976,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-27T14:20:10.909Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://log-explorer-parser.vercel.app","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/monkey-mode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-11T12:56:21.000Z","updated_at":"2026-06-27T12:43:12.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/monkey-mode/log-explorer-parser","commit_stats":null,"previous_names":["monkey-mode/log-explorer-parser"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/monkey-mode/log-explorer-parser","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monkey-mode%2Flog-explorer-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monkey-mode%2Flog-explorer-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monkey-mode%2Flog-explorer-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monkey-mode%2Flog-explorer-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/monkey-mode","download_url":"https://codeload.github.com/monkey-mode/log-explorer-parser/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monkey-mode%2Flog-explorer-parser/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35372639,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-11T02:00:05.354Z","response_time":104,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-11T18:30:28.591Z","updated_at":"2026-07-11T18:30:30.123Z","avatar_url":"https://github.com/monkey-mode.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Log Explorer\n\nA GCP Logs Explorer-style viewer for OpenSearch CSV exports. Built to make it easy to investigate logs when your DevOps team migrates GCP structured logs into OpenSearch and exports them with `json_payload` grouped into a single CSV column.\n\n## The Problem\n\nOpenSearch exports logs as CSV with this structure:\n\n| `@timestamp` | `kubernetes.container_name` | `json_payload` | `text_payload` |\n|---|---|---|---|\n| Apr 11, 2026 @ 17:30:51 | orch-payment-transaction | `{ \"message\": \"...\", \"severity\": \"ERROR\", ... }` | |\n| Apr 11, 2026 @ 17:30:52 | orch-payment-transaction | `-` | `plain log line from stdout` |\n\nThe `json_payload` column is a multiline JSON blob — impossible to read in a spreadsheet and hard to filter in a text editor.\n\n## Features\n\n- **Severity filter** — quickly isolate ERROR / WARN / INFO / DEBUG entries\n- **Full-text search** — searches across message, error, URL, caller, correlation ID, and the entire JSON payload\n- **Service filter** — filter to a specific `kubernetes.container_name`\n- **Correlation ID trace** — click any log row's \"Filter by Corr-ID\" button to show all logs in the same request trace\n- **Inline JSON tree** — click any row to expand a collapsible JSON tree viewer; nested JSON strings (e.g. `http_response_body`) are auto-parsed and rendered as trees\n- **Stack trace panel** — ERROR logs with a `stacktrace` field get a dedicated highlighted block\n- **HTTP summary** — request logs show `METHOD /path [status] latency` in the row summary\n- **Copy buttons** — copy Correlation ID or the full JSON payload with one click\n- **Drag \u0026 drop** — drop a CSV file anywhere on the page to load it\n- **Keyboard shortcuts** — `/` to focus search, `Esc` to clear Correlation ID filter\n\n## Getting Started\n\n```bash\nnpm install\nnpm run dev\n```\n\nOpen [http://localhost:3000](http://localhost:3000), then load your OpenSearch CSV export via the **Load CSV** button or drag \u0026 drop.\n\n## GCS bucket viewer (`/gcs`)\n\nWhen logs are routed to a Cloud Storage bucket via a **Cloud Logging → GCS sink** (instead of OpenSearch), use the GCS viewer at [http://localhost:3000/gcs](http://localhost:3000/gcs). It runs **entirely in the browser**: you **sign in with your own Google account**, then **choose a project and bucket** and read it directly via the Cloud Storage JSON API with your own IAM permissions — no server, no service-account key, no shared credentials. Handy when you only have **read/download** access and can't use GCP Log Explorer.\n\nAfter sign-in you pick a **project** (from the projects you can access) and a **bucket** — the bucket dropdown only lists buckets whose name ends with `k8s_container_logs`. Each bucket is organized as `\u003clogId\u003e/YYYY/MM/DD/\u003chourly\u003e.json`, where each object is newline-delimited Cloud Logging `LogEntry` JSON (top-level folders like `stdout/`, `stderr/`). You can then filter by **severity**, **service** (searchable multi-select), a **time range** (relative \"last N min/hours\" or an absolute from/to), full-text search, and correlation ID. Files over **50 MB** are skipped to keep the browser responsive.\n\n### One-time setup: create an OAuth Client ID\n\nYou need a public OAuth 2.0 **Client ID** so the browser can sign you in. In the [GCP Console](https://console.cloud.google.com/) for the project that owns the bucket:\n\n1. **APIs \u0026 Services → OAuth consent screen** — choose **Internal** if your account is in the same Google Workspace org as the project (simplest); otherwise choose **External** and add your email under **Test users**. Set an app name + support email, Save.\n2. **APIs \u0026 Services → Library** — enable both **Cloud Storage API** (read buckets/objects) and **Cloud Resource Manager API** (list your projects).\n3. **APIs \u0026 Services → Credentials → Create credentials → OAuth client ID** — Application type **Web application**.\n4. Under **Authorized JavaScript origins**, add `http://localhost:3000` (add your deployed origin later too). No redirect URI is needed. Click **Create**.\n5. Copy the **Client ID** (looks like `…apps.googleusercontent.com`).\n\n\u003e The Client ID is public and safe to ship. There is **no client secret** — this app uses the browser token flow.\n\n### Run\n\n```bash\ncp .env.local.example .env.local\n# then edit .env.local:\n#   NEXT_PUBLIC_GOOGLE_CLIENT_ID=\u003cyour client id\u003e.apps.googleusercontent.com\n#   NEXT_PUBLIC_GCS_PROJECT=\u003cproject to pre-select\u003e          # optional\n#   NEXT_PUBLIC_GCS_LOG_BUCKET=\u003cbucket to pre-select\u003e        # optional\n\nnpm install\nnpm run dev\n```\n\nOpen [`/gcs`](http://localhost:3000/gcs), click **Sign in with Google**, choose a **project** and **bucket**, then pick a log (e.g. `stdout`) and a date, click **List files**, and load one (or **Load all** for the day).\n\n\u003e Notes:\n\u003e - The token requests the `devstorage.read_only` and `cloud-platform.read-only` scopes (the latter is needed to list your projects). Access to any project/bucket/object is still enforced by your own IAM.\n\u003e - Your sign-in token is kept in `sessionStorage` so a page refresh keeps you signed in; it clears when you close the tab and expires after ~1 hour (then sign in again). Nothing is stored server-side.\n\u003e - Only buckets whose name ends with `k8s_container_logs` are shown in the bucket picker.\n\u003e - Files over the size cap (default 50 MB, set via `NEXT_PUBLIC_GCS_MAX_FILE_MB`; use `0` to disable) are skipped on load to keep the browser responsive.\n\u003e - Loading parses without buffering a full line array and keeps at most `NEXT_PUBLIC_GCS_MAX_ENTRIES` rows in memory (default 100,000; `0` = keep all). If a file has more, you're told how many were kept — narrow by hour/log/date to see the rest. The list also renders a bounded slice (refine the filters to focus results).\n\u003e - Downloads show per-file and overall progress, and each loaded object is cached locally in **IndexedDB** (keyed by bucket + object + generation) so reloading it is instant and offline. Use **Clear cache** in the file bar to purge it; nothing is written to your filesystem.\n\u003e - Sink files are written in hourly batches, so the most recent entries may lag by up to an hour — this is browse/near-real-time, not a true live tail.\n\n## CSV Format\n\nExport from OpenSearch with these columns (in this order):\n\n```\n@timestamp, kubernetes.container_name, json_payload, text_payload\n```\n\nThe parser handles multiline `json_payload` fields correctly — no pre-processing needed.\n\nWhen `json_payload` is `-` or blank (e.g. plain stdout/stderr lines), the parser falls back to `text_payload` and surfaces it as the log message. The `text_payload` column is optional — rows with only 3 columns are still accepted.\n\n## Browser Extension (live mode)\n\nThe [`extension/`](extension/) folder contains a Chrome side-panel extension that skips the CSV export entirely. It captures the OpenSearch Dashboards search response (`opensearch-with-long-numerals`) live and renders it with the same explorer UI.\n\n- **No file needed** — run a search in OSD and results stream into the side panel\n- **Single `_source` field** — parses each hit's `_source`, preferring `json_payload` and falling back to `text_payload`\n- **Tab-scoped** — the panel is enabled only on the OSD origin and reads only its own tab's data, so it never follows you to unrelated tabs\n- **No build step** — load the folder directly via `chrome://extensions` → **Load unpacked**\n\nSee [extension/README.md](extension/README.md) for loading instructions and how to point it at a different OSD host.\n\n## Project Structure\n\n```\nsrc/\n├── app/\n│   ├── page.tsx          # Entry point\n│   ├── layout.tsx\n│   └── globals.css\n├── components/\n│   ├── LogExplorer.tsx   # Main shell: state, filters, file loading\n│   ├── LogRow.tsx        # Individual log row + expandable detail panel\n│   └── JsonViewer.tsx    # Recursive collapsible JSON tree renderer\n└── lib/\n    ├── csvParser.ts      # RFC 4180 CSV parser + log entry parser\n    ├── osParser.ts       # OpenSearch response parser (_source → log entry)\n    └── logTypes.ts       # TypeScript types\n\nextension/                # Chrome side-panel extension (live OSD capture)\n├── manifest.json         # MV3 config\n├── interceptor.js        # MAIN world: patches fetch/XHR to capture the search response\n├── relay.js              # ISOLATED world: forwards captured JSON to the background\n├── background.js         # Service worker: per-tab storage + side-panel scoping\n├── parser.js             # _source → log entry (port of osParser.ts)\n└── sidepanel.{html,css,js} # The explorer UI\n```\n\n## Tech Stack\n\n- [Next.js 16](https://nextjs.org) (App Router)\n- [Tailwind CSS v4](https://tailwindcss.com)\n- TypeScript\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonkey-mode%2Flog-explorer-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmonkey-mode%2Flog-explorer-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonkey-mode%2Flog-explorer-parser/lists"}