{"id":44737588,"url":"https://github.com/pizzabits/secrets-snitcher","last_synced_at":"2026-04-02T12:21:50.699Z","repository":{"id":338530200,"uuid":"1158193241","full_name":"pizzabits/secrets-snitcher","owner":"pizzabits","description":"300 lines eBPF tool that shows which pods are reading your K8s secrets and how often.","archived":false,"fork":false,"pushed_at":"2026-03-24T23:03:26.000Z","size":3900,"stargazers_count":70,"open_issues_count":0,"forks_count":12,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-03-26T04:25:23.764Z","etag":null,"topics":["devsecops","ebpf","k8s"],"latest_commit_sha":null,"homepage":"https://github.com/pizzabits/secrets-snitcher/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pizzabits.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"buy_me_a_coffee":"ridner"}},"created_at":"2026-02-15T00:07:35.000Z","updated_at":"2026-03-24T23:03:29.000Z","dependencies_parsed_at":"2026-02-18T23:00:40.714Z","dependency_job_id":null,"html_url":"https://github.com/pizzabits/secrets-snitcher","commit_stats":null,"previous_names":["pizzabits/secrets-snitcher"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/pizzabits/secrets-snitcher","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pizzabits%2Fsecrets-snitcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pizzabits%2Fsecrets-snitcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pizzabits%2Fsecrets-snitcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pizzabits%2Fsecrets-snitcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pizzabits","download_url":"https://codeload.github.com/pizzabits/secrets-snitcher/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pizzabits%2Fsecrets-snitcher/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31306005,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T09:48:21.550Z","status":"ssl_error","status_checked_at":"2026-04-02T09:48:19.196Z","response_time":89,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["devsecops","ebpf","k8s"],"created_at":"2026-02-15T20:08:29.615Z","updated_at":"2026-04-02T12:21:50.688Z","avatar_url":"https://github.com/pizzabits.png","language":"Go","funding_links":["https://buymeacoffee.com/ridner"],"categories":[],"sub_categories":[],"readme":"# secrets-snitcher\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/eBPF-kernel_probe-3fb950?style=flat-square\u0026logo=linux\u0026logoColor=white\" alt=\"eBPF\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Kubernetes-secret_monitor-326ce5?style=flat-square\u0026logo=kubernetes\u0026logoColor=white\" alt=\"Kubernetes\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Prometheus-metrics-e6522c?style=flat-square\u0026logo=prometheus\u0026logoColor=white\" alt=\"Prometheus\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/v0.4.0-latest-79c0ff?style=flat-square\" alt=\"v0.4.0\"\u003e\n  \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-MIT-8b949e?style=flat-square\" alt=\"MIT\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\neBPF-powered Kubernetes secret access monitor.\n\nWatches which pods read secret files, how often, and whether they cache values in memory or re-read from disk on every request. Catches suspicious access patterns like a compromised pod hammering service account tokens.\n\n\u003e **NEW** - [Web dashboard](#web-dashboard) with live anomaly timeline, sparklines, and per-pod bar charts. [Prometheus /metrics](#prometheus-metrics) endpoint for Grafana integration. [Interactive TUI](#terminal-ui-tui) for terminal-native monitoring.\n\n## How it works\n\n```\n  ┌──────────────────┐       ┌──────────────┐       ┌──────────────────┐\n  │ Linux Kernel     │       │ secrets-     │       │ You / your tools │\n  │                  │       │ snitcher     │       │                  │\n  │ openat() syscall │──────\u003e│ aggregator   │──────\u003e│ GET :9100        │\n  │ on secret paths  │ eBPF  │ (60s window) │ HTTP  │ /api/v1/         │\n  │                  │       │              │       │ secret-access    │\n  └──────────────────┘       └──────────────┘       └──────────────────┘\n```\n\n~50 lines of BPF C sit inside the kernel, filtering at the syscall level before anything reaches userspace. Zero overhead for non-secret file access.\n\n1. **eBPF tracepoint** hooks `sys_enter_openat` - the syscall every file open goes through\n2. **Kernel-side path filter** checks if the filename starts with a known secret mount path. Non-matching opens are dropped inside the kernel, never copied to userspace\n3. **Perf buffer** streams matching events (pid, process name, filename, timestamp) to the Python aggregator\n4. **Rolling window aggregator** tracks per-pod read frequency over 60 seconds, resolves pod names via `/proc/{pid}/environ`\n5. **HTTP API** on port 9100 serves the current state as JSON\n\n![Demo](/demo/demo.gif)\n\n### What it watches\n\n| Mount path | Source |\n|---|---|\n| `/var/run/secrets/kubernetes.io/serviceaccount/` | Default K8s service account tokens |\n| `/var/secrets/` | Custom secret volume mounts |\n| `/mnt/secrets-store/` | CSI Secrets Store driver (Azure Key Vault, AWS Secrets Manager, HashiCorp Vault) |\n| `/run/secrets/` | Docker secrets / alternative mounts |\n\n### The `cached` field\n\nA service with `reads_per_sec \u003e= 1` is actively opening the secret file on every request - **not cached**. If you rotate or delete that secret, the service will immediately see the change (or break). A service with `reads_per_sec \u003c 1` has likely read the secret once and cached the value in memory. Nothing in Kubernetes tells you which behavior you're dealing with. This tool does.\n\n## Quick start\n\nNo Docker build required. The probe runs as a pod using a public Ubuntu image with BCC installed at runtime.\n\n```bash\n# 1. Create namespace + RBAC\nkubectl apply -f k8s/rbac.yaml\n\n# 2. Deploy the probe (installs BCC, mounts code as ConfigMap)\nkubectl apply -f k8s/pod-inline.yaml\n\n# 3. Wait for it to be ready (~30s for apt-get)\nkubectl -n secrets-snitcher wait --for=condition=Ready pod/secrets-snitcher --timeout=120s\n\n# 4. Port-forward and query\nkubectl -n secrets-snitcher port-forward svc/secrets-snitcher 9100:9100 \u0026\ncurl http://localhost:9100/api/v1/secret-access | jq\n```\n\nOr use the one-liner:\n\n```bash\ncurl -sL https://raw.githubusercontent.com/pizzabits/secrets-snitcher/main/install.sh | bash\n```\n\n## Testing with a malicious pod\n\nThe `demo/` directory includes a deliberately suspicious pod for testing:\n\n```bash\n# Deploy test secrets + a pod that hammers them\nkubectl apply -f demo/sample-secrets.yaml\nkubectl apply -f demo/malicious-pod.yaml\n\n# Wait a few seconds, then query the API\ncurl http://localhost:9100/api/v1/secret-access | jq\n\n# You should see \"totally-legit-app\" with very high reads_per_sec\n\n# Clean up\nkubectl delete -f demo/malicious-pod.yaml\nkubectl delete -f demo/sample-secrets.yaml\n```\n\nThe `totally-legit-app` pod runs a tight loop reading service account tokens as fast as possible. It will light up in the API as a clear outlier compared to normal workloads.\n\n## API\n\n### `GET /api/v1/secret-access`\n\nReturns all secret file access observed in the rolling window.\n\n```json\n{\n  \"timestamp\": \"2026-02-14T12:00:00+00:00\",\n  \"observation_window_seconds\": 60,\n  \"entries\": [\n    {\n      \"pod\": \"totally-legit-app\",\n      \"container\": \"sh\",\n      \"secret_path\": \"/var/run/secrets/kubernetes.io/serviceaccount/token\",\n      \"reads_per_sec\": 4872.3,\n      \"last_read\": \"2026-02-14T11:59:59+00:00\",\n      \"cached\": false\n    },\n    {\n      \"pod\": \"auth-service-7x8d\",\n      \"container\": \"auth-svc\",\n      \"secret_path\": \"/var/secrets/db-password\",\n      \"reads_per_sec\": 0.02,\n      \"last_read\": \"2026-02-14T11:59:30+00:00\",\n      \"cached\": true\n    }\n  ]\n}\n```\n\n| Field | Description |\n|-------|-------------|\n| `pod` | Pod name (resolved from `/proc/{pid}/environ`) |\n| `container` | Process name from the kernel (`comm`) |\n| `secret_path` | File path that was accessed |\n| `reads_per_sec` | Access frequency over the observation window |\n| `cached` | `true` if `reads_per_sec \u003c 1` (likely cached in memory) |\n\n### `GET /healthz`\n\n```json\n{\n  \"status\": \"ok\",\n  \"ebpf_attached\": true\n}\n```\n\n## Terminal UI (TUI)\n\nA live dashboard for watching secret access in real time. Built with Go + [Bubble Tea](https://github.com/charmbracelet/bubbletea).\n\n![TUI Dashboard](/demo/tui-demo.gif)\n\n```bash\n# Build\nmake tui\n\n# Run (with port-forward active)\n./secrets-snitcher-tui --api http://localhost:9100\n\n# Or try with the mock API for a quick demo\nmake mock-api            # Terminal 1\ncurl localhost:9100/toggle  # Terminal 2\n./secrets-snitcher-tui      # Terminal 3\n```\n\nFeatures: anomaly detection banner, color-coded read rates, NEW pod badges, vim-style navigation, search, sortable columns, resizable layout.\n\nSee [cmd/tui/README.md](cmd/tui/README.md) for full keyboard shortcuts and options, and [cmd/tui/DEVGUIDE.md](cmd/tui/DEVGUIDE.md) for an architecture walkthrough aimed at C/C++ developers.\n\n## Web Dashboard\n\nAn embedded web dashboard served directly from the snitcher pod - no extra dependencies, no CDN, no build step.\n\n```bash\n# With port-forward active, open in browser:\nopen http://localhost:9100\n\n# Or try with the mock API:\npython3 demo/mock-api.py    # Terminal 1\ncurl localhost:9100/toggle   # Terminal 2 (enable mock data)\nopen http://localhost:9100   # Browser\n```\n\nFeatures:\n- Dark theme with color-coded anomaly/active/cached status\n- Anomaly timeline chart (built from client-side history buffer)\n- Per-pod horizontal bar chart with log-scale reads/sec\n- Per-entry sparklines showing read rate trends\n- Configurable client-side history buffer (5min - unlimited)\n- Live updating every 2 seconds with connection status indicator\n- Pulsing anomaly banner when suspicious access is detected\n\nThe dashboard stores history in browser memory while the tab is open. Use the buffer dropdown to control retention. Data resets when you close the tab.\n\nTo disable: set `SNITCHER_DASHBOARD_ENABLED=false`.\n\n## Prometheus Metrics\n\nA `/metrics` endpoint exposes secret access data in Prometheus text format for Grafana integration.\n\n```bash\ncurl http://localhost:9100/metrics\n```\n\n```\n# HELP snitcher_secret_reads_per_second Current read rate over the observation window.\n# TYPE snitcher_secret_reads_per_second gauge\nsnitcher_secret_reads_per_second{pod=\"totally-legit-app\",container=\"sh\",secret_path=\"/var/run/secrets/kubernetes.io/serviceaccount/token\"} 4872.3\n```\n\nExposed metrics:\n\n| Metric | Type | Labels |\n|--------|------|--------|\n| `snitcher_secret_reads_per_second` | gauge | pod, container, secret_path |\n| `snitcher_secret_reads_total` | gauge | pod, container, secret_path |\n| `snitcher_secret_cached` | gauge | pod, container, secret_path |\n| `snitcher_secret_last_read_timestamp_seconds` | gauge | pod, container, secret_path |\n| `snitcher_observation_window_seconds` | gauge | - |\n| `snitcher_tracked_secrets` | gauge | - |\n| `snitcher_ebpf_attached` | gauge | - |\n\nPrometheus scrapes these gauges every 15-30 seconds and stores them in its own time-series database, giving you full historical data in Grafana even though secrets-snitcher only keeps a 60-second rolling window in memory.\n\nTo disable: set `SNITCHER_METRICS_ENABLED=false`.\n\n## Configuration\n\nBoth the dashboard and metrics endpoint are enabled by default and can be independently toggled via environment variables:\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `SNITCHER_DASHBOARD_ENABLED` | `true` | Serve web dashboard at `/` |\n| `SNITCHER_METRICS_ENABLED` | `true` | Serve Prometheus metrics at `/metrics` |\n| `SECRETS_SNITCHER_NO_TELEMETRY` | unset | Set to `1` to disable anonymous telemetry |\n\nSet these in the pod spec or pass as environment variables when running standalone.\n\n## Makefile targets\n\n```bash\nmake deploy     # kubectl apply rbac + pod-inline, wait for ready\nmake undeploy   # remove everything\nmake demo       # deploy a suspicious test pod\nmake demo-clean # remove the test pod\nmake logs       # tail the snitcher logs\nmake test       # pytest tests/ -v\nmake tui        # build the terminal UI binary\nmake mock-api   # run the mock API for TUI development\n```\n\n## Running tests\n\n```bash\npip install pytest\npytest tests/ -v\n```\n\n## Compatibility\n\nRequires Linux nodes with kernel headers and BCC support:\n\n| Platform | Status | Notes |\n|---|---|---|\n| **AKS** | Works | Ubuntu node images have kernel headers |\n| **EKS** | Works | Amazon Linux 2 / Ubuntu AMIs |\n| **GKE** | Partial | Ubuntu node images only. COS nodes lack kernel headers |\n| **K3s** | [Tested](k3s/) | Ubuntu 24.04 + kernel 6.x verified. See [platform guide](k3s/) |\n| **Bare metal / kubeadm** | Works | If kernel headers are installed |\n| **kind / minikube** | Works | For local testing |\n\nRequires privileged containers (`CAP_BPF` + `CAP_SYS_ADMIN` + `hostPID: true`).\n\n## Architecture\n\n```\nk8s/pod-inline.yaml\n├── ConfigMap (secrets-snitcher-code)\n│   ├── api.py          # All-in-one: BPF loader + aggregator + HTTP server\n│   ├── live.py         # Terminal monitor (port-forward + curses-style refresh)\n│   └── dashboard.html  # Embedded web dashboard (single-file, no dependencies)\n├── Pod (privileged, hostPID: true)\n│   ├── mounts /proc as /host-proc (read-only, for pod name resolution)\n│   ├── mounts /sys/kernel/debug (required by BCC)\n│   ├── mounts /lib/modules (read-only, kernel module symbols)\n│   ├── mounts /usr/src (read-only, kernel headers for BPF compilation)\n│   └── installs python3 + BCC at startup from ubuntu:22.04\n└── Service (ClusterIP :9100)\n```\n\nEverything ships as a single YAML file. No Docker build. The Python code lives in a ConfigMap, the pod installs BCC at startup from the ubuntu base image, and the BPF program compiles in-place on the node using the node's own kernel headers.\n\n**Why BCC instead of libbpf/CO-RE:** BCC compiles the BPF C at runtime using Clang, which means it works on any kernel version without pre-compiled bytecode. The tradeoff is startup time (~30s for apt-get + compile) and requiring kernel headers on the node. A production hardened version would use libbpf with CO-RE for faster startup and smaller footprint.\n\n## Platform guides\n\n| Platform | Guide |\n|---|---|\n| K3s | [k3s/README.md](k3s/) - tested on Ubuntu 24.04 + kernel 6.x, includes verified output |\n\nMore platforms coming. If you've tested on a platform not listed here, open a PR with a guide under `\u003cplatform\u003e/README.md`.\n\n## Contributing\n\nPRs welcome. Before submitting:\n\n1. **Run tests:** `pytest tests/ -v` - all must pass\n2. **Test on a real cluster** if your change touches `k8s/` manifests or the BPF program. The BPF C compiles at runtime on the node, so YAML-level correctness isn't enough.\n3. **One concern per PR.** Don't bundle unrelated changes.\n4. **Platform guides** go in `\u003cplatform\u003e/README.md` with: prerequisites, deploy steps, verified output showing real data, and known issues.\n\n## Telemetry\n\nsecrets-snitcher sends a single anonymous ping when the probe starts (at most once per 24 hours). It reports: tool version, kernel version, CPU architecture, Python version, and whether it's running as a DaemonSet or standalone.\n\nNo IP addresses, hostnames, secret paths, or cluster information is collected. To opt out: set `SECRETS_SNITCHER_NO_TELEMETRY=1`.\n\n### Why we collect this\n\nThis is a solo open source project. Telemetry is the only way to know if anyone is actually using it, what kernels and platforms to support, and whether to keep investing time in it. Without it, the project is built blind.\n\n### What we send\n\n| Field | Example | Why |\n|-------|---------|-----|\n| tool | secrets-snitcher | Which tool sent the ping |\n| version | 0.4.0 | Know which versions are in the wild |\n| kernel | 6.17.0-1008-gcp | Know which kernel offsets to support |\n| arch | x86_64 | Know if ARM support matters |\n| python | 3.10.12 | Know minimum Python version to target |\n| deployment_type | pod / daemonset / standalone | Know how people deploy |\n| uptime_hours | 48 | Distinguish \"tried once\" from \"running in prod\" |\n| distinct_id | a1b2c3... (SHA-256 hash) | Count unique installs without identifying anyone |\n\n### How the install ID works\n\nTo count unique installations without collecting identifiable information, secrets-snitcher reads your cluster's `kube-system` namespace UID (a UUID that Kubernetes assigns when the cluster is created). This is why `rbac.yaml` includes a ClusterRole with read access to the `kube-system` namespace - it's only used to generate the telemetry hash.\n\nThe raw UID never leaves your cluster. It's hashed with SHA-256 before sending. The hash cannot be reversed. On standalone installs (no Kubernetes), `/etc/machine-id` is used instead.\n\n## Limitations\n\nThis is a weekend project / proof of concept, not production-hardened. Known gaps:\n\n- No persistence -- data is lost on pod restart\n- No authentication on the HTTP API\n- BCC requires kernel headers installed on every node\n- Rolling window is in-memory only (no cross-node aggregation)\n- Pod name resolution reads `/proc` which may not work in all container runtimes\n\n\n## License\n\n[MIT](LICENSE) - Copyright (c) 2026 Michael Ridner\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpizzabits%2Fsecrets-snitcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpizzabits%2Fsecrets-snitcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpizzabits%2Fsecrets-snitcher/lists"}