{"id":51915812,"url":"https://github.com/ik-labs/splunk-app-lifecycle-copilot","last_synced_at":"2026-07-27T10:02:47.692Z","repository":{"id":364483750,"uuid":"1268102246","full_name":"ik-labs/splunk-app-lifecycle-copilot","owner":"ik-labs","description":"Self-healing loops that carry a Splunk app from raw logs to AppInspect-green to cost-clean SPL, validated through the Splunk MCP server with an auditable provenance ledger.","archived":false,"fork":false,"pushed_at":"2026-06-13T06:46:04.000Z","size":164,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-13T08:14:43.349Z","etag":null,"topics":["agentic-ops","appinspect","hackathon","mcp-server","self-healing","spl","splunk","splunk-mcp"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ik-labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-13T06:27:34.000Z","updated_at":"2026-06-13T06:46:07.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ik-labs/splunk-app-lifecycle-copilot","commit_stats":null,"previous_names":["ik-labs/splunk-app-lifecycle-copilot"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ik-labs/splunk-app-lifecycle-copilot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ik-labs%2Fsplunk-app-lifecycle-copilot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ik-labs%2Fsplunk-app-lifecycle-copilot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ik-labs%2Fsplunk-app-lifecycle-copilot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ik-labs%2Fsplunk-app-lifecycle-copilot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ik-labs","download_url":"https://codeload.github.com/ik-labs/splunk-app-lifecycle-copilot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ik-labs%2Fsplunk-app-lifecycle-copilot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35947061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-27T02:00:06.776Z","response_time":101,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic-ops","appinspect","hackathon","mcp-server","self-healing","spl","splunk","splunk-mcp"],"created_at":"2026-07-27T10:02:46.899Z","updated_at":"2026-07-27T10:02:47.676Z","avatar_url":"https://github.com/ik-labs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Splunk App Lifecycle Copilot\n\nSplunk App Lifecycle Copilot is a hackathon MVP for carrying a Splunk app from raw logs to a CIM-clean, AppInspect-green package with self-healing loops and an auditable provenance trail.\n\nPositioning: Splunk's own AI can explain failures. This project resolves, validates, and remembers them.\n\n## Judges: 60-Second Path\n\n**Fastest look — no Splunk, no MCP, no install (needs only [Bun](https://bun.sh)):**\n\n```bash\nmake dashboard          # cd ui/dashboard \u0026\u0026 bun install \u0026\u0026 bun run dev\n```\n\nOpen the printed URL. The dashboard lands on a **Lifecycle** overview of both\nself-heal loops, then drill into the **Onboarding** and **AppInspect** stages.\nThe **Provenance Ledger** panel is the \"resolve, validate, *and remember*\"\nthesis made literal — every diagnosis, patch, rationale, and validation result\nfrom a verified run, replayed from committed demo events.\n\n**Run the real software end-to-end:**\n\n```bash\nmake setup              # Python 3.13 venv + install (see Requirements re: 3.13)\nmake demo               # runs the loops, prints where every artifact landed\n```\n\n`make demo` always runs the dependency-free AppInspect loop. It also runs the\nlive onboarding loop (HEC ingest → MCP `splunk_run_query` validation) when\n`.env` carries the Splunk + MCP credentials; otherwise it points back to the\nzero-deps dashboard replay. `make help` lists every target.\n\n## Hackathon Scope\n\n- Event: Splunk Agentic Ops Hackathon\n- Track: Platform \u0026 Developer Experience\n- Bonus target: Best Use of Splunk MCP Server\n- Submission deadline: June 15, 2026, 9:00 AM PDT\n\nThe MVP builds three loops on one shared self-heal engine:\n\n- Stage 1, onboarding: raw UPI/GST-style logs -\u003e inline `rex` / `eval` extraction candidates -\u003e validation against real Splunk events through `splunk_run_query` -\u003e final `props.conf` / `transforms.conf` only after convergence.\n- Stage 2, AppInspect: deliberately broken app -\u003e AppInspect JSON failures -\u003e deterministic patch functions -\u003e re-run until green.\n- Stage 4, cost-aware SPL lint: a deliberately costly search -\u003e deterministic cost findings (`index=*`, no time bound, unbounded `| sort`) -\u003e deterministic rewrites -\u003e re-lint until clean. Static analysis, no live Splunk.\n\nStages 3 (scaffold + test data) and 5 (dashboard migration) remain architecture-only future extensions.\n\n## Current Repo Contents\n\n- `SCOPE.md`: source-of-truth scope, build sequencing, risks, and submission checklist.\n- `UX_DEMO_PLAN.md`: dashboard/CLI/IDE surfaces and the agent-to-UI event contract.\n- `architecture_diagram.md`: required architecture diagram artifact.\n- `demo/architecture_demo.md`: technical, presentation-ready architecture walkthrough (system overview, the self-heal engine, live-MCP onboarding, and live-mode SSE — five Mermaid diagrams) for the demo.\n- `docker-compose.yml`: local Splunk Enterprise container with HEC enabled.\n- `smoke_test.py`: Day-1 Splunk SDK, HEC, search, and AppInspect smoke test.\n- `fixtures/onboarding/sample_upi.log`: 150-line synthetic UPI transaction fixture.\n- `fixtures/appinspect/broken_app`: AppInspect failure fixture for the first build milestone.\n\n## Requirements\n\n- Python 3.13 (use 3.13 specifically, not 3.14 — see note below)\n- Docker Desktop\n- Homebrew `libmagic` on macOS for `splunk-appinspect`\n- Splunk Enterprise Docker image (`splunk/splunk:latest`)\n- Splunk MCP Server app for the onboarding loop\n- Optional: Splunk AI Assistant for `saia_generate_spl`, `saia_explain_spl`, and `saia_optimize_spl`\n\nPython dependency notes:\n\n- PyPI package is `splunk-sdk`; import name is `splunklib`.\n- `splunklib.ai` requires `splunk-sdk\u003e=3.0.0`, which requires Python 3.13+.\n- Pin the interpreter to **Python 3.13**, not 3.14. On 3.14, AppInspect's bundled\n  Python static analyzer fails to initialize and reports every Python check as an\n  `error` (\"Python analyzer is failed in initialization\"). That leaves the\n  AppInspect self-heal loop unable to reach a clean result even after the three\n  real failures are patched. On macOS: `brew install python@3.13` and build the\n  venv with `/opt/homebrew/opt/python@3.13/bin/python3.13 -m venv .venv`.\n- `splunk_run_query` is the required MCP validation tool. `saia_*` tools are optional graceful enhancement only.\n\n## Setup\n\n```bash\npython3.13 -m venv .venv\nsource .venv/bin/activate\npip install -e .\ncp .env.example .env\n```\n\nEdit `.env` with your local values. The MCP token must be the encrypted token generated by the Splunk MCP Server app. It is not a plain Splunk REST bearer token. Do not build the demo around OAuth; OAuth for Splunk MCP is Controlled Access / closed preview.\n\nOn macOS, install AppInspect's system dependency if needed:\n\n```bash\nbrew install libmagic\n```\n\nStart Splunk:\n\n```bash\ndocker compose up -d\n```\n\nPorts are driven by `.env` (`SPLUNK_WEB_PORT`, `SPLUNK_HEC_PORT`, `SPLUNK_MGMT_PORT`).\nIf host port 8088 is already in use, set a different `SPLUNK_HEC_PORT` (the local dev\nbox uses `18088`); the onboarding loop reads the same value, so HEC ingest stays in sync.\n\nWait for the container to become healthy, then run:\n\n```bash\npython smoke_test.py\n```\n\n## AppInspect Fixture Check\n\nThe current AppInspect fixture is designed to fire three deterministic failures:\n\n- `check_that_local_does_not_exist`: forbidden `local/` directory.\n- `check_user_seed_conf_deny_list`: forbidden `default/user-seed.conf`.\n- `check_if_outputs_conf_exists`: forwarding enabled in `default/outputs.conf`.\n\nRun the self-heal loop:\n\n```bash\ncopilot appinspect fixtures/appinspect/broken_app --out runs/appinspect-demo\n```\n\nThe command copies the fixture into `runs/appinspect-demo/work/broken_app`,\npatches only that working copy, and writes:\n\n- `runs/appinspect-demo/appinspect/iteration-XX.json`\n- `runs/appinspect-demo/events.jsonl`\n- `runs/appinspect-demo/events.json`\n- `runs/appinspect-demo/provenance.jsonl`\n- `runs/appinspect-demo/summary.json`\n\nValidate it directly:\n\n```bash\nsplunk-appinspect inspect fixtures/appinspect/broken_app \\\n  --mode test \\\n  --data-format json \\\n  --output-file /tmp/broken_app_result.json\n```\n\nExpected summary:\n\n```text\nfailure: 3\nerror: 0\n```\n\n## Splunk MCP Server Setup (onboarding loop only)\n\nThe onboarding loop is the only path that needs the Splunk MCP Server app and an\nencrypted token. The static loops (AppInspect, SPL lint), the dashboard, and Live\nmode need none of this — skip this section if you are not running onboarding.\n\nVerified against Splunk MCP Server **v1.2.0** on `splunk/splunk:latest`. The\ncommands below use `$SPLUNK_MGMT_PORT` (default `8089`) and the admin password\nfrom `.env`.\n\n**1. Install the app.** Download \"Splunk MCP Server\" from Splunkbase, then in\nSplunk Web go to **Apps -\u003e Manage Apps -\u003e Install app from file** and upload the\n`.tar.gz` (or `docker cp` it into `$SPLUNK_HOME/etc/apps/` and restart the\ncontainer). Confirm it is installed and enabled:\n\n```bash\ncurl -sk -u \"admin:$SPLUNK_PASSWORD\" \\\n  \"https://localhost:8089/services/apps/local/Splunk_MCP_Server?output_mode=json\" \\\n  | python3 -c \"import sys,json;e=json.load(sys.stdin)['entry'][0]['content'];print('enabled' if not e.get('disabled') else 'DISABLED','v'+str(e.get('version')))\"\n```\n\n**2. Enable token authentication** (once; idempotent):\n\n```bash\ncurl -sk -u \"admin:$SPLUNK_PASSWORD\" -X POST \\\n  \"https://localhost:8089/services/admin/token-auth/tokens_auth\" -d disabled=0\n```\n\n**3. Mint the encrypted token.** The app RSA-encrypts a JWT (audience `mcp`);\n`mcp.conf` sets `require_encrypted_token = true`, so this is **not** a plain REST\nbearer token. The `+` in the relative expiry **must** be URL-encoded as `%2B` or\nthe app rejects the request:\n\n```bash\ncurl -sk -u \"admin:$SPLUNK_PASSWORD\" \\\n  \"https://localhost:8089/services/mcp_token?username=admin\u0026expires_on=%2B30d\"\n# -\u003e {\"token\": \"\u003cencrypted-token\u003e\"}   (valid 30 days)\n```\n\n**4. Put the values in `.env`:**\n\n```bash\nSPLUNK_MCP_ENDPOINT=https://localhost:8089/services/mcp\nSPLUNK_MCP_ENCRYPTED_TOKEN=\u003cthe token value from step 3\u003e\nSPLUNK_MCP_TLS_VERIFY=false   # local self-signed Docker dev\n```\n\nThe token expires after 30 days; re-mint by repeating steps 2-3. The `saia_*`\ntools are optional and only appear when Splunk AI Assistant is installed — the\nonboarding loop succeeds with `splunk_run_query` alone.\n\n## Onboarding MCP Slice\n\nThe onboarding slice is live-only: it ingests `fixtures/onboarding/sample_upi.log`\nthrough HEC, validates inline SPL candidates with MCP `splunk_run_query`, and\nhard-fails if the Splunk MCP Server app or `splunk_run_query` tool is unavailable.\nIt does not use Splunk AI Assistant, Splunk SDK search fallback, or generated\n`props.conf` / `transforms.conf` yet.\n\nRequired `.env` values:\n\n- `SPLUNK_HEC_TOKEN`\n- `SPLUNK_ONBOARDING_INDEX=main`\n- `SPLUNK_ONBOARDING_SOURCETYPE=upi_gateway_raw`\n- `SPLUNK_MCP_ENDPOINT`\n- `SPLUNK_MCP_ENCRYPTED_TOKEN`\n- `SPLUNK_MCP_TLS_VERIFY=false` for local self-signed Docker dev\n\nRun it:\n\n```bash\ncopilot onboard fixtures/onboarding/sample_upi.log --out runs/onboarding-demo\n```\n\nExpected flow: `candidate-00` fails coverage checks, the deterministic patcher\nswitches to `candidate-01`, MCP revalidation passes, six CIM mapping events and\ntwo PII flag events are written for dashboard replay.\n\n## Cost-Aware SPL Lint (Stage 4)\n\nThe SPL lint loop is static like AppInspect — no live Splunk required. It lints\na search for cost anti-patterns, heals each with a deterministic rewrite, and\nre-lints until clean.\n\n```bash\ncopilot lint fixtures/spl_lint/costly_search.spl --out runs/spl-lint-demo\n```\n\nThe fixture fires three findings, each healed in its own iteration:\n\n- `spl_wildcard_index`: `index=*` scans every index -\u003e rewritten to `index=main`.\n- `spl_all_time`: no `earliest`/`latest` bound -\u003e prepended `earliest=-24h`.\n- `spl_unbounded_sort`: `| sort` with no limit -\u003e capped at `| sort 1000`.\n\nIt writes the same artifact set as the other loops\n(`spl/iteration-XX.json`, `events.json`, `provenance.jsonl`, `summary.json`),\nso the run drops straight into the dashboard's SPL Lint stage.\n\n## Demo Flow\n\n1. Terminal starts the agent and proves this is real software.\n2. Dashboard shows onboarding: fields appear as `splunk_run_query` validates inline extraction candidates against the 150-line fixture, PII is flagged, and CIM mapping converges.\n3. Dashboard shows AppInspect: three red failures are diagnosed, patched by deterministic functions, and revalidated to green.\n4. VS Code cutaway shows the same agent entry point from the IDE.\n5. Provenance ledger shows every diagnosis, patch, rationale, and validation result.\n\n## Dashboard Replay Mode\n\nThe dashboard replays all three self-heal loops. It opens on a **Lifecycle**\noverview that summarizes every loop side by side (status, failures healed,\niterations, MCP calls) to make the one-engine/many-loops thesis legible at a\nglance. From there, use the sidebar to open the **Onboarding**, **AppInspect**,\nor **SPL Lint** stage; each renders committed demo events\n(`demo/onboarding_events.json`, `demo/appinspect_events.json`,\n`demo/spl_lint_events.json`) and requires no Splunk, MCP, or live WebSocket. The\nonboarding stage adds an MCP tool-call count and a CIM-mapping / PII panel\nsourced from a verified live run.\n\nEvery stage also renders a **Provenance Ledger** panel: the complete, durable\naudit trail read from the persisted `*_provenance.jsonl` — each entry's\ndiagnosis, patch, rationale, validation result, changed paths, and timestamp.\nThat panel is the \"and remember\" half of the thesis, and it is exactly what a\nreviewer would inspect to trust an automated fix.\n\n```bash\ncd ui/dashboard\nbun install\nbun run dev\n```\n\nVerification:\n\n```bash\nbun run test\nbun run build\n```\n\n### Live Mode\n\nThe dashboard can also stream a self-heal loop **as it runs**, instead of\nreplaying committed events. Start the SSE server, then click **Go Live** on the\nAppInspect or SPL Lint stage:\n\n```bash\ncopilot serve            # or: make serve  (defaults to 127.0.0.1:8765)\n```\n\nThe server (`lifecycle_copilot.server`) runs the chosen static loop — AppInspect\nor SPL lint, neither needs a live Splunk — in a background thread and streams\neach event over Server-Sent Events. The browser feeds those events through the\nsame reducer used for replay, so live and replay render identically; only the\nevent source differs. Point the dashboard at a non-default server with\n`VITE_LIVE_URL`. Onboarding is replay-only here because it requires Splunk + MCP.\n\n## Run from VS Code\n\nThe repo ships a `.vscode/` workspace so the agent runs from the IDE — the same\n`copilot` entry point, surfaced as **Run Task** and **Run and Debug** entries:\n\n- **Terminal → Run Task** lists each loop: *AppInspect self-heal*, *SPL lint\n  self-heal*, *Onboard (live MCP)*, plus *Live stream server*, *Dashboard*, and\n  *Run Python tests*. AppInspect and SPL lint need no Splunk.\n- **Run and Debug** (`launch.json`) starts any loop under `debugpy` with the\n  venv interpreter, so you can set breakpoints in the self-heal engine.\n- `extensions.json` recommends the Python, debugpy, and Bun extensions.\n\nRun `make setup` first so `.venv` exists; the tasks call `.venv/bin/copilot`.\n\n## Design Principle\n\nThe self-heal engine is intentionally constrained. The LLM produces diagnosis and rationale text; deterministic patch functions make file changes. That gives the demo repeatability, keeps patch provenance reviewable, and makes the platform thesis credible.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fik-labs%2Fsplunk-app-lifecycle-copilot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fik-labs%2Fsplunk-app-lifecycle-copilot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fik-labs%2Fsplunk-app-lifecycle-copilot/lists"}