{"id":31183752,"url":"https://github.com/kubeflow/mcp-apache-spark-history-server","last_synced_at":"2025-09-19T18:10:54.625Z","repository":{"id":306292199,"uuid":"1009230091","full_name":"kubeflow/mcp-apache-spark-history-server","owner":"kubeflow","description":"MCP Server for Apache Spark History Server. The bridge between Agnetic AI and Apache Spark.","archived":false,"fork":false,"pushed_at":"2025-09-15T21:59:34.000Z","size":2423,"stargazers_count":71,"open_issues_count":21,"forks_count":18,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-09-15T22:12:55.336Z","etag":null,"topics":["apache-spark","big-data","data-processing","kubernetes","mcp","mcp-server"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kubeflow.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-06-26T19:38:52.000Z","updated_at":"2025-09-15T21:59:38.000Z","dependencies_parsed_at":"2025-08-25T23:18:09.646Z","dependency_job_id":"94871fcd-e440-4c8d-a732-357e54a3a575","html_url":"https://github.com/kubeflow/mcp-apache-spark-history-server","commit_stats":null,"previous_names":["deepdiagnostix-ai/mcp-apache-spark-history-server","deepdiagnostix-ai/spark-history-server-mcp","kubeflow/mcp-apache-spark-history-server"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/kubeflow/mcp-apache-spark-history-server","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeflow%2Fmcp-apache-spark-history-server","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeflow%2Fmcp-apache-spark-history-server/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeflow%2Fmcp-apache-spark-history-server/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeflow%2Fmcp-apache-spark-history-server/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kubeflow","download_url":"https://codeload.github.com/kubeflow/mcp-apache-spark-history-server/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubeflow%2Fmcp-apache-spark-history-server/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275696223,"owners_count":25511352,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-17T02:00:09.119Z","response_time":84,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","big-data","data-processing","kubernetes","mcp","mcp-server"],"created_at":"2025-09-19T18:10:53.094Z","updated_at":"2025-09-19T18:10:54.616Z","avatar_url":"https://github.com/kubeflow.png","language":"Python","funding_links":[],"categories":["Data \u0026 Analytics","📚 Projects (2474 total)"],"sub_categories":["Data Platforms","MCP Servers"],"readme":"# MCP Server for Apache Spark History Server\n\n[![CI](https://github.com/kubeflow/mcp-apache-spark-history-server/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/kubeflow/mcp-apache-spark-history-server/actions)\n[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)\n[![MCP](https://img.shields.io/badge/MCP-Compatible-green.svg)](https://modelcontextprotocol.io/)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![Kubeflow](https://img.shields.io/badge/Kubeflow-Official-orange.svg)](https://github.com/kubeflow)\n[![Slack](https://img.shields.io/badge/Slack-CNCF%20%23mcp--apache--spark--history--server-purple.svg)](https://cloud-native.slack.com/archives/C09FRRM6QM7)\n\n\u003e **🤖 Connect AI agents to Apache Spark History Server for intelligent job analysis and performance monitoring**\n\nTransform your Spark infrastructure monitoring with AI! This Model Context Protocol (MCP) server enables AI agents to analyze job performance, identify bottlenecks, and provide intelligent insights from your Spark History Server data.\n\n## 🎯 What is This?\n\n**Spark History Server MCP** bridges AI agents with your existing Apache Spark infrastructure, enabling:\n\n- 🔍 **Query job details** through natural language\n- 📊 **Analyze performance metrics** across applications\n- 🔄 **Compare multiple jobs** to identify regressions\n- 🚨 **Investigate failures** with detailed error analysis\n- 📈 **Generate insights** from historical execution data\n\n📺 **See it in action:**\n\n[![Watch the demo video](https://img.shields.io/badge/YouTube-Watch%20Demo-red?style=for-the-badge\u0026logo=youtube)](https://www.youtube.com/watch?v=e3P_2_RiUHw)\n\n\n## 🏗️ Architecture\n\n```mermaid\ngraph TB\n    A[🤖 AI Agent/LLM] --\u003e F[📡 MCP Client]\n    B[🦙 LlamaIndex Agent] --\u003e F\n    C[🌐 LangGraph] --\u003e F\n    D[�️ Claudep Desktop] --\u003e F\n    E[🛠️ Amazon Q CLI] --\u003e F\n\n    F --\u003e G[⚡ Spark History MCP Server]\n\n    G --\u003e H[🔥 Prod Spark History Server]\n    G --\u003e I[🔥 Staging Spark History Server]\n    G --\u003e J[🔥 Dev Spark History Server]\n\n    H --\u003e K[📄 Prod Event Logs]\n    I --\u003e L[📄 Staging Event Logs]\n    J --\u003e M[📄 Dev Event Logs]\n```\n\n**🔗 Components:**\n- **🔥 Spark History Server**: Your existing infrastructure serving Spark event data\n- **⚡ MCP Server**: This project - provides MCP tools for querying Spark data\n- **🤖 AI Agents**: LangChain, custom agents, or any MCP-compatible client\n\n## ⚡ Quick Start\n\n### 📋 Prerequisites\n- 🔥 Existing Spark History Server (running and accessible)\n- 🐍 Python 3.12+\n- ⚡ [uv](https://docs.astral.sh/uv/getting-started/installation/) package manager\n\n### 🚀 Setup \u0026 Testing\n\n```bash\ngit clone https://github.com/kubeflow/mcp-apache-spark-history-server.git\ncd mcp-apache-spark-history-server\n\n# Install Task (if not already installed)\nbrew install go-task  # macOS, see https://taskfile.dev/installation/ for others\n\n# Setup and start testing\ntask start-spark-bg            # Start Spark History Server with sample data (default Spark 3.5.5)\n# Or specify a different Spark version:\n# task start-spark-bg spark_version=3.5.2\ntask start-mcp-bg             # Start MCP Server\n\n# Optional: Opens MCP Inspector on http://localhost:6274 for interactive testing\n# Requires Node.js: 22.7.5+ (Check https://github.com/modelcontextprotocol/inspector for latest requirements)\ntask start-inspector-bg       # Start MCP Inspector\n\n# When done, run `task stop-all`\n```\n\nIf you just want to run the MCP server without cloning the repository:\n\n```bash\n# Run with uv without installing the module\nuvx --from mcp-apache-spark-history-server spark-mcp\n\n# OR run with pip and python. Use of venv is highly encouraged.\npython3 -m venv spark-mcp \u0026\u0026 source spark-mcp/bin/activate\npip install mcp-apache-spark-history-server\npython3 -m spark_history_mcp.core.main\n# Deactivate venv\ndeactivate\n```\n### ⚙️ Server Configuration\nEdit `config.yaml` for your Spark History Server:\n\n**Config File Options:**\n- Command line: `--config /path/to/config.yaml` or `-c /path/to/config.yaml`\n- Environment variable: `SHS_MCP_CONFIG=/path/to/config.yaml`\n- Default: `./config.yaml`\n```yaml\nservers:\n  local:\n    default: true\n    url: \"http://your-spark-history-server:18080\"\n    auth:  # optional\n      username: \"user\"\n      password: \"pass\"\nmcp:\n  transports:\n    - streamable-http # streamable-http or stdio.\n  port: \"18888\"\n  debug: true\n```\n\n\n### 📊 Sample Data\nThe repository includes real Spark event logs for testing:\n- `spark-bcec39f6201b42b9925124595baad260` - ✅ Successful ETL job\n- `spark-110be3a8424d4a2789cb88134418217b` - 🔄 Data processing job\n- `spark-cc4d115f011443d787f03a71a476a745` - 📈 Multi-stage analytics job\n\nSee **[TESTING.md](TESTING.md)** for using them.\n\n## 📸 Screenshots\n\n### 🔍 Get Spark Application\n![Get Application](screenshots/get-application.png)\n\n### ⚡ Job Performance Comparison\n![Job Comparison](screenshots/job-compare.png)\n\n\n## 🛠️ Available Tools\n\n\u003e **Note**: These tools are subject to change as we scale and improve the performance of the MCP server.\n\nThe MCP server provides **18 specialized tools** organized by analysis patterns. LLMs can intelligently select and combine these tools based on user queries:\n\n### 📊 Application Information\n*Basic application metadata and overview*\n| 🔧 Tool | 📝 Description |\n|---------|----------------|\n| `list_applications` | 📋 Get a list of all applications available on the Spark History Server with optional filtering by status, date ranges, and limits |\n| `get_application` | 📊 Get detailed information about a specific Spark application including status, resource usage, duration, and attempt details |\n\n### 🔗 Job Analysis\n*Job-level performance analysis and identification*\n| 🔧 Tool | 📝 Description |\n|---------|----------------|\n| `list_jobs` | 🔗 Get a list of all jobs for a Spark application with optional status filtering |\n| `list_slowest_jobs` | ⏱️ Get the N slowest jobs for a Spark application (excludes running jobs by default) |\n\n### ⚡ Stage Analysis\n*Stage-level performance deep dive and task metrics*\n| 🔧 Tool | 📝 Description |\n|---------|----------------|\n| `list_stages` | ⚡ Get a list of all stages for a Spark application with optional status filtering and summaries |\n| `list_slowest_stages` | 🐌 Get the N slowest stages for a Spark application (excludes running stages by default) |\n| `get_stage` | 🎯 Get information about a specific stage with optional attempt ID and summary metrics |\n| `get_stage_task_summary` | 📊 Get statistical distributions of task metrics for a specific stage (execution times, memory usage, I/O metrics) |\n\n### 🖥️ Executor \u0026 Resource Analysis\n*Resource utilization, executor performance, and allocation tracking*\n| 🔧 Tool | 📝 Description |\n|---------|----------------|\n| `list_executors` | 🖥️ Get executor information with optional inactive executor inclusion |\n| `get_executor` | 🔍 Get information about a specific executor including resource allocation, task statistics, and performance metrics |\n| `get_executor_summary` | 📈 Aggregates metrics across all executors (memory usage, disk usage, task counts, performance metrics) |\n| `get_resource_usage_timeline` | 📅 Get chronological view of resource allocation and usage patterns including executor additions/removals |\n\n### ⚙️ Configuration \u0026 Environment\n*Spark configuration, environment variables, and runtime settings*\n| 🔧 Tool | 📝 Description |\n|---------|----------------|\n| `get_environment` | ⚙️ Get comprehensive Spark runtime configuration including JVM info, Spark properties, system properties, and classpath |\n\n### 🔎 SQL \u0026 Query Analysis\n*SQL performance analysis and execution plan comparison*\n| 🔧 Tool | 📝 Description |\n|---------|----------------|\n| `list_slowest_sql_queries` | 🐌 Get the top N slowest SQL queries for an application with detailed execution metrics |\n| `compare_sql_execution_plans` | 🔍 Compare SQL execution plans between two Spark jobs, analyzing logical/physical plans and execution metrics |\n\n### 🚨 Performance \u0026 Bottleneck Analysis\n*Intelligent bottleneck identification and performance recommendations*\n| 🔧 Tool | 📝 Description |\n|---------|----------------|\n| `get_job_bottlenecks` | 🚨 Identify performance bottlenecks by analyzing stages, tasks, and executors with actionable recommendations |\n\n### 🔄 Comparative Analysis\n*Cross-application comparison for regression detection and optimization*\n| 🔧 Tool | 📝 Description |\n|---------|----------------|\n| `compare_job_environments` | ⚙️ Compare Spark environment configurations between two jobs to identify differences in properties and settings |\n| `compare_job_performance` | 📈 Compare performance metrics between two Spark jobs including execution times, resource usage, and task distribution |\n\n### 🤖 How LLMs Use These Tools\n\n**Query Pattern Examples:**\n- *\"Show me all applications between 12 AM and 1 AM on 2025-06-27\"* → `list_applications`\n- *\"Why is my job slow?\"* → `get_job_bottlenecks` + `list_slowest_stages` + `get_executor_summary`\n- *\"Compare today vs yesterday\"* → `compare_job_performance` + `compare_job_environments`\n- *\"What's wrong with stage 5?\"* → `get_stage` + `get_stage_task_summary`\n- *\"Show me resource usage over time\"* → `get_resource_usage_timeline` + `get_executor_summary`\n- *\"Find my slowest SQL queries\"* → `list_slowest_sql_queries` + `compare_sql_execution_plans`\n\n## 📔 AWS Integration Guides\n\nIf you are an existing AWS user looking to analyze your Spark Applications, we provide detailed setup guides for:\n\n- **[AWS Glue Users](examples/aws/glue/README.md)** - Connect to Glue Spark History Server\n- **[Amazon EMR Users](examples/aws/emr/README.md)** - Use EMR Persistent UI for Spark analysis\n\nThese guides provide step-by-step instructions for setting up the Spark History Server MCP with your AWS services.\n\n## 🚀 Kubernetes Deployment\n\nDeploy using Kubernetes with Helm:\n\n\u003e ⚠️ **Work in Progress**: We are still testing and will soon publish the container image and Helm registry to GitHub for easy deployment.\n\n```bash\n# 📦 Deploy with Helm\nhelm install spark-history-mcp ./deploy/kubernetes/helm/spark-history-mcp/\n\n# 🎯 Production configuration\nhelm install spark-history-mcp ./deploy/kubernetes/helm/spark-history-mcp/ \\\n  --set replicaCount=3 \\\n  --set autoscaling.enabled=true \\\n  --set monitoring.enabled=true\n```\n\n📚 See [`deploy/kubernetes/helm/`](deploy/kubernetes/helm/) for complete deployment manifests and configuration options.\n\n\u003e **Note**: When using Secret Store CSI Driver authentication, you must create a `SecretProviderClass` externally before deploying the chart.\n\n## 🌐 Multi-Spark History Server Setup\nSetup multiple Spark history servers in the config.yaml and choose which server you want the LLM to interact with for each query.\n\n```yaml\nservers:\n  production:\n    default: true\n    url: \"http://prod-spark-history:18080\"\n    auth:\n      username: \"user\"\n      password: \"pass\"\n  staging:\n    url: \"http://staging-spark-history:18080\"\n```\n\n💁 User Query: \"Can you get application \u003capp_id\u003e using production server?\"\n\n🤖 AI Tool Request:\n```json\n{\n  \"app_id\": \"\u003capp_id\u003e\",\n  \"server\": \"production\"\n}\n```\n🤖 AI Tool Response:\n```json\n{\n  \"id\": \"\u003capp_id\u003e\u003e\",\n  \"name\": \"app_name\",\n  \"coresGranted\": null,\n  \"maxCores\": null,\n  \"coresPerExecutor\": null,\n  \"memoryPerExecutorMB\": null,\n  \"attempts\": [\n    {\n      \"attemptId\": null,\n      \"startTime\": \"2023-09-06T04:44:37.006000Z\",\n      \"endTime\": \"2023-09-06T04:45:40.431000Z\",\n      \"lastUpdated\": \"2023-09-06T04:45:42Z\",\n      \"duration\": 63425,\n      \"sparkUser\": \"spark\",\n      \"appSparkVersion\": \"3.3.0\",\n      \"completed\": true\n    }\n  ]\n}\n```\n\n### 🔐 Environment Variables\n```\nSHS_MCP_PORT - Port for MCP server (default: 18888)\nSHS_MCP_DEBUG - Enable debug mode (default: false)\nSHS_MCP_ADDRESS - Address for MCP server (default: localhost)\nSHS_MCP_TRANSPORT - MCP transport mode (default: streamable-http)\nSHS_SERVERS_*_URL - URL for a specific server\nSHS_SERVERS_*_AUTH_USERNAME - Username for a specific server\nSHS_SERVERS_*_AUTH_PASSWORD - Password for a specific server\nSHS_SERVERS_*_AUTH_TOKEN - Token for a specific server\nSHS_SERVERS_*_VERIFY_SSL - Whether to verify SSL for a specific server (true/false)\nSHS_SERVERS_*_TIMEOUT - HTTP request timeout in seconds for a specific server (default: 30)\nSHS_SERVERS_*_EMR_CLUSTER_ARN - EMR cluster ARN for a specific server\n```\n\n## 🤖 AI Agent Integration\n\n### Quick Start Options\n\n| Integration | Transport | Best For |\n|-------------|-----------|----------|\n| **[Local Testing](TESTING.md)** | HTTP | Development, testing tools |\n| **[Claude Desktop](examples/integrations/claude-desktop/)** | STDIO | Interactive analysis |\n| **[Amazon Q CLI](examples/integrations/amazon-q-cli/)** | STDIO | Command-line automation |\n| **[Kiro](examples/integrations/kiro/)** | HTTP | IDE integration, code-centric analysis |\n| **[LangGraph](examples/integrations/langgraph/)** | HTTP | Multi-agent workflows |\n| **[Strands Agents](examples/integrations/strands-agents/)** | HTTP | Multi-agent workflows |\n\n## 🎯 Example Use Cases\n\n### 🔍 Performance Investigation\n```\n🤖 AI Query: \"Why is my ETL job running slower than usual?\"\n\n📊 MCP Actions:\n✅ Analyze application metrics\n✅ Compare with historical performance\n✅ Identify bottleneck stages\n✅ Generate optimization recommendations\n```\n\n### 🚨 Failure Analysis\n```\n🤖 AI Query: \"What caused job 42 to fail?\"\n\n🔍 MCP Actions:\n✅ Examine failed tasks and error messages\n✅ Review executor logs and resource usage\n✅ Identify root cause and suggest fixes\n```\n\n### 📈 Comparative Analysis\n```\n🤖 AI Query: \"Compare today's batch job with yesterday's run\"\n\n📊 MCP Actions:\n✅ Compare execution times and resource usage\n✅ Identify performance deltas\n✅ Highlight configuration differences\n```\n\n## 🤝 Contributing\n\nCheck [CONTRIBUTING.md](CONTRIBUTING.md) for full guidelines on contributions\n\n## 📄 License\n\nApache License 2.0 - see [LICENSE](LICENSE) file for details.\n\n\n## 📝 Trademark Notice\n\n*This project is built for use with Apache Spark™ History Server. Not affiliated with or endorsed by the Apache Software Foundation.*\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n**🔥 Connect your Spark infrastructure to AI agents**\n\n[🚀 Get Started](#-quick-start) | [🛠️ View Tools](#%EF%B8%8F-available-tools) | [🧪 Test Now](TESTING.md) | [🤝 Contribute](#-contributing)\n\n*Built by the community, for the community* 💙\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkubeflow%2Fmcp-apache-spark-history-server","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkubeflow%2Fmcp-apache-spark-history-server","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkubeflow%2Fmcp-apache-spark-history-server/lists"}