{"id":20194949,"url":"https://github.com/newrelic-experimental/nri-spark","last_synced_at":"2025-04-10T10:13:28.378Z","repository":{"id":48301294,"uuid":"291822287","full_name":"newrelic-experimental/nri-spark","owner":"newrelic-experimental","description":"This New Relic standalone integration polls the Apache Spark REST API for metrics and pushes them into New Relic using Metrics API It uses the New Relic Telemetry sdk for go","archived":false,"fork":false,"pushed_at":"2024-05-07T13:08:56.000Z","size":16939,"stargazers_count":6,"open_issues_count":6,"forks_count":8,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-03-24T09:05:02.329Z","etag":null,"topics":["apache-spark","databricks","databricks-notebooks","metrics","newrelic","nrlabs","nrlabs-data","nrlabs-odp","spark"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/newrelic-experimental.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-31T20:53:04.000Z","updated_at":"2024-04-08T18:15:46.000Z","dependencies_parsed_at":"2022-08-25T09:52:57.401Z","dependency_job_id":null,"html_url":"https://github.com/newrelic-experimental/nri-spark","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/newrelic-experimental%2Fnri-spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/newrelic-experimental%2Fnri-spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/newrelic-experimental%2Fnri-spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/newrelic-experimental%2Fnri-spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/newrelic-experimental","download_url":"https://codeload.github.com/newrelic-experimental/nri-spark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248198889,"owners_count":21063628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","databricks","databricks-notebooks","metrics","newrelic","nrlabs","nrlabs-data","nrlabs-odp","spark"],"created_at":"2024-11-14T04:14:59.482Z","updated_at":"2025-04-10T10:13:28.358Z","avatar_url":"https://github.com/newrelic-experimental.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Experimental Project header](https://github.com/newrelic/opensource-website/raw/master/src/images/categories/Experimental.png)](https://opensource.newrelic.com/oss-category/#experimental)\n\n![GitHub forks](https://img.shields.io/github/forks/newrelic-experimental/nri-spark?style=social)\n![GitHub stars](https://img.shields.io/github/stars/newrelic-experimental/nri-spark?style=social)\n![GitHub watchers](https://img.shields.io/github/watchers/newrelic-experimental/nri-spark?style=social)\n\n![GitHub all releases](https://img.shields.io/github/downloads/newrelic-experimental/nri-spark/total)\n![GitHub release (latest by date)](https://img.shields.io/github/v/release/newrelic-experimental/nri-spark)\n![GitHub last commit](https://img.shields.io/github/last-commit/newrelic-experimental/nri-spark)\n![GitHub Release Date](https://img.shields.io/github/release-date/newrelic-experimental/nri-spark)\n\n\n![GitHub issues](https://img.shields.io/github/issues/newrelic-experimental/nri-spark)\n![GitHub issues closed](https://img.shields.io/github/issues-closed/newrelic-experimental/nri-spark)\n![GitHub pull requests](https://img.shields.io/github/issues-pr/newrelic-experimental/nri-spark)\n![GitHub pull requests closed](https://img.shields.io/github/issues-pr-closed/newrelic-experimental/nri-spark)\n\n\n# New Relic integration for Apache Spark \n\nThis New Relic  standalone integration polls the Apache Spark [REST API](https://spark.apache.org/docs/latest/monitoring.html#rest-api) for metrics and pushes them into New Relic  using Metrics API\nIt uses the New Relic [Telemetry sdk for go](https://github.com/newrelic/newrelic-telemetry-sdk-go)\n\n## Installation\n\n*Requires Apache Spark runnning in standalone mode (YARN and mesos not yet supported)*\n\n1. Download the latest package from Release.\n2. Install the NR Spark Metric plugin plugin using the following command \n    \u003e sudo tar -xvzf /tmp/nri-spark-metric.tar.gz -C /\n\n    The following files will be installed \n    ```\n    /etc/nri-spark-metric/\n    /etc/nri-spark-metric/nr-spark-metric\n    /etc/systemd/system/\n    /etc/systemd/system/nr-spark-metric.service\n    /etc/init/nr-spark-metric.conf\n    ```\n\n\n## Installation\n\nThe integration can be deployed independently on linux 64 system or as a databricks integration using a notebook. The sections below suggests each.\n\n\n###### Standalone deployment \n\n1. Create \"*nr-spark-metric-settings.yml*\" file in the the folder \"*/etc/nri-spark-metric/*\"  using the following format\n    ```\n    sparkmasterurl: \"http://localhost:8080\"  \u003c== FQDN ofspark master URL\n    clustername:  mylocalcluster             \u003c== Name of the cluster\n    insightsapikey: xxxx                     \u003c== Insights api key\n    pollinterval: 5                          \u003c== Polling interval\n    clustermode:                             \u003c== Set mode to *spark_driver_mode* for Single Node clusters\n    tags:                                    \u003c== Additional tags to be added to metrics\n      nr_sample_tag_org: newrelic_labs\n      nr_sample_tag_practice: odp\n    ```\n2. Run the following command.\n    \u003e service nr-spark-metric start\n\n3. Check for metrics in \"Metric\" event type in Insights\n\n\n\n\n###### Databricks Init script creator notebook\n\n\u003e**This notebook and configuration is for reference purpose only, deployment should customize this to fulfill the needs**\n\n\n1. Create a new notebook to deploy the cluster intialization script \n2. Copy the relevant script below. You do not need to set or touch the $DB_ values in the script, Databricks populates these for us. \n   a Optional : Based on cluster install mode, uncommment  SingleNodeCluster install , comment Standalone\n   b Optional : Install infra agent, update with latest version\n4. Replace **\u003cAdd your insights key\u003e\u003e** with your New Relic Insights Insert Key. \n5. Add/Remove/Update tags require in the tag section, sample tags are configured using *nr_sample_tag\\** \n6. Run this notebook to create to deploy the new_relic_install.sh script in dbfs in configured folder.\n7. Ensure the script is attached to your cluster and is listed in the notebooks of the cluster\n8. Running this script will create the file at dbfs:/nr/nri-spark-metric.sh\n9. Configure target cluster with the ***newrelic_install.sh*** cluster-scoped init script using the UI, Databricks CLI, or by invoking the Clusters API. This setting is found in Cluster configuration tab -\u003e Advanced Options -\u003e Init Scripts\n10. Add dbfs:/nr/nri-spark-metric.sh and click add. \n11. Restart your cluster\n12. Metrics should start reporting under the Metrics section in New Relic with the prefix of spark.X.X - you should get Job, Stage Executors and Stream metrics.\n\n```\ndbutils.fs.put(\"dbfs:/nr/nri-spark-metric.sh\",\"\"\" \n#!/bin/sh\necho \"\u003e\u003e\u003e Check if this is driver? $DB_IS_DRIVER\"\necho \"\u003e\u003e\u003e Spark Driver ip: $DB_DRIVER_IP\"\n\n#Create Cluster init script\ncat \u003c\u003cEOF \u003e\u003e /tmp/start_spark-metric.sh\n\n#!/bin/sh\n\nif [ \\$DB_IS_DRIVER ]; then\n  # Root user detection\n  if [ \\$(echo \"$UID\") = \"0\" ];                                      \n  then                                                                     \n    sudo=''                                                                \n  else\n    sudo='sudo'                                                        \n  fi\n  \n  echo \"\u003e\u003e\u003e Check if this is driver? $DB_IS_DRIVER\"\n  echo \"\u003e\u003e\u003e Spark Driver ip: $DB_DRIVER_IP\"\n    \n# Optional install infra agent\n  # Add license key \n  echo \"license_key: \u003c\u003cNR LICENCE KEY \u003e\u003e\" | \\$sudo tee -a /etc/newrelic-infra.yml         \n \n  #Determine OS version. Assuming this is Ubuntu\n  OS_VERSION=\\$(grep VERSION_ID /etc/os-release | cut -d = -f 2 | xargs echo | cut -d \".\" -f 1)\n  echo \"\u003e\u003e\u003e OS_VERSION: \\$OS_VERSION\"\n  \n  #add Newrelic GPG key \n  \\$sudo curl -s https://download.newrelic.com/infrastructure_agent/gpg/newrelic-infra.gpg | sudo apt-key add -\n \n  #Add the infrastructure monitoring agent repository, midify this if OS version changes\n  if [ \\$OS_VERSION = \"18\" ];\n  then \n    echo \"\u003e\u003e\u003e Bionic release\"\n    \\$sudo printf \"deb https://download.newrelic.com/infrastructure_agent/linux/apt bionic main\" | sudo tee -a /etc/apt/sources.list.d/newrelic-infra.list\n  else\n    echo \"\u003e\u003e\u003e Other release, customize script\"\n  fi\n \n  #Refresh repos \n  \\$sudo apt-get update\n\n #install newreli-infra\n \\$sudo apt-get install newrelic-infra -y\n\n ## adding logs configuration \n    echo \"logs:\n  - name: databricks.\\$DB_CLUSTER_NAME\n    file: /databricks/driver/logs/*.log\n    attributes:\n      nrlabs: data\n      entity: databricks\n      clustername: \\$DB_CLUSTER_NAME\n      IP: $DB_DRIVER_IP\" \u003e /etc/newrelic-infra/logging.d/spark.yml\n\n# end of infra agent install   \n\n# Install nr-spark-metric integration\n   #Download nr-spark-metric integration\n  \\$sudo wget https://github.com/hsinghkalsi/nri-spark/releases/download/1.2.0/nri-spark-metric.tar.gz -P /tmp\n\n  #Extract the contents to right place\n  \\$sudo tar -xvzf /tmp/nri-spark-metric.tar.gz -C /\n  \n  # Check which mode is the cluster running in  \n  # Start of  SingleNodeCluster install , using \"spark_driver_mode\"', uncomment this section and comment out Standalone cluster\n  #  echo '  \u003e SingleNodeCluster, using \"spark_driver_mode\"'\n  #  DB_DRIVER_PORT=\\$(grep -i \"CONF_UI_PORT\" /tmp/driver-env.sh | cut -d'=' -f2)\n  #  SPARK_CLUSTER_MODE='spark_driver_mode'\n  # end of SingleNodeCluster install\n    \n  # Start of Standalone Cluster, use the below section \n  # Identify driver port in standalone mode \n    echo '  \u003e Standalone cluster, using \"spark_standalone_mode\", waiting for master-params...'\n    while [ -z \\$is_available ]; do\n      if [ -e \"/tmp/master-params\" ]; then\n        DB_DRIVER_PORT=\\$(cat /tmp/master-params | cut -d' ' -f2)\n        SPARK_CLUSTER_MODE=''\n        is_available=TRUE\n      fi\n      sleep 2\n    done\n  # end of Standalone Cluster section\n  \n  # Configure nr-spark-metric-settings.yml file \n\n  echo \"sparkmasterurl: http://\\$DB_DRIVER_IP:\\$DB_DRIVER_PORT\nclustername: \\$DB_CLUSTER_NAME\ninsightsapikey: NRII-XXXXXXXXXXXXXXXX\t\npollinterval: 5\nclustermode: \\$SPARK_CLUSTER_MODE\ntags:\n  nr_sample_tag_org: newrelic_labs\n  nr_sample_tag_practice: odp\" \u003e /etc/nri-spark-metric/nr-spark-metric-settings.yml\n\n  echo ' \u003e\u003e\u003e Configured  nr-spark-metric-settings.yml \\n $(\u003c/etc/nri-spark-metric/nr-spark-metric-settings.yml)'\n\n#Enable the service\n \\$sudo systemctl enable nr-spark-metric.service\n\n  #Start the service \n  \n \\$sudo systemctl start nr-spark-metric.service\n \\$sudo start nr-spark-metric\n\nfi\nEOF\n\n# Start \nif [ \\$DB_IS_DRIVER ]; then\n  chmod a+x /tmp/start_spark-metric.sh\n  /tmp/start_spark-metric.sh \u003e\u003e /tmp/start_spark-metric.log 2\u003e\u00261 \u0026 disown\nfi\n\n\"\"\",True)\n```\n\n## Support\n\nNew Relic has open-sourced this project. This project is provided AS-IS WITHOUT WARRANTY OR DEDICATED SUPPORT. Issues and contributions should be reported to the project here on GitHub.\n\u003eWe encourage you to bring your experiences and questions to the [Explorers Hub](https://discuss.newrelic.com) where our community members collaborate on solutions and new ideas.\n\n\n## Contributing\nWe encourage your contributions to improve [project name]! Keep in mind when you submit your pull request, you'll need to sign the CLA via the click-through using CLA-Assistant. You only have to sign the CLA one time per project.\nIf you have any questions, or to execute our corporate CLA, required if your contribution is on behalf of a company,  please drop us an email at opensource@newrelic.com.\n\n## License\nNew Relic Infrastructure Integration for Apache Spark is licensed under the [Apache 2.0](http://apache.org/licenses/LICENSE-2.0.txt) License.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnewrelic-experimental%2Fnri-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnewrelic-experimental%2Fnri-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnewrelic-experimental%2Fnri-spark/lists"}