{"id":27996372,"url":"https://github.com/groupon/spark-metrics","last_synced_at":"2025-08-10T04:10:20.951Z","repository":{"id":57724747,"uuid":"56889888","full_name":"groupon/spark-metrics","owner":"groupon","description":"A library to expose more of Apache Spark's metrics system","archived":true,"fork":false,"pushed_at":"2019-11-27T09:41:22.000Z","size":44,"stargazers_count":146,"open_issues_count":16,"forks_count":53,"subscribers_count":31,"default_branch":"master","last_synced_at":"2025-07-11T21:31:47.190Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/groupon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-04-22T23:02:26.000Z","updated_at":"2025-04-28T08:20:30.000Z","dependencies_parsed_at":"2022-09-02T07:02:00.384Z","dependency_job_id":null,"html_url":"https://github.com/groupon/spark-metrics","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/groupon/spark-metrics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groupon%2Fspark-metrics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groupon%2Fspark-metrics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groupon%2Fspark-metrics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groupon%2Fspark-metrics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/groupon","download_url":"https://codeload.github.com/groupon/spark-metrics/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groupon%2Fspark-metrics/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269673954,"owners_count":24457238,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-10T02:00:08.965Z","response_time":71,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-08T21:40:01.303Z","updated_at":"2025-08-10T04:10:20.939Z","avatar_url":"https://github.com/groupon.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"#spark-metrics\n\nA library to expose more of [Apache Spark](http://spark.apache.org/)'s metrics system. This library allows you to use APIs like the [Dropwizard/Codahale Metrics](http://metrics.dropwizard.io/3.1.0/) library on Spark applications to publish metrics that are aggregated across all executors.\n\n## Dependencies\n\n### Spark 2.x\n`spark-metrics` by default will be targeting the Spark 2.x line of releases. To use this library for a Spark 2.x application, add the following dependency:\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.groupon.dse\u003c/groupId\u003e\n    \u003cartifactId\u003espark-metrics\u003c/artifactId\u003e\n    \u003cversion\u003e2.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nNote that `spark-metrics` targets Scala 2.11 by default, as that is the default Scala version supported by Spark 2.x. To support Scala 2.10 on the Spark 2.x releases, this library will need to be recompiled with the Spark dependencies that target Scala 2.10.\n\n### Spark 1.x\nTo use this library with the Spark 1.x line of releases, add a dependency to `spark-metrics_spark-1.x` instead:\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.groupon.dse\u003c/groupId\u003e\n    \u003cartifactId\u003espark-metrics_spark-1.x\u003c/artifactId\u003e\n    \u003cversion\u003e2.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nThe default Spark version targeted for `spark-metrics_spark-1.x` is Spark 1.6.3, but it is compatible with 1.5 and 1.4 as well. Note that `spark-metrics_spark-1.x` targets Scala 2.10 by default, as that is the default Scala version supported by Spark 1.x. To support Scala 2.11 on the Spark 1.x releases, this library will need to be recompiled with the Spark dependencies that target Scala 2.11.\n\nUpdates to `spark-metrics` will be backported to `spark-metrics_spark-1.x` whenever possible, but support for `spark-metrics_spark-1.x` will be discontinued at some point in the future.\n\n## Usage\nInclude this import in your main Spark application file:\n```scala\nimport org.apache.spark.groupon.metrics.UserMetricsSystem\n```\n\nIn the Spark driver, add the following call to `UserMetricsSystem.initialize()` right after the application's `SparkContext` is instantiated:\n```scala\nval sparkContext = new SparkContext()\nUserMetricsSystem.initialize(sparkContext, \"MyMetricNamespace\")\n```\n\n(Technically, it need not necessarily be *right* after the `SparkContext` is created, so long as `initialize` is called before `SparkMetric` instances are created. But invoking it here will help prevent issues related to initialization from occuring, so it is highly recommended.)\n\nAfter this, you can create `SparkMetric` instances that report to Spark's metrics servlet anywhere in your application. **These instances must be declared `lazy` for this library to work properly.**\n```scala\nlazy val counter: SparkCounter = UserMetricsSystem.counter(\"MyCounter\")\n\nlazy val gauge: SparkGauge = UserMetricsSystem.gauge(\"MyGauge\")\n\nlazy val histogram: SparkHistogram = UserMetricsSystem.histogram(\"MyHistogram\")\n\nlazy val meter: SparkMeter = UserMetricsSystem.meter(\"MyMeter\")\n\nlazy val timer: SparkTimer = UserMetricsSystem.timer(\"MyTimer\")\n```\n\nThe `metricName` parameter is the only identifier for metrics, so different metric types cannot have the same name (e.g. a `Counter` and `Histogram` both with the same `metricName`).\n\nThe APIs for these are kept as close as possible to Dropwizard's APIs, but they don't actually extend a common interface at the language level. The only significant difference is in the `SparkGauge` class. Whereas Dropwizard's `Gauge` class is instantiated basically by passing in a function that knows how to obtain the value for the `Gauge`, this version simply has a `set()` method to set its value.\n\n\n## Viewing and Publishing Metrics\nThis library is integrated with Spark's built-in metrics servlet. This means that metrics collected using this library are visible at the `/metrics/json/` endpoint. Here, any of the metrics from this library will be published with the key `\u003cappName\u003e.\u003cmetricNamespace\u003e.\u003cmetricName\u003e`.\n\nThese metric JSONs look something like this:\n```json\napplication_1454970304040_0030.driver.MyAppName.MyMetricNamespace.MyTimer: {\n    count: 4748,\n    max: 21683.55228,\n    mean: 662.7780978119895,\n    min: 434.211779,\n    p50: 622.795788,\n    p75: 672.358402,\n    p95: 1146.214833,\n    p98: 1146.214833,\n    p99: 1146.214833,\n    p999: 1572.286154,\n    stddev: 163.37016417936547,\n    m15_rate: 0.06116903443100036,\n    m1_rate: 0.019056182723172856,\n    m5_rate: 0.051904011476711955,\n    mean_rate: 0.06656686539563786,\n    duration_units: \"milliseconds\",\n    rate_units: \"calls/second\"\n}\n```\n\nOther methods of publishing these metrics are also possible by configuring Spark. Any of the sinks listed [here](http://spark.apache.org/docs/latest/monitoring.html#metrics) will also report the metrics collected by this library as long as the `driver` instance is enabled. See [this page](https://github.com/apache/spark/blob/master/conf/metrics.properties.template) for a sample metrics configuration.\n\n\n## How It Works\nThis library is implemented using a combination of Spark's internal RPC APIs and the Dropwizard APIs. The Dropwizard APIs are used on the driver to aggregate metrics that get collected across different executors and the driver. Spark itself uses the Dropwizard library for some of its own metrics, so this library integrates with Spark's existing metrics system to report user metrics alongside Spark's built-in metrics.\n\nWhen a `SparkMetric` instance is created in an executor or the driver, it sets up a connection to the [`MetricsReceiver`](src/main/scala/org/apache/spark/groupon/metrics/MetricsReceiver.scala) on the driver, which gets set up by the call to `UserMetricsSystem.initialize`. Whenever a metric is collected (e.g. calling `meter.mark()`, `gauge.set(x)`, etc.), that value is sent to the `MetricsReceiver`, which uses those values to update its corresponding stateful Dropwizard `Metric` instance. A `SparkMetric` instance is stateless, in that there are no actual values stored there - its only functionality is to send values to the `MetricsReceiver`. A metric is uniquely identified by its name, so, for example, all values sent by instances of a `SparkHistogram` named `MyHistogram` will get aggregated on the `MetricsReceiver` in a single instance of a Dropwizard `Histogram` that corresponds to `MyHistogram`.\n\nMetrics are sent to the `MetricsReceiver` using a [`MetricMessage`](src/main/scala/org/apache/spark/groupon/metrics/MetricMessage.scala), which contains the actual metric value and metadata about that metric. This metadata contains information that determines how its corresponding Dropwizard metric will be instantiated in the `MetricsReceiver`. For example, a `MetricMessage` for a `SparkHistogram` contains not only the metric value and name, but also the Dropwizard `Reservoir` class used to determine what kind of windowing behavior the histogram will have. Having this metadata allows for metrics to be created dynamically during runtime, rather than having to define them all beforehand. This can, for example, enable the creation of a `Meter` which is named after an `Exception`, where what the `Exception` instance could be doesn't need to be known ahead of time:\n```scala\nUserMetricsSystem.meter(s\"exceptionRate.${exception.getClass.getSimpleName}\")\n```\n\n\n## Troubleshooting\n* A `NotInitializedException` is thrown:\n\n  The most likely reason is that `UserMetricsSystem.initialize` was not called on the driver before a `SparkMetric` instance was created. A `SparkMetric` instance needs to connect to the `MetricsReceiver` when instantiated, so if `initialize` was not invoked, there is no `MetricsReceiver` to connect to. Another likely reason is that the `SparkMetric` instance was not declared `lazy`. This is important because, even if `initialize` was called on the driver, there's no guarantee that the `SparkMetric` instance will be instantiated on a remote JVM after the `MetricsReceiver` is set up. The only way to have this guarantee is to delay instantiating the `SparkMetric` until it is actually used by the application, which means that these need to be `lazy`. This isn't the most user-friendly API, so future work will aim to not require these `lazy` declarations.\n\n* A `SparkContextNotFoundException` is thrown:\n\n  This can happen if `UserMetricsSystem.initialize` is called before a `SparkContext` exists. This error can also happen if a `SparkMetric` instance isn't declared `lazy` and is instantiated as a field on the driver singleton object. A broken example:\n  ```scala\n  object OffsetMigrationTool {\n    val myHistogram = UserMetricsSystem.histogram(\"MyHistogram\")\n\n    def main(args: Array[String]) {\n      val sc = new SparkContext()\n      UserMetricsSystem.initialize(sc)\n      // Rest of driver code...\n    }\n  }\n  ```\n\n  `myHistogram` above needs to instead be declared `lazy`:\n  ```scala\n  lazy val myHistogram = UserMetricsSystem.histogram(\"MyHistogram\")\n  ```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgroupon%2Fspark-metrics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgroupon%2Fspark-metrics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgroupon%2Fspark-metrics/lists"}