{"id":18424106,"url":"https://github.com/willfleury/prometheus-metrics-agent","last_synced_at":"2025-10-16T15:02:38.012Z","repository":{"id":91533543,"uuid":"105042764","full_name":"willfleury/prometheus-metrics-agent","owner":"willfleury","description":"JVM agent metrics instrumentation and monitoring with Prometheus (Scala, Java, Kotlin, Clojure, etc)","archived":false,"fork":false,"pushed_at":"2017-10-04T15:08:38.000Z","size":144,"stargazers_count":18,"open_issues_count":0,"forks_count":8,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-31T22:07:38.040Z","etag":null,"topics":["agent","apm","java","jvm","metrics","monitoring","prometheus"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/willfleury.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-27T16:34:21.000Z","updated_at":"2022-06-02T09:22:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"c5e31d72-e1c5-423b-92b5-1d9a043dd64b","html_url":"https://github.com/willfleury/prometheus-metrics-agent","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/willfleury/prometheus-metrics-agent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willfleury%2Fprometheus-metrics-agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willfleury%2Fprometheus-metrics-agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willfleury%2Fprometheus-metrics-agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willfleury%2Fprometheus-metrics-agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/willfleury","download_url":"https://codeload.github.com/willfleury/prometheus-metrics-agent/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willfleury%2Fprometheus-metrics-agent/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279207896,"owners_count":26126462,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-16T02:00:06.019Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","apm","java","jvm","metrics","monitoring","prometheus"],"created_at":"2024-11-06T04:39:47.667Z","updated_at":"2025-10-16T15:02:37.998Z","avatar_url":"https://github.com/willfleury.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n- [Motivation](#motivation)\n    - [Code Bloat Problem](#code-bloat-problem)\n  - [Instrumentation Metadata](#instrumentation-metadata)\n    - [Annotations](#annotations)\n    - [Configuration](#configuration)\n      - [Class Imports](#class-imports)\n    - [Metric Labels](#metric-labels)\n      - [Dynamic Label Values](#dynamic-label-values)\n    - [What we actually Transform](#what-we-actually-transform)\n    - [Supported Languages](#supported-languages)\n  - [Agent Configuration](#agent-configuration)\n    - [Prometheus Configuration](#prometheus-configuration)\n      - [JVM Metrics](#jvm-metrics)\n      - [JMX Metrics](#jmx-metrics)\n    - [Agent Reporting](#agent-reporting)\n    - [Black \u0026 White Lists](#black-and-white-lists)\n    - [Logger Configuration](#logger-configuration)\n  - [Performance](#performance)\n  - [Dependencies](#dependencies)\n- [Binaries \u0026 Releases](#binaries-releases)\n- [Building](#building)\n- [Usage](#usage)\n- [Debugging](#debugging)\n- [Examples](#examples)\n\n\nForked from the following agent project [https://github.com/willfleury/metrics-agent](https://github.com/willfleury/metrics-agent) and customised specifically for Prometheus and performance.\n\n## Motivation\nAgent based bytecode instrumentation is a far more elegant, faster and safer approach to instrumenting code on the JVM. Programmatic addition of metrics into client code leads to severe code bloat and lack of clarity of the underlying business logic. \n\nAnnotation driven instrumentation using dependency injection frameworks such as Spring or Guice are an attempt to reduce the coat bloat caused by manual instrumentation. However, a clear advantage of agent based bytecode instrumentation is quite simple, you don't need to be using Spring or Guice to benefit from it. Another issue with such annotation driven DI frameworks is that they can only inject the logic on code you own and 3rd party libraries cannot be instrumented. The agent does not care if the code you want to instrument is yours, a third party library or the JDK itself.\n\nThe ability to quickly update a configuration file indicating the metrics and code locations we want to measure, and simply restart the application to begin gathering new measurements is invaluable. It saves a considerable amount of developer time and results in faster performance debugging sessions.\n\nThis metrics agent performs bytecode instrumentation as if you had written the metrics manually, thereby minimising any impact to performance or stack trace readability while allowing one to keep the code clean from metric pollution.\n\n\n### Code Bloat Problem\n\t\nLets illustrate the code bloat problem. Say we want to instrument a method which calls some third party library or service, and tracks the number of failures (as exceptions thrown). To do this we need to track both the total number of method invocations and the number of failed invocations. Most of the time in modern Java libraries, exceptions are unchecked which allows them to propagate up to an appropriate handler without polluting the code base. \n\nThe following is an example of a basic block of code which performs a basic service call prior to instrumentation\n\n```java\npublic Result performSomeTask() {\n    return callSomeServiceMethodWhichCanThrowException(createArgs());\n}\n```\n\nTo instrument this programmatically we perform the following\n\n```java\n// add class fields\n\nstatic final Counter total = Metrics.createCounter(\"requests_total\");\nstatic final Counter failed = Metrics.createCounter(\"requests_failed\");\n\npublic Result performSomeTask() {\n    total.inc();\n\n    Result result = null;\n    try {\n        //perform actual original call\n        result = callSomeServiceMethodWhichCanThrowException(createArgs());\n    } catch (Exception e) {\n        failed.inc();\n        throw e;\n    }\n\n    return result;\n}\n```\n\nNow lets add a timer to this also so we can see how long the method call takes.\n\n```java\n// add class fields\n\nstatic final Counter total = Metrics.createCounter(\"requests_total\");\nstatic final Counter failed = Metrics.createCounter(\"requests_failed\");\nstatic final Timer timer = Metrics.createTimer(\"requests_timer\");\n\npublic Result performSomeTask() {\n    long startTime = System.nanoTime();\n    total.inc();\n\n    Result result = null;\n    try {\n        result = callSomeServiceMethodWhichCanThrowException(createArgs());\n    } catch (Exception e) {\n        failed.inc();\n        throw e;\n    } finally {\n        timer.record(System.nanoTime() - startTime);\n    }\n\n    return result;\n}\n```\n\t\t\nWOW! That turned ugly fast! We started with 3 LOC (lines of code) representing the business logic and ended up with 17 LOC, 14 of which were due to our metrics. This has the potential to destroy the clarity of a code base.\n\nWith agent based instrumentation, we can inject the exact same method bytecode as would be produced by writing it manually, but without touching the source.\n\n\n## Instrumentation Metadata \n\nFor those who like marking methods to measure programmatically, we provide annotations to do just that. We also provide a configuration driven system where you define the methods you want to instrument in a yaml format file. We encourage the configuration driven approach over annotations.\n\nHow all the metric types are use should be self explanatory with the exception of Gauges. We use Gauges to track the number of invocations of a particular method or constructor that are `in flight`. That effectively means we increment the gauge value as the method enters and decrements it when it exits. This is very useful for things like Http Request Handlers etc where you want to know the number of in flight requests. \n\n\n### Annotations\n\n```java\n@Counted (name = \"\", labels = { }, doc = \"\")\n@Gauged (name = \"\", mode=in_flight, labels = { }, doc = \"\")\n@Timed (name = \"\", labels = { }, doc = \"\")\n@ExceptionCounted (name = \"\", labels = { }, doc = \"\")\n```\n\nAnnotations are provided for all metric types and can be added to methods including\nconstructors. \n\n```java\n@Counted(name = \"taskx_total\", doc = \"total invocations of task x\")\n@Timed (name = \"taskx_time\", doc = \"duration of task x\")\npublic Result performSomeTask() {\n    //...\n}\n```\n\n### Configuration\n\n\tmetrics:\n\t  {class name}.{method name}{method signature}:\n\t    - type: Counted\n\t\t  name: {name}\n\t\t  doc: {metric documentation}\n\t\t  labels: ['{name:value}', '{name:value}']\n\t    - type: Gauged\n\t\t  name: {name}\n\t\t  mode: {mode}\n\t\t  doc: {metric documentation}\n\t\t  labels: ['{name:value}']\n\t    - type: ExceptionCounted\n\t\t  name: {name}\n\t\t  doc: {metric documentation}\n\t\t  labels: ['{name:value}']\n\t    - type: Timed\n\t\t  name: {name}\n\t\t  doc: {metric documentation}\n\t\t  labels: ['{name:value}']\n\nEach metric is defined on a per method basis. A method is uniquely identified by the \ncombination of `{class name}.{method name}{method signature}`. As an example, if we \nwanted to instrument the following method via configuration instead of annotations\n\n```java\npackage com.fleury.test;\n....\n\npublic class TestClass {\n    ....\n    \n    @Counted(name = \"taskx_total\", doc = \"total invocations of task x\")\n    public Result performSomeTask() {\n        ...\n    }\n}\n```\n\nWe write the configuration as follows\n\n\tmetrics:\n\t  com/fleury/test/TestClass.performSomeTask()V:\n\t    - type: Counted\n\t\t  name: taskx_total\n\t\t  doc: total invocations of task x\n\n\nNote the method signature is based on the method parameter types and return type. The parameter types are between the brackets `()` with the return type after. In this case we have no parameters and the return type is void which results in `()V`. [Here](http://journals.ecs.soton.ac.uk/java/tutorial/native1.1/implementing/method.html) is a good overview of Java method signature mappings.\n\nIn previous versions we allowed the package name to be specified using `.` instead of the internal `/` separator. While this is still supported for the metrics configuration section, it is not supported anywhere else and should be updated to only have the `/` package separator. \n\n\n#### Class Imports\n\nTo simplify the metrics definition section of the configuration, we allow an imports section. Here we can define the fully qualified class names for any classes we use or re-use in the definitions. This includes method type descriptors.\n\n\n    imports:\n      - com/fleury/test/TestClass\n      - java/lang/Object\n      - java/lang/String\n\n    metrics:\n      TestClass.performSomeTask(LString;)V:\n        - type: Counted\n          name: taskx_total\n          doc: total invocations of task x\n          \n       TestClass.performSomeOtherTask(LString;)LObject;:\n         - type: Counted\n           name: tasky_total\n           doc: total invocations of task y\n\n\n\n### Metric Labels\n\nLabels are a concept in some reporting systems that allow for multi-dimensional metric capture and analysis. Labels are composed of name value pairs `({name}:{value})`. You can have up to a maximum of five labels per metric. See the Prometheus metric library guidelines on metric and label naming [here](https://prometheus.io/docs/practices/naming/). \n\n\n#### Dynamic Label Values\n\nA powerful feature is the ability to set label values dynamic based on variables available on the method stack. Metric names cannot be dynamic. The way we specify dynamic label values is using the `${index}` syntax followed by the method argument index. The special value `$this` can be used to access the current instance reference in non static methods. We prevent usage of `$this` in constructors to prevent initialisation leakage.\n\nNote that we restrict the stack usage to the method arguments only. That is, we don't allow use of variables created within the method as that is a very fragile thing to do. The String representation as given by `String.valueOf()` of the parameter is used as the label value. That means for primitive types we perform boxing first and null objects will result in the String `\"null\"`. Argument indexes start at index `0` up to the number of `args.length - 1` (i.e. array index syntax). We manage any special logic that occurs with the actual location on the stack due to non static methods (`this` is index `0` on the stack) and static methods (no `this`). Therefore you can always assume index `0` is the first method argument. \n\n```java\n@Counted (name = \"service_total\", labels = { \"client:$0\" })\npublic void callService(String client) \n```\n\nEach time this method is invoked it will use the value of the `client` parameter as the metric label value. We also support accessing nested property values. For example, `($1.httpMethod)` where `$1` is the first method parameter and is e.g. of type `HttpRequest`. This means you are essentially doing `httpRequest.getHttpMethod().toString();`. This nesting can be arbitrarily deep. We use `PropertyUtils` from the `commons-beanutils` library to perform the nested property reading. Typically this means you can only use JavaBeans conforming properties, however, we have added a `GenericBeanIntrospector` which allows for accessing properties in methods like `name()` via e.g. `$1.name` etc. This gives better cross languages support.\n\n\n### What we actually Transform\nAs we allow the use of annotations to register metrics to track, if no black/white lists are defined we must scan all classes as they are loaded and check for the annotations. However, we do not want to have to rewrite all of these classes if we have not changed anything. There are many reasons you want to modify as little as possible with an agent but the general motto is, only touch what you have to. Hence, we only rewrite classes which have been changed due to the addition of metrics and all other classes, even though scanned, are returned untouched to the classloader.\n\n### Supported Languages\nAs the agent works at the bytecode level, we support any language which runs on the JVM. Every language which compiles and runs on the JVM must obey by the bytecode rules. This simply means we need to understand the translation mechanisms of each language for the language level method name to the bytecode level. In Java this is usually 1:1 (excluding some generics fun). You can always examine the `javap` (the [Java Disassembler](http://docs.oracle.com/javase/7/docs/technotes/tools/windows/javap.html)) command to view the bytecode contents in a more `Java` centric way.\n\nAs an example. Lets take the followin Scala class \n```scala\nclass Person(val name:String) {\n}\n```\nIf we run `javap` on this\n\n    javap Person.class\n    \nwe get\n\n    Compiled from \"Person.scala\"\n    public class Person {\n      private final java.lang.String name;   // field\n      public java.lang.String name();        // getter method\n      public Person(java.lang.String);       // constructor\n    }\n\nYou can do the same for Kotlin, Clojure and any other JVM language. Be aware that there may be some quirks in the naming translations and it may not always be as simple as shown above.\n\n\n## Agent Configuration\n\n### Prometheus Configuration\n\n#### JVM Metrics\nPrometheus supports adding JVM level metrics information obtained from the JVM via MBeans for \n\n- gc\n- memory\n- classloading\n- threads\n\nTo enable each, simply add the metrics you want to a `jvm` property in the `system` section of the configuration yaml. For example, to add `gc` and `memory` information to the registry used:\n\n    system:\n        jvm:\n           - gc\n           - memory\n\n#### JMX Metrics\nWe also include the Prometheus JmxCollector from the [JmxExporter](https://github.com/prometheus/jmx_exporter) project in this agent as it allows collecting other JMX metrics from the JVM one might want to. To enable the JmxCollector simply add a `jmx` section in the system configuration and add configuration as shown in the [JmxExporter](https://github.com/prometheus/jmx_exporter) project. The configuration is passed straight through to the JmxCollector constructor.\n\n    system:\n        jmx:\n            startDelaySeconds: 0\n            whitelistObjectNames: [\"org.apache.cassandra.metrics:*\"]\n            blacklistObjectNames: [\"org.apache.cassandra.metrics:type=ColumnFamily,*\"]\n            rules:\n              - pattern: 'org.apache.cassandra.metrics\u003ctype=(\\w+), name=(\\w+)\u003e\u003c\u003eValue: (\\d+)'\n                name: cassandra_$1_$2\n                value: $3\n                valueFactor: 0.001\n                labels: {}\n                help: \"Cassandra metric $1 $2\"\n                type: GAUGE\n                attrNameSnakeCase: false\n\n\n### Agent Reporting\n\nWe start the default reporting (endpoint) for Prometheus which is the HttpServer. The default port for the Prometheus endpoint is `9899` and it can be changed by specifying the property `httpPort` in the system configuration section as follows\n\n    system:\n        httpPort: 9899\n\nSupport for push based reporting could be easily added and made configurable. \n\n\n### \u003ca name=\"black-and-white-lists\"\u003e\u003c/a\u003eBlack and White Lists\n\nSometimes we only want to scan certain packages or classes which we wish to instrument. This could be to reduce the agent startup time or to work around problematic instrumentation situations. Note that the black and white lists do not take any annotations or metric configuration into account and essentially override them.\n\nTo white list a class or package include the fully qualified class or package name under the `whiteList` property. If no white list is specified, then all classes are scanned and eligible for transforming. \n\n    whiteList:\n      - com/fleury/test/ClassName\n      - com/fleury/package2\n        \nTo black list a class or package add the fully qualified class or package name under the `blackList` property. If a class or package is in both white and black list, the black list wins and the class will not be touched.\n\n    blackList:\n       - com/\n               \n### Logger Configuration        \n\nj.u.l is used for logging and can be configured by passing the agent argument `log-config:\u003cproperties path\u003e` to the agent with the path to the logger properties file. \n\n\n## Performance\nWe use the Java ASM bytecode manipulation library. This is the lowest level bytecode manipulation library and is the basis of most other higher level libraries such as cglib. It allows us to inject bytecode in a precise way which means we can craft the exact same bytecode as if it was hand written. We create static level fields to hold the metric references which means there is no lookup required when performing an operation on the metric. This is again how you would write it manually if taking care for speed. \n\nIt should be noted that as with hand crafted metrics, the additional bytecode and hence method size required to handle capturing all metrics could potentially lead to methods which might otherwise have been inlined or compiled by the JIT being skipped instead. This should be considered regardless off the instrumentation choice and if unsure, the appropriate JVM output should be checked (-XX:+UnlockDiagnosticVMOptions -XX:+PrintInlining -XX:+PrintCompilation).\n \n\n## Dependencies \nVery lightweight.\n\t\n\tasm\n\tjackson\n\tprometheus\n\nNote that the final agent binaries are shaded and all dependencies relocated to prevent possible conflicts.\n\n\n# \u003ca name=\"binaries-releases\"\u003e\u003c/a\u003eBinaries \u0026 Releases\n\nSee the releases section of the github repository for releases along with the prebuilt agent binaries.\n\n# Building\n\n\tmvn clean package\n\t\nThe uber jar can be found under `/target/metrics-agent.jar`\n\n# Usage\n\nThe agent must be attached to the JVM at startup. It cannot be attached to a running JVM.\n\n\t-javaagent:metrics-agent.jar\n\nExample\n\t\n\tjava -javaagent:metrics-agent.jar -jar myapp.jar \n\nUsing the configuration file config.yaml is performed as follows\n\n\tjava -javaagent:metrics-agent.jar=agent-config:agent.yaml -jar myapp.jar \n\n\nUsing the configuration file config.yaml and logging configuration logger.properties is performed as follows\n\n\tjava -javaagent:metrics-agent.jar=agent-config:agent.yaml,log-config:logger.properties -jar myapp.jar \n\n\n# Debugging\n\nNote if you want to debug the metrics agent you should put the debugger agent first.\n\n\t-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=\u003cport\u003e -javaagent:metrics-agent.jar myapp.jar\n\n\n# Examples \n\nWe provide some example configurations for popular frameworks. They serve as examples for how to instrument others and can be combined as desired (e.g. you can have both dropwizard request metrics and hibernate in the same configuration). As you will see, it is very simple and light weight to add new frameworks. In the case where different major versions of frameworks required different classes and methods to be instrumented, this simply becomes the addition of a new configuration file for that version. \n\n - [Jersey](example-configurations/jersey.yaml)\n - [Dropwizard (via Jersey)](example-configurations/dropwizard.yaml)\n - [Tomcat Servlet, JSP, Jersey](example-configurations/tomcat.yaml)\n - [Hibernate](example-configurations/hibernate.yaml)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwillfleury%2Fprometheus-metrics-agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwillfleury%2Fprometheus-metrics-agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwillfleury%2Fprometheus-metrics-agent/lists"}