{"id":18141566,"url":"https://github.com/skjolber/json-log-filter","last_synced_at":"2025-07-14T21:34:55.966Z","repository":{"id":37024190,"uuid":"255716305","full_name":"skjolber/json-log-filter","owner":"skjolber","description":"World's fastest JSON filter for the JVM","archived":false,"fork":false,"pushed_at":"2025-04-14T23:13:26.000Z","size":4719,"stargazers_count":7,"open_issues_count":14,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-22T13:18:57.768Z","etag":null,"topics":["cost-optimization","gdpr","high-performance","java","jmh","json","json-filter","logging","optimization"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/skjolber.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"support/test/.gitignore","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-14T20:17:43.000Z","updated_at":"2025-01-27T00:16:56.000Z","dependencies_parsed_at":"2023-10-03T04:04:17.234Z","dependency_job_id":"29372aeb-dff9-412b-b956-533d938272ee","html_url":"https://github.com/skjolber/json-log-filter","commit_stats":null,"previous_names":[],"tags_count":38,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skjolber%2Fjson-log-filter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skjolber%2Fjson-log-filter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skjolber%2Fjson-log-filter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skjolber%2Fjson-log-filter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/skjolber","download_url":"https://codeload.github.com/skjolber/json-log-filter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250246730,"owners_count":21398919,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cost-optimization","gdpr","high-performance","java","jmh","json","json-filter","logging","optimization"],"created_at":"2024-11-01T17:07:37.505Z","updated_at":"2025-04-22T13:19:18.114Z","avatar_url":"https://github.com/skjolber.png","language":"Java","readme":"![Build Status](https://github.com/skjolber/json-log-filter/actions/workflows/maven.yml/badge.svg) \n[![Maven Central](https://img.shields.io/maven-central/v/com.github.skjolber.json-log-filter/parent.svg)](https://mvnrepository.com/artifact/com.github.skjolber.json-log-filter)\n[![codecov](https://codecov.io/gh/skjolber/json-log-filter/graph/badge.svg?token=8mCiHxVFbz)](https://codecov.io/gh/skjolber/json-log-filter)\n\n# json-log-filter\nHigh-performance filtering of to-be-logged JSON. Reads, filters and writes JSON in a single step - drastically increasing throughput ([by ~3x-9x](https://jmh.morethan.io/?source=https://raw.githubusercontent.com/skjolber/json-log-filter/master/benchmark/jmh/results/jmh-results-4.1.2.jdk17.json\u0026topBar=off)). Typical use-cases:\n\n  * Filter sensitive values from logs (i.e. on request-/response-logging)\n     * technical details like passwords and so on\n     * sensitive personal information, for [GDPR](https://en.wikipedia.org/wiki/General_Data_Protection_Regulation) compliance and such\n  * Improve log readability, filtering\n     * large String elements like base64-encoded binary data, or\n     * whole JSON subtrees with low informational value\n  * Reduce amount of data sent to log accumulation tools\n    * lower cost\n    * potentially reduce search / visualization latency\n    * keep within max log-statement size\n       * GCP: [256 KB](https://cloud.google.com/logging/quotas)\n       * Azure: 32 KB\n\nFeatures:\n\n * Truncate large text values\n * Mask (anonymize) scalar values like String, Number, Boolean and so on.\n * Remove (prune) whole subtrees\n * Truncate large documents (max total output size)\n * Skip or speed up filtering for remainder of document after a number of anonymize and/or prune hits \n * Remove whitespace (for pretty-printed documents)\n * Metrics for the above operations + total input and output size\n\nThe library contains multiple filter implementations as to accommodate combinations of the above features with as little overhead as possible. The equivalent filters are also implemented using [Jackson]. \n\nBugs, feature suggestions and help requests can be filed with the [issue-tracker].\n\n## License\n[Apache 2.0]\n\n## Obtain\nThe project is built with [Maven] and is available on the central Maven repository. \n\n\u003cdetails\u003e\n  \u003csummary\u003eMaven coordinates\u003c/summary\u003e\n\nAdd the property\n```xml\n\u003cjson-log-filter.version\u003ex.x.x\u003c/json-log-filter.version\u003e\n```\n\nthen add\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.skjolber.json-log-filter\u003c/groupId\u003e\n    \u003cartifactId\u003eapi\u003c/artifactId\u003e\n    \u003cversion\u003e${json-log-filter.version}\u003c/version\u003e\n\u003c/dependency\u003e\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.skjolber.json-log-filter\u003c/groupId\u003e\n    \u003cartifactId\u003ecore\u003c/artifactId\u003e\n    \u003cversion\u003e${json-log-filter.version}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nand optionally\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.skjolber.json-log-filter\u003c/groupId\u003e\n    \u003cartifactId\u003ejackson\u003c/artifactId\u003e\n    \u003cversion\u003e${json-log-filter.version}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n\u003c/details\u003e\n\nor\n\n\u003cdetails\u003e\n  \u003csummary\u003eGradle coordinates\u003c/summary\u003e\n\nFor\n\n```groovy\next {\n  jsonLogFilterVersion = 'x.x.x'\n}\n```\n\nadd\n\n```groovy\napi(\"com.github.skjolber.json-log-filter:api:${jsonLogFilterVersion}\")\napi(\"com.github.skjolber.json-log-filter:core:${jsonLogFilterVersion}\")\n```\n\nand optionally\n\n```groovy\napi(\"com.github.skjolber.json-log-filter:jackson:${jsonLogFilterVersion}\")\n```\n\u003c/details\u003e\n\n# Usage\nUse a `DefaultJsonLogFilterBuilder` or `JacksonJsonLogFilterBuilder` to configure a filter instance (all filters are thread safe): \n\n```java\nJsonFilter filter = DefaultJsonLogFilterBuilder.createInstance()\n                       .withMaxStringLength(127) // cuts long texts\n                       .withAnonymize(\"$.customer.email\") // inserts ***** for values\n                       .withPrune(\"$.customer.account\") // removes whole subtree\n                       .withMaxPathMatches(16) // halt anon/prune after a number of hits\n                       .withMaxSize(128*1024)\n                       .build();\n                       \nbyte[] json = ...; // obtain JSON\n\nString filtered = filter.process(json); // perform filtering                       \n```\n\n### Max string sizes\nConfigure max string length for output like\n\n```json\n{\n    \"icon\": \"QUJDREVGR0hJSktMTU5PUFFSU1... + 46\"\n}\n```\n\n### Mask (anonymize)\nConfigure anonymize for output like\n\n```json\n{\n    \"password\": \"*****\"\n}\n```\n\nfor scalar values, and/or for objects / arrays all contained scalar values:\n\n```json\n{\n    \"credentials\": {\n        \"username\": \"*****\",\n        \"password\": \"*****\"\n    }\n}\n```\n\n### Remove arrays or objects (prune subtrees) \nConfigure prune to turn input\n\n```json\n{\n    \"context\": {\n        \"boringData\": {\n        ...\n        },\n        \"staticData\": [ ... ]\n    }\n}\n```\n\nto output like\n\n```json\n{\n    \"context\": \"PRUNED\"\n}\n```\n\n### Path syntax\nA simple syntax is supported, where each path segment corresponds to a `field name`. Expressions are case-sensitive. Supported syntax:\n\n    /my/field/name\n\nwith support for wildcards; \n\n    /my/field/*\n\nor a simple any-level field name search \n\n    //myFieldName\n\nThe filters within this library support using multiple expressions at once. Note that path expressions are see through arrays.\n\n### Max path matches\nConfigure max path matches; so that filtering stops after a number of matches. This means the __filter speed can be increased considerably if the number of matches is known to be a fixed number__; and will approach pass-through performance if those matches are in the beginning of the document.\n\nFor example if the to-be filtered JSON document has a schema definition with a header + body structure, and the target value is in the header.   \n\n### Max size\nConfigure max size to limit the size of the resulting document. This reduces the size of the document by (silently) deleting the JSON content after the limit is reached.\n\n### Metrics\nPass in a `JsonFilterMetrics` argument to the `process` method like so:\n\n```\nJsonFilterMetrics myMetrics = new DefaultJsonFilterMetrics();\nString filtered = filter.process(json, myMetrics); // perform filtering\n```\n\nThe resulting metrics could be logged as metadata alongside the JSON payload or passed to sensors like [Micrometer](https://micrometer.io/) for further processing, for example for\n\n * Measuring the impact of the filtering, i.e. reduction in data size\n * Make sure filters are actually operating as intended\n\n## Performance\nThe `core` processors within this project are faster than the `Jackson`-based processors. This is expected as parser/serializer features have been traded for performance:\n\n * `core` is something like 3x-9x as fast as `Jackson` processors, where\n * skipping large parts of JSON documents (prune) decreases the difference, and\n * small documents increase the difference, as `Jackson` is more expensive to initialize.\n * working directly on bytes is faster than working on characters for the `core` processors.\n\nFor a typical, light-weight web service, the overall system performance improvement for using the `core` filters over the `Jackson`-based filters will most likely be a few percent.\n\nMemory use will be at 2-8 times the raw JSON byte size; depending on the invoked `JsonFilter` method (some accept string, other raw bytes or chars).\n\nSee the benchmark results ([JDK 17](https://jmh.morethan.io/?source=https://raw.githubusercontent.com/skjolber/json-log-filter/master/benchmark/jmh/results/jmh-results-4.1.2.jdk17.json\u0026topBar=off)) and the [JMH] module for running detailed benchmarks.\n\nThere is also a [path](impl/path) artifact which helps facilitate per-path filters for request/response-logging applications, which should further improve performance.\n\n# See also\nSee the [xml-log-filter] for corresponding high-performance filtering of XML, and [JsonPath](https://github.com/json-path/JsonPath) for more advanced filtering.\n\nUsing SIMD for parsing JSON: \n * [simdjson](https://github.com/simdjson/simdjson)\n * [sparser](https://blog.acolyer.org/2018/08/20/filter-before-you-parse-faster-analytics-on-raw-data-with-sparser/)\n \nAlternative JSON filters:\n\n * [json-masker](https://github.com/Breus/json-masker) (included in benchmark).\n\n[Apache 2.0]:\t\t\thttps://www.apache.org/licenses/LICENSE-2.0.html\n[issue-tracker]:\t\thttps://github.com/skjolber/json-log-filter/issues\n[Maven]:\t\t\t\thttps://maven.apache.org/\n[JMH]:\t\t\t\t\tbenchmark/jmh\n[xml-log-filter]:      \thttps://github.com/skjolber/xml-log-filter\n[High-performance]:\t\thttps://jmh.morethan.io/?source=https://raw.githubusercontent.com/skjolber/json-log-filter/master/docs/benchmark/jmh-result.json\u0026topBar=off\n[Jackson]:\t\t\t\thttps://github.com/FasterXML/jackson-core\n[JSON]:\t\t\t\t\thttps://www.json.org/json-en.html\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskjolber%2Fjson-log-filter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fskjolber%2Fjson-log-filter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskjolber%2Fjson-log-filter/lists"}