{"id":34250136,"url":"https://github.com/arun0009/jsontrim","last_synced_at":"2026-04-19T22:07:15.139Z","repository":{"id":326361218,"uuid":"1105205704","full_name":"arun0009/jsontrim","owner":"arun0009","description":"A high-performance Go library for trimming oversized JSON payloads. Enforces total size limits, preserves array order, and sanitizes PII with wildcard blacklisting.","archived":false,"fork":false,"pushed_at":"2025-12-12T06:50:59.000Z","size":24,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-19T18:57:32.285Z","etag":null,"topics":["api","go","golang","json","logging","middleware","serialization","size-limit","timmer","wildcard"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arun0009.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-27T09:30:23.000Z","updated_at":"2025-12-12T06:48:47.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/arun0009/jsontrim","commit_stats":null,"previous_names":["arun0009/jsontrim"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/arun0009/jsontrim","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arun0009%2Fjsontrim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arun0009%2Fjsontrim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arun0009%2Fjsontrim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arun0009%2Fjsontrim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arun0009","download_url":"https://codeload.github.com/arun0009/jsontrim/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arun0009%2Fjsontrim/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32024321,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T20:23:30.271Z","status":"online","status_checked_at":"2026-04-19T02:00:07.110Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","go","golang","json","logging","middleware","serialization","size-limit","timmer","wildcard"],"created_at":"2025-12-16T09:12:43.302Z","updated_at":"2026-04-19T22:07:15.121Z","avatar_url":"https://github.com/arun0009.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# jsontrim\n\n[![CI](https://img.shields.io/github/actions/workflow/status/arun0009/jsontrim/ci.yml?branch=main\u0026logo=github)](https://github.com/arun0009/jsontrim/actions/workflows/ci.yml)\n[![Go Report Card](https://goreportcard.com/badge/github.com/arun0009/jsontrim)](https://goreportcard.com/report/github.com/arun0009/jsontrim)\n[![PkgGoDev](https://pkg.go.dev/badge/github.com/arun0009/jsontrim)](https://pkg.go.dev/github.com/arun0009/jsontrim)\n\nA lightweight, high-performance Go library for trimming JSON payloads to enforce size limits while preserving validity and data structure. Perfect for API responses, structured logging, and event recording where oversized data can cause outages.\n\nKey features:\n- **Smart Limits**: Drop or truncate fields \u003e N bytes recursively.\n- **Total Size Cap**: Iteratively remove elements until under the limit using size estimation to reduce GC pressure.\n- **Wildcard Blacklisting**: Exclude sensitive paths dynamically (e.g., `users.*.password`).\n- **Ghost Markers**: Optionally replace dropped fields with `\"[TRIMMED]\"` instead of deleting them, preserving schema visibility.\n- **Order Preservation**: Safely trims arrays without destroying element order.\n- **Strategies**: Choose removal order (largest-first, FIFO, prioritize keys).\n- **Hooks**: Custom pre/post processing.\n- Zero dependencies beyond `encoding/json`.\n\n## Installation\n\n```bash\ngo get github.com/arun0009/jsontrim\n```\n### Quick Start\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n    \"strings\"\n\n\t\"github.com/arun0009/jsontrim\"\n)\n\nfunc main() {\n    // A large JSON payload with sensitive data\n\traw := []byte(`{\n        \"id\": \"123\", \n        \"data\": \"` + strings.Repeat(\"x\", 2000) + `\", \n        \"users\": [{\"id\":1, \"pass\":\"secret\"}, {\"id\":2, \"pass\":\"secret\"}]\n    }`)\n\n\ttrimmer := jsontrim.New(jsontrim.Config{\n\t\tFieldLimit:        500,\n\t\tTotalLimit:        1024,\n\t\tBlacklist:         []string{\"users.*.pass\"}, // Wildcard support\n\t\tReplaceWithMarker: true,                     // Leave a trace of what was removed\n\t\tStrategy:          jsontrim.RemoveLargest{},\n\t})\n\n\tout, err := trimmer.Trim(raw)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\tfmt.Printf(\"Trimmed JSON (%d bytes): %s\\n\", len(out), out)\n\t// Output: {\"id\":\"123\",\"data\":\"[TRIMMED]\",\"users\":[{\"id\":1},{\"id\":2}]}\n}\n```\n\n## Configuration\n\nPass a `Config` to `New()`:\n\n- **FieldLimit** (`int`, default: 500): Max bytes per field/object/array (after nested trim).\n- **TotalLimit** (`int`, default: 1024): Max total output bytes.\n- **Blacklist** (`[]string`, default: `[]`): Dot-notation paths to exclude. Supports * wildcards.\n- **ReplaceWithMarker** (bool, default: false): If true, removed fields/items are replaced with \"[TRIMMED]\" string value instead of being deleted. Useful for debugging.\n- **Strategy** (`TruncStrategy`, default: `RemoveLargest`): Removal policy: `RemoveLargest{}`, `FIFO{}`, or `PrioritizeKeys`.\n- **MaxDepth** (`int`, default: 10): Recursion depth to prevent stack overflows.\n- **TruncateStrings** (`bool`, default: `false`): Append \"...\" to oversized strings instead of dropping.\n- **Hooks** (`Hooks`, default: `{}`): `PreTrim`/`PostTrim` funcs for custom logic.\n\n## Strategies\n\n* `RemoveLargest{}`: Greedily drops the biggest fields/items to maximize retention (default).\n* `FIFO{}`: Removes in iteration order (faster for ordered data).\n* `PrioritizeKeys{KeepKeys: []string{\"id\", \"ts\"}, Fallback: \u0026FIFO{}}`: Delays removal of key fields.\n\n## Blacklisting \u0026 Wildcards\n\nUses dot-notation. Supports * as a wildcard for array indices or dynamic map keys.\n\n* \"user.password\": Matches exact path.\n* \"users.*.password\": Matches password inside any element in `users` (e.g., array index `users[0].password` or map key `users.primary.password`).\n* \"logs.*\": Matches everything inside logs.\n\n## Hooks Example\n\n```go\nimport (\n\t\"log\"\n\t\"time\"  \n)\n\nHooks{\n    PreTrim: func(v interface{}) interface{} {\n        if m, ok := v.(map[string]interface{}); ok {\n            m[\"trimmed_at\"] = time.Now().Format(time.RFC3339)\n        }\n        return v\n    },\n    PostTrim: func(v interface{}, err error) interface{} {\n        if err != nil {\n            log.Printf(\"Trim error: %v\", err)\n        }\n        return v\n    },\n}\n```\n\n## Use Cases\n* **Structured Logging**: Prevent large fields (like massive stack traces, base64 images, or entire HTTP bodies) from **crashing log aggregators** (ELK, Splunk) or consuming excessive bandwidth. jsontrim acts as a safety valve in log hooks.\n* **API Middleware**: Ensure **API responses** strictly adhere to size contracts, preventing issues in client-side applications or with platform limits (e.g., Lambda/API Gateway payload size caps).\n* **Audit and Compiance**: Use the **Blacklist** (especially with wildcards) to sanitize Personally Identifiable Information (**PII**) or sensitive credentials before persisting the data to an audit log or database.\n* **Event Stream Processing**: Guarantee that messages pushed to queues (Kafka, RabbitMQ, SQS) never exceed topic size limits, avoiding message rejection and processing failures.\n\n## License\n\nMIT License. See LICENSE.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farun0009%2Fjsontrim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farun0009%2Fjsontrim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farun0009%2Fjsontrim/lists"}