{"id":43728978,"url":"https://github.com/wakumaku/jsonshredder","last_synced_at":"2026-02-05T09:50:20.470Z","repository":{"id":36960586,"uuid":"308856387","full_name":"wakumaku/jsonshredder","owner":"wakumaku","description":"Transform JSON objects to other objects with the same values","archived":false,"fork":false,"pushed_at":"2025-09-24T20:55:03.000Z","size":227,"stargazers_count":4,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-09-24T22:33:45.706Z","etag":null,"topics":["forwarder","golang","golang-application","json-parser"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wakumaku.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["wakumaku"]}},"created_at":"2020-10-31T10:26:14.000Z","updated_at":"2025-09-24T20:54:03.000Z","dependencies_parsed_at":"2024-01-18T09:49:12.420Z","dependency_job_id":"a214ecc9-e430-485d-b0e4-0c38eba06fe8","html_url":"https://github.com/wakumaku/jsonshredder","commit_stats":{"total_commits":109,"total_committers":4,"mean_commits":27.25,"dds":"0.21100917431192656","last_synced_commit":"6366bcabdc7bbb8581952f621f858fb061fe3dc4"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/wakumaku/jsonshredder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wakumaku%2Fjsonshredder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wakumaku%2Fjsonshredder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wakumaku%2Fjsonshredder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wakumaku%2Fjsonshredder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wakumaku","download_url":"https://codeload.github.com/wakumaku/jsonshredder/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wakumaku%2Fjsonshredder/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29119073,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-05T09:40:36.738Z","status":"ssl_error","status_checked_at":"2026-02-05T09:36:49.977Z","response_time":65,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["forwarder","golang","golang-application","json-parser"],"created_at":"2026-02-05T09:50:19.922Z","updated_at":"2026-02-05T09:50:20.462Z","avatar_url":"https://github.com/wakumaku.png","language":"Go","funding_links":["https://github.com/sponsors/wakumaku"],"categories":[],"sub_categories":[],"readme":"# JSON Shredder (Oroku Saki)\n\n⚠️ **WIP** 🚨\n\nTransform JSON objects to other objects with the same values.\n\nHave you ever ...\n\n- ... received big payloads where you just need a couple of fields?\n- ... wanted to build new endpoints without waiting for the caller re-implementation?\n- ... received complex nested payloads and you just need a key/value flatten object?\n\n**jsonshredder can help you,**\n**receive the payload you deserve! 💪**\n\n- Define transformations to create **new payloads** from old ones\n- Use it as **transformation** service\n- Use it as **forwarder** service (transform and forward)\n\n## Example case\n\nIf you want to transform this:\n\n```json\n{\n    \"payload\": {\n        \"user\": {\n            \"data\":{\n                \"user_id\": 12345,\n                \"city\":    \"NY\",\n                \"geo\":     {\"lat\": 1, \"long\": 1},\n                \"user\":    \"john.doe\",\n                \"address\": {\n                    \"street\": \"foo avenue\",\n                    \"number\": 42\n                },\n                \"email\":    \"john.doe@bar.tld\"\n            }\n        }\n    }\n}\n```\n\nTo this:\n\n```json\n{\n    \"username\":      \"john.doe\",\n    \"email_address\": \"john.doe@bar.tld\"\n}\n```\n\nYou can use this transformation:\n\n```yaml\n# ...\n    mappings:\n      - path: payload.user.data.user\n        path_out: username\n      - path: payload.user.data.email\n        path_out: email_address\n```\n\nHow?\n\n### Transformation\n\n1. Create a config file named `myconfig.yml`:\n\n```yaml\n---\nport: 8080\n\ntransformations:\n  - name: myfirsttransformation\n    mappings:\n      - path: payload.user.data.user\n        path_out: username\n      - path: payload.user.data.email\n        path_out: email_address\n```\n\n2. Run jsonshredder passing this config:\n\n```shell\n$ docker run --rm -p 8080:8080 -v $(pwd)/myconfig.yml:/config.yml wakumaku/jsonshredder:latest\n...\n```\n\n3. Test it:\n\n```shell\n$ curl -X POST -d '{\"payload\":{\"user\":{\"data\":{\"user_id\":12345,\"city\":\"NY\",\"geo\":{\"lat\":1,\"long\":1},\"user\":\"john.doe\",\"address\":{\"street\":\"foo avenue\",\"number\":42},\"email\":\"john.doe@bar.tld\"}}}}' http://localhost:8080/myfirsttransformation\n\n{\"email_address\":\"john.doe@bar.tld\",\"username\":\"john.doe\"}\n```\n\n### Transformation and forwarding\n\nImagine you want to forward the resulting transformation to an AWS SQS Queue.\n\n1. Add an SQS forwarder configuration in your config file like this:\n\n```yaml\n---\nport: 8080\n\ntransformations:\n  - name: myfirsttransformation\n    mappings:\n      - path: payload.user.data.user\n        path_out: username\n      - path: payload.user.data.email\n        path_out: email_address\n\nforwarders:\n  - name: mysqs\n    kind: sqs\n    params:\n      aws_profile: mydelegatedrole\n      aws_region: us-east-1\n      aws_resource_name: \"queue\"\n```\n\n2. Run the jsonshredder:\n\n```shell\n$ docker run --rm -p 8080:8080 \\\n  -v ~/.aws:/.aws:ro \\\n  -e AWS_SHARED_CREDENTIALS_FILE=/.aws/credentials \\\n  -v $(pwd)/config.dev.yaml:/config.yml \\\n  wakumaku/jsonshredder:latest\n...\n```\n\nNOTE: here we are sharing our AWS credentials because we are using a profile.\n\nIf you prefer, you can specify the `aws_access_key_id` and `aws_secret_access_key` params.\n\n\n3. Test it:\n\n```shell\n$ curl -X POST -d '{\"payload\":{\"user\":{\"data\":{\"user_id\":12345,\"city\":\"NY\",\"geo\":{\"lat\":1,\"long\":1},\"user\":\"john.doe\",\"address\":{\"street\":\"foo avenue\",\"number\":42},\"email\":\"john.doe@bar.tld\"}}}}' http://localhost:8080/myfirsttransformation/mysqs\n\n{\"email_address\":\"john.doe@bar.tld\",\"username\":\"john.doe\"}\n```\n\nTake a look to the URL: `http://localhost:8080/myfirsttransformation/mysqs`\n\nNote that the first segment of the path is the Transformation name and the second one the Forwarder name.\n\nThis way you can combine Transformations and Forwardings as you wish!\n\n4. Check for new items in your queue!\n\n```shell\n$ aws sqs get-message --queue-url http://...\n``` \n\n## Configuration file\n\n### Global\n\n- `port`: HTTP Server port number\n- `loglevel`: debug, info, warn or error\n\n### Transformations\n\nEach transformation MUST have a unique name. Those names will be used in the HTTP Endpoints, please use `[A-Za-z0-9_-]` chars.\n\n```yaml\n  - name: client_payment     # name that will be used in the URL to identify the transformation\n    operation: extract       # add | extract, default: extract. Add appends new fields to the original structure.\n    mappings:\n      - path: user.data.age  # jmespath expression: https://jmespath.org/\n        path_out: age        # (optional) path to create with the value (or object) found in path\n        type_out: string     # (optional) string | int | float, default: original value type. Can force conversions. Adds quotes when is string and try to convert to int/float.\n        default_null: 0      # (optional) default value when the path doesn't exist or the value is null\n```\n\n### Forwarders\n\nThere is a limited of forwarders implemented, currently: http, sns, sqs, kinesis. They can be used to simplify the transformation result delivery.\n\nAWS Family forwarders:\n\n```yaml\n  - name: myforwarder # unique name to identify the forwarder\n    kind: sns         # sns, sqs, kinesis\n    params:\n      aws_endpoint: http://localstack:4566 # allows point to an AWS Mock service\n      aws_access_key_id: foo               # AWS KeyID\n      aws_secret_access_key: bar           # AWS SecretAccessKey\n      aws_profile: profile                 # AWS Profile name\n      aws_resource_arn: aws:::resource/... # When kind is: SNS\n      aws_resource_name: resource_name     # When kind is: SQS, KINESIS\n      aws_region: us-east-1                # AWS Region\n```\n\nOther forwarders:\n\n```yaml\n  - name: myforwarder\n    kind: http\n    params:\n      http_endpoint: http://....com/post # endpoint destination\n      http_header_auth: Bearer mytoken   # (optional) Authorization header value\n      http_status_ok: 200                # expected status code from the destination\n```\n\n## Full Configuration example\n\n```yaml\n### Global\nport: 8080\nloglevel: debug\n\n### Transformations\ntransformations:\n  - name: myfirsttransformation\n    mappings:\n      - path: payload.user.data.user\n        path_out: username\n      - path: payload.user.data.email\n        path_out: email_address\n\n  - name: transform1\n    mappings:\n      - path: user.data.name\n        path_out: username\n      - path: user.data.email\n        path_out: email\n\n  - name: transform2\n    operation: add\n    mappings:\n      - path: username\n        path_out: username2\n      - path: email\n        path_out: email2\n        default_null: \"empty@email.com\"\n\n  - name: transform3\n    operation: add\n    mappings:\n      - path: username2\n        path_out: username3\n      - path: email2\n        path_out: email3\n\n  - name: transform4\n    operation: add\n    mappings:\n      - path: username3\n        path_out: username4\n      - path: email3\n        path_out: email4\n\n### Forwarders\nforwarders:\n\n  - name: sendtosns\n    kind: sns\n    params:\n      aws_endpoint: http://localstack:4566\n      aws_access_key_id: foo\n      aws_secret_access_key: bar\n      aws_region: us-east-1\n      aws_resource_arn: \"arn:aws:sns:us-east-1:000000000000:topic\"\n\n  - name: sendtosqs\n    kind: sqs\n    params:\n      aws_endpoint: http://localstack:4566\n      aws_access_key_id: foo\n      aws_secret_access_key: bar\n      aws_region: us-east-1\n      aws_resource_name: \"queue\"\n\n  - name: sendtosqs2\n    kind: sqs\n    params:\n      aws_endpoint: http://localstack:4566\n      aws_access_key_id: foo\n      aws_secret_access_key: bar\n      aws_region: us-east-1\n      aws_resource_name: \"queue2\"\n\n  - name: sendtokinesis\n    kind: kinesis\n    params:\n      aws_endpoint: http://localstack:4566\n      aws_access_key_id: foo\n      aws_secret_access_key: bar\n      aws_region: us-east-1\n      aws_resource_name: \"stream\"\n\n  - name: sendtohttp\n    kind: http\n    params:\n      http_endpoint: http://localhost:8080/transform2/sendtohttp2\n      http_header_auth: Bearer mytoken\n      http_status_ok: 200\n\n  - name: sendtohttp2\n    kind: http\n    params:\n      http_endpoint: http://localhost:8080/transform3/sendtohttp3\n      http_header_auth: Bearer mytoken\n      http_status_ok: 200\n\n  - name: sendtohttp3\n    kind: http\n    params:\n      http_endpoint: http://localhost:8080/transform4/sendtosqs\n      http_header_auth: Bearer mytoken\n      http_status_ok: 200\n```\n\nTry this config chaining HTTP Forwarders!\n\n```shell\n$ curl -X POST -H 'Content-type:application/json' -d '{\"user\":{\"data\":{\"name\":\"john\"}}}'  http://localhost:8080/transform1/sendtohttp\n```\n\nThis is the result you'll find in the queue:\n\n```json\n{\n    \"email\": null,\n    \"email2\": \"empty@email.com\",\n    \"email3\": \"empty@email.com\",\n    \"email4\": \"empty@email.com\",\n    \"username\": \"john\",\n    \"username2\": \"john\",\n    \"username3\": \"john\",\n    \"username4\": \"john\"\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwakumaku%2Fjsonshredder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwakumaku%2Fjsonshredder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwakumaku%2Fjsonshredder/lists"}