{"id":20578774,"url":"https://github.com/cclient/nifi-mutate-bundle","last_synced_at":"2026-05-28T18:31:30.891Z","repository":{"id":93977421,"uuid":"343074062","full_name":"cclient/nifi-mutate-bundle","owner":"cclient","description":"应用官方expression-language（EL）实现类似logstash的mutate，对每行一条json对象string的flowfile,按行应用mutate规则,修改原始json对象的值,输出变更后的flowfile","archived":false,"fork":false,"pushed_at":"2021-02-28T13:48:01.000Z","size":14,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-06T11:43:49.098Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://www.cnblogs.com/zihunqingxin/p/14460713.html","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cclient.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-28T10:13:07.000Z","updated_at":"2023-12-29T07:07:49.000Z","dependencies_parsed_at":"2023-04-20T10:34:21.369Z","dependency_job_id":null,"html_url":"https://github.com/cclient/nifi-mutate-bundle","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cclient/nifi-mutate-bundle","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cclient%2Fnifi-mutate-bundle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cclient%2Fnifi-mutate-bundle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cclient%2Fnifi-mutate-bundle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cclient%2Fnifi-mutate-bundle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cclient","download_url":"https://codeload.github.com/cclient/nifi-mutate-bundle/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cclient%2Fnifi-mutate-bundle/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33622067,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-28T02:00:06.440Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T06:14:31.156Z","updated_at":"2026-05-28T18:31:30.816Z","avatar_url":"https://github.com/cclient.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"Nifi Mutate FlowFile Content Processor\n================================\n\n## 功能描述\n\n适于于 flowfile为文本数据，一行一个json 'JSON-per-line' format\n\n对每行一条json对象string的flowfile,按行应用mutate规则,修改原始json对象的值,输出变更后的flowfile\n\n通常是根据需要因需实现语法解析规则\n\n该项目主要是应用官方[expression-language](http://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html)（EL）实现mutate\n\n## 应用原理\n\n首先nifi官方 EL 默认可以应用在 'Variable Registry and FlowFile Attributes'上\n\n查看源码和开发自定义Processor的经验可知应用 EL 方法为propertyValue.evaluateAttributeExpressions(flowFile)\n\nevaluateAttributeExpressions有多个重载方法,其中一项为\n\n`public PropertyValue evaluateAttributeExpressions(Map\u003cString, String\u003e attributes)`\n\n查看nifi源码和实际验证知`evaluateAttributeExpressions(Map\u003cString, String\u003e attributes)`可以应用在任何Map\u003cString, String\u003e上\n\n因此只要能将flowfile content转换为Map\u003cString,String\u003e,便可对flowfile content应用官方 EL\n\n这点很容易,一行一条json对象string是比较常见的数据传输格式,例如(file,kafka,elasticsearch,logstash event等各种数据组件)\n\n逐行读取flowfile,每行转为json对象后转换为Map\u003cString,String\u003e结构,应用官方 EL 取值,以一定的规则应用新值更改原始json后，把新的flowfile输出\n\n---\n\n目前转换规则封装的只是应用 EL 取出value后的行为,取value本身完全依赖官方EL\n\n官方EL支持操作\\'Variable Registry and FlowFile Attributes\\',通过这种方式更改flowfile的content,增大了官方EL的应用场景\n\n转换规则配置方式为json obj 目前只支持最外层key,a.b.c类的多级结构后续再支持\n\n目标是实现部分logstash的常见功能[logstash-plugins-filters-mutate](https://www.elastic.co/guide/en/logstash/6.3/plugins-filters-mutate.html)\n\n逐步替换logstash组件\n\nEL执行后的值内容为PropertyValue，丢失了类型信息,因此取值后都需再指定一次targetType作类型转换,targetType参照[logstash-plugins-filters-mutate-convert](https://www.elastic.co/guide/en/logstash/6.3/plugins-filters-mutate.html#plugins-filters-mutate-convert)\n\n目前支持targetType: boolean、string、integer、long、float、double，后期扩展其他类型\n\n目前支持行为\n\n* 1 delete key 删除key {\"test_delete_key\":{\"operate\":\"remove\"}}\n\n* 2 add key 不存在该key则新增 {\"test_value_sum\":{\"el\":\"${test_value_key1:plus(${test_value_key2}):plus(${test_value_key3})}\",\"targetType\":\"long\"}}\n\n* 3 replace/update value 存在该key则更新 {\"test_update_value\": {\"el\": \"2\",\"targetType\": \"string\"}}\n\n* 4 rename key 更改key名称(实际为添加一个新key,并删除旧key) {\"rename_key\": {\"el\": \"${test_rename_key}\",\"targetType\": \"string\",\"orginalKey\": \"test_rename_key\",\"operate\": \"remove\"}}\n\n### demo\n\n#### 原始json string\n\n```org\n{\n    \"test_delete_key\": 1,\n    \"test_update_value\": \"1 to 2\",\n    \"test_rename_key\": \"test_rename_key -\u003e rename_key\",\n    \"test_value_key1\": 200,\n    \"test_value_key2\": 200,\n    \"test_value_key3\": 4,\n    \"test_value_toUpper\": \"cclient@hotmail.com\",\n    \"test_value_timeStampToStr\": \"1590117918909\"\n}\n```\n\n#### 转换规则\n\n```org\n{\n    \"test_delete_key\": {\n        \"operate\": \"remove\"\n    },\n    \"test_update_value\": {\n        \"el\": \"2\",\n        \"targetType\": \"string\"\n    },\n    \"rename_key\": {\n        \"el\": \"${test_rename_key}\",\n        \"targetType\": \"string\",\n        \"orginalKey\": \"test_rename_key\",\n        \"operate\": \"remove\"\n    },\n    \"test_value_toUpper\": {\n        \"el\": \"${test_value_toUpper:toUpper()}\",\n        \"targetType\": \"string\"\n    },\n    \"test_value_timeStampToStr\": {\n        \"el\": \"${test_value_timeStampToStr:format(\\\"yyyy/MM/dd\\\", \\\"GMT\\\")}\",\n        \"targetType\": \"string\"\n    },\n    \"test_value_sum\": {\n        \"el\": \"${test_value_key1:plus(${test_value_key2}):plus(${test_value_key3})}\",\n        \"targetType\": \"long\"\n    }\n}\n```\n\n#### 变更后\n\n```mutated\n{\n    \"test_delete_key\": 1,\n    \"test_update_value\": \"2\",\n    \"test_value_key1\": 200,\n    \"test_value_key2\": 200,\n    \"test_value_key3\": 4,\n    \"test_value_toUpper\": \"CCLIENT@HOTMAIL.COM\",\n    \"test_value_timeStampToStr\": \"2020/05/22\",\n    \"rename_key\": \"test_rename_key -\u003e rename_key\",\n    \"test_value_sum\": 404\n}\n```\n\n### 需求\n\n数据处理会有对原始数据做微调的场景,过去通过大量logstash的filter实现\n\n较为常用logstash filter有mutate,split,aggregate等，复杂场景通过写ruby代码和自定义filter解决\n\nlogstash很优秀,但随着更定制功能的需求,长期使用下也累积了一些使用上的痛点,通过logstash本身较难低成本的解决，调研学习nifi后，逐步将部分数据流迁移至nifi生态\n\n最大的痛点\n\n该Processor用来实现较为初级的数据微调需求\n\n实际难度不高，主要创新点是把官方通常只应用在'Variable Registry and FlowFile Attributes'的EL，扩展应用在FlowFile Content上\n\n项目只是展示可行性，实际可以更进一步的定制和添加功能\n\n### deploy\n\n#### 1 compile\n\n`mvn package -Dmaven.test.skip=true`\n\n#### 2 upload to one of\n\n```nifi\nnifi.nar.library.directory=./lib\nnifi.nar.library.directory.custom=./lib_custom\nnifi.nar.library.autoload.directory=./extensions\nnifi.nar.working.directory=./work/nar/\n\n```\n\ncp nifi-mutate-nar/target/nifi-mutate-nar-0.1.nar nifi/lib_custom/\n\n#### 3 restart nifi if need\n\nnifi/bin/nifi.sh restart\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcclient%2Fnifi-mutate-bundle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcclient%2Fnifi-mutate-bundle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcclient%2Fnifi-mutate-bundle/lists"}