{"id":22213985,"url":"https://github.com/xmlking/nifi-scripting","last_synced_at":"2025-08-09T19:18:53.643Z","repository":{"id":142313655,"uuid":"46114718","full_name":"xmlking/nifi-scripting","owner":"xmlking","description":"NiFi Dynamic Script Executors","archived":false,"fork":false,"pushed_at":"2016-07-17T08:46:38.000Z","size":141,"stargazers_count":15,"open_issues_count":0,"forks_count":8,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-05-01T20:22:19.727Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Groovy","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xmlking.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2015-11-13T10:10:38.000Z","updated_at":"2021-09-18T03:44:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"d4c89979-e01a-467b-97af-b8f70b1799aa","html_url":"https://github.com/xmlking/nifi-scripting","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xmlking%2Fnifi-scripting","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xmlking%2Fnifi-scripting/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xmlking%2Fnifi-scripting/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xmlking%2Fnifi-scripting/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xmlking","download_url":"https://codeload.github.com/xmlking/nifi-scripting/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227802821,"owners_count":17822113,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-02T21:12:52.372Z","updated_at":"2024-12-02T21:12:52.876Z","avatar_url":"https://github.com/xmlking.png","language":"Groovy","readme":"NiFi-Scripting\n--------------\nThe goal of this project is to enable processing NiFi *FlowFiles* using scripting languages.   \n   \n1. **ExecuteJavaScript**        Execute supplied javaScript with arguments configured. Use case: JSON -\u003e Mapping -\u003e JSON\n2. **ExecuteGroovy**:           Execute supplied groovy script with arguments configured. \n3. **ExecuteGroovyLdap**:       Provide configured LDAP object to the script. Execute supplied groovy script with arguments configured. See [Groovy LDAP](https://directory.apache.org/api/groovy-api/2-groovy-ldap-user-guide.html)\n4. **ExecuteRemoteProcess**:    Similar to NiFi built-in **ExecuteProcess** but run on remote host. See [Groovy SSH](https://github.com/int128/groovy-ssh)\n\n\n\u003e **ExecuteJavaScript** and **ExecuteGroovy** are depricated since NiFi from version 0.5.1 includes **ExecuteScript** and **InvokeScriptProcessor**\n\nYou can still take advantage of **nifi-sumo-common** lib in scripting processors, e.g., convert `FlowFile \u003c--\u003e String`\n\n\n### Install NiFi\n1. Manual: Download [Apache NiFi](https://nifi.apache.org/download.html) binaries and unpack to a folder. \n2. On Mac: brew install nifi\n\n### Deploy NAR files.\n```bash\n# Assume you unpacked nifi-0.6.1-bin.zip to /Developer/Applications/nifi\ngradle clean deploy -Pnifi_home=/Developer/Applications/nifi\n```\n\n```\n# if you install NiFi via brew\ngradle clean deploy -Pnifi_home=/usr/local/Cellar/nifi/0.6.1/libexec\n```\n\n### Start NiFi\n```bash\ncd /Developer/Applications/nifi\n./bin/nifi.sh  start\n./bin/nifi.sh  stop\n```\n\n```\n# If you install NiFi via brew\n# Working Directory: /usr/local/Cellar/nifi/0.6.1/libexec\nnifi start|stop|run|restart|status|dump|install\nnifi start \nnifi status  \nnifi stop\n```\n\n### Testing \n\nUpload the [sample flow](./scripting-flow.xml) into NiFi and use [test data](./src/test/resources/test.json) and below javascript for testing:\n\n```js\nfunction convert(val) {\n    var g = JSON.parse(val);\n    var d = {\n        widget: g.widget.window.title,\n        imageURI: g.widget.image.src\n    };\n    return JSON.stringify(d);\n}\n\nprint(\"Hello from inside scripting!\");\n\nvar  fs = util.flowFileToString(flowFile, session);\nlog.error(fs);\nvar flowString = convert(fs);\nlog.error(flowString);\n\n//flowFile = session.importFrom(buff, true, flowFile);\nsession.remove(flowFile);\nflowFile = util.stringToFlowFile(flowString, session);\n\n\nflowFile = session.putAttribute(flowFile, \"JS\", 2222 );\n```\n\n#### Using 3rd party libs with Groovy Script via @Grab\n\u003e You need to copy Apache Ivy [JAR](http://ant.apache.org/ivy/download.cgi) to **NiFi/bin** for @Grab to work.\n\nAdd your twitter's *consumerKey, consumerSecret accessToken and secretToken* to ExecuteScript's dynamic properties \n\n```Groovy\n@Grab('org.codehaus.groovy.modules.http-builder:http-builder:0.7')\n@Grab('oauth.signpost:signpost-core:1.2.1.2')\n@Grab('oauth.signpost:signpost-commonshttp4:1.2.1.2')\n\nimport groovy.json.JsonOutput\nimport groovyx.net.http.RESTClient\nimport static groovyx.net.http.ContentType.*\nimport org.apache.http.params.HttpConnectionParams\nimport com.crossbusiness.nifi.processors.NiFiUtils as util\n\n//Access dynamic property\nconsumerKey = consumerKey.value\nconsumerSecret = consumerSecret.value\naccessToken = accessToken.value\nsecretToken = secretToken.value\n\ndef twitter = new RESTClient( 'https://api.twitter.com/1.1/statuses/' )\ntwitter.auth.oauth  consumerKey, consumerSecret, accessToken, secretToken\ntwitter.contentType = JSON\nHttpConnectionParams.setSoTimeout twitter.client.params, 15000\n\ndef resp = twitter.get( path: 'home_timeline.json' )\nassert resp.status == 200\nassert resp.contentType == JSON.toString()\nassert ( resp.data instanceof List )\nassert resp.data.status.size() \u003e 0\n\nflowFile = util.stringToFlowFile(JsonOutput.toJson(resp.data), session);\nsession.transfer(flowFile, REL_SUCCESS)\n```\n\n\n### Using `nifi-sumo-common` lib in scripting processors\n\n1. Copy `nifi-sumo-common-x.y.x-SNAPSHOT.jar` from [releases](https://github.com/xmlking/nifi-scripting/releases) to  *Module Directory* set in the `ExecuteScript` Processor's properties. \n\n#### How to use NiFiUtil?\n\n2. Import `NiFiUtils` into `ExecuteScript`'s Script \n\n```groovy\nimport com.crossbusiness.nifi.processors.NiFiUtils as util\nflowFile = util.stringToFlowFile(\"test 123\", session);\nflowString = util.flowFileToString(flowFile, session)\nlog.info \"flowString: ${flowString}\"\nsession.transfer(flowFile, REL_SUCCESS)\n```\n\n#### How to use distributed cache in scripting processors?\n\n2. Import `StringSerDe` or `LongSerDe` etc., into `ExecuteScript`'s Script \n\n```groovy\nimport org.apache.nifi.controller.ControllerService\nimport com.crossbusiness.nifi.processors.StringSerDe\n\nfinal StringSerDe stringSerDe = new StringSerDe();\n\ndef lookup = context.controllerServiceLookup\ndef cacheServiceName = DistributedMapCacheClientServiceName.value\n\nlog.error  \"cacheServiceName: ${cacheServiceName}\"\n\ndef cacheServiceId = lookup.getControllerServiceIdentifiers(ControllerService).find {\n    cs -\u003e lookup.getControllerServiceName(cs) == cacheServiceName\n}\n\nlog.error  \"cacheServiceId:  ${cacheServiceId}\"\n\ndef cache = lookup.getControllerService(cacheServiceId)\nlog.error cache.get(\"aaa\", stringSerDe, stringSerDe )\n```\n\n\n#### ExecuteRemoteProcess testing\n\n**SSH Config DSL**\n```groovy\nremotes {\n    web01 {\n        role 'masterNode'\n        host = '192.168.1.5'\n        user = 'sumo'\n        password = 'demo'\n        knownHosts = allowAnyHosts\n    }\n    web02 {\n        host = '192.168.1.5'\n        user = 'sumo'\n        knownHosts = allowAnyHosts\n    }\n}\n```\n\n**Run DSL**\n```groovy\nssh.run {\n    session(ssh.remotes.web01) {\n          result = execute 'uname -a' \n    }\n}\n```\n\n### Build \n```bash\ngradle nar\n```\n\n#### If you are using MapR hadoop distribution\n\n1. Follow steps [NiFi Hadoop Library for MapR](https://github.com/xmlking/mapr-nifi-hadoop-libraries-bundle)\n2. Set auth login config in $NIFI_HOME/conf/bootstrap.conf\n\n    `java.arg.15=-Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf`\n\n\n### TODO\n1. Support adding popular javaScript libraries (lodash.js, moment.js etc.,) via processor configuration.\n1. ExecuteRemoteProcess: add expression language support for RUN_DSL.\n \n\n### Reference  \n1. [Groovy Script](http://www.groovy-lang.org/integrating.html)\n2. [java8-nashorn-tutorial](http://winterbe.com/posts/2014/04/05/java8-nashorn-tutorial/)\n3. [Mapping Complex JSON Structures With JDK8 Nashorn](https://dzone.com/articles/mapping-complex-json-structures-with-jdk8-nashorn)\n4. [Groovy SSH](https://github.com/int128/groovy-ssh)\n5. See [document of Gradle SSH Plugin](https://gradle-ssh-plugin.github.io/docs/) for details of DSL.\n6. Groovy Goodness: [Store Closures in Script Binding](http://mrhaki.blogspot.com/2010/08/groovy-goodness-store-closures-in.html)\n7. Matt Burgess's Blog: [Fun with Apache NiFi](http://funnifi.blogspot.com/)","funding_links":[],"categories":["Processors and Bundles"],"sub_categories":["Mailing List Best Of"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxmlking%2Fnifi-scripting","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxmlking%2Fnifi-scripting","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxmlking%2Fnifi-scripting/lists"}