Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/aperepel/nifi-workshop
A complete custom processor project, for your reference.
https://github.com/aperepel/nifi-workshop
Last synced: 21 days ago
JSON representation
A complete custom processor project, for your reference.
- Host: GitHub
- URL: https://github.com/aperepel/nifi-workshop
- Owner: aperepel
- Created: 2015-09-28T20:39:28.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2015-09-29T13:18:10.000Z (over 9 years ago)
- Last Synced: 2024-02-11T18:45:49.860Z (12 months ago)
- Language: Java
- Homepage:
- Size: 336 KB
- Stars: 17
- Watchers: 1
- Forks: 19
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-nifi - aperepel/nifi-workshop - A complete custom processor project, for your reference (Learning NiFi)
- awesome-nifi - aperepel/nifi-workshop - A complete custom processor project, for your reference (Learning NiFi)
README
# Overview
These are the accompanying materials for the NiFi workshop - Developing a Custom Processor.
![flow overview]
(https://github.com/aperepel/nifi-workshop/blob/master/screenshot.png)
## Install Ambari NiFi Service
https://github.com/abajwa-hw/ambari-nifi-service
## Add the 9090 port forwarding rule to your VirtualBox
### Find the HDP Sandbox VM id:
```
~ ♨ > VBoxManage list vms
"boot2docker-vm" {2e7c902c-34fa-45e9-863b-313beb2b4b33}
"centos64_c6401_1440615279050_15827" {b3f9d4a5-a9cf-4495-90ff-139240a11be7}
"NiFi" {10a766b0-358c-4f33-8d66-f2fc3bd8d770}
"docker-vm" {06e9e8ec-929a-4968-96f0-4f54e20b4cec}
"default" {0fedc097-a65a-47d0-90ab-399824ef069f}
"Hortonworks Sandbox with HDP 2.3" {ab54bfd3-a33b-4844-88c3-924f7df9498a}
```
### Create a rule for the target VM (replace with your ID)
```
~ ♨ > VBoxManage controlvm ab54bfd3-a33b-4844-88c3-924f7df9498a natpf1 nifi,tcp,,9090,,9090
```
## Maven reference documentation for custom NiFi processors
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
### Full command for project generation (non-interactive)
```
mvn archetype:generate \
-DarchetypeGroupId=org.apache.nifi \
-DarchetypeArtifactId=nifi-processor-bundle-archetype \
-DarchetypeVersion=0.3.0 \
-DnifiVersion=0.3.0 \
-DartifactBaseName=workshop-demo \
-Dpackage=com.hortonworks.iot.demo
```
### Update maven central indices if applicable (e.g. in IntelliJ IDEA)
### Fix the top-level pom parent
Change the top-level pom declaration to update the parent version to 0.3.0
### Build the project
```
mvn clean install
```
The NAR package will be placed in the `nifi-workshop-demo-nar/target` directory.
## Upload the NAR via Ambari
* `Ambari -> Local Files`
* Upload to the `/opt/nifi-*/lib`
*If you can't go down into the `lib` directory*: right click on `lib -> Permissions -> RWX for 'others'`
### Re-deploy
* Delete the old NAR file
* Upload a new version
* Restart the NiFI via `Ambari -> NiFi -> Restart All`
But it's painful if you have to repeat it many times. Best practice - extensive NiFi unit testing of components and processors in a local IDE.
See the testcase in the project.
# Misc
* Example of a processor implementation
https://github.com/apache/nifi/blob/master/nifi-external/nifi-example-bundle/nifi-nifi-example-processors/src/main/java/org/apache/nifi/processors/WriteResourceToStream.java
* Example of a processor testcase
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestRouteOnAttribute.java
* Default NiFi bundles
https://github.com/apache/nifi/tree/master/nifi-nar-bundles
* Final result
https://github.com/aperepel/nifi-workshop/tree/master
### NiFi template (can be imported)
* The template is in the `Chuck_Norris_Speaking.xml`
* In NiFi UI go to `Templates` admin section (on the righthand side) and upload the XML
* In NiFi UI main toolbar click `Templates`
* In a resulting popup your new template should be listed. Select and Ok.
# Extra Credit
- Send jokes to a Kafka topic. Different Kafka topics? (Flow design)
- Improve user experience by restricting allowable values for properties (e.g. true/false, pre-defined buffer sizes) (Coding)
- Accumulate jokes to be written in one file, e.g. as a batch of 20 or every 30 seconds (MergeContent Processor)
- Let's do some searching! Route JSON events from the REST API into Solr. Banana dashboard anyone?
- Give Chuck a voice! Find a Text-to-Speech (TTS) library or service and wrap it either in the REST invocation or custom processor.
*Tip: you can download output sound files via a `Data Provenance` event viewer (sorry, servers don't play sounds ;)*