Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/bbende/nifi-streaming-examples

Collection of examples integrating NiFi with stream process frameworks.
https://github.com/bbende/nifi-streaming-examples

Last synced: about 1 month ago
JSON representation

Collection of examples integrating NiFi with stream process frameworks.

Host: GitHub
URL: https://github.com/bbende/nifi-streaming-examples
Owner: bbende
License: apache-2.0
Created: 2016-02-01T18:06:51.000Z (over 8 years ago)
Default Branch: master
Last Pushed: 2016-09-20T15:33:07.000Z (almost 8 years ago)
Last Synced: 2024-03-15T05:21:44.744Z (4 months ago)
Language: Java
Size: 1.59 MB
Stars: 55
Watchers: 4
Forks: 32
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Lists

awesome-nifi - Examples integrating NiFi with stream process frameworks
awesome-nifi - Examples integrating NiFi with stream process frameworks
awesome-nifi - Examples integrating NiFi with stream process frameworks

README

# Apache NiFi Streaming Examples
Collection of examples integrating NiFi with stream process frameworks.

## Initial Setup

* Download the latest [Apache NiFi release](https://nifi.apache.org/download.html)

* Extract the tar and create two instances of NiFi:

  

    tar xzvf nifi-1.0.0-bin.tar.gz

    mv nifi-1.0.0 nifi-edge

    tar xzvf nifi-1.0.0-bin.tar.gz

    mv nifi-1.0.0 nifi-core

* Configure the edge instance by editing nifi-edge/conf/nifi.properties and setting the following properties:

  

    nifi.remote.input.socket.port=7088

    nifi.remote.input.secure=false

    nifi.web.http.port=7080

* Configure the core instance by editing nifi-core/conf/nifi.properties and setting the following properties:

 

    nifi.remote.input.socket.port=8088

    nifi.remote.input.secure=false

    nifi.web.http.port=8080

* Start both instances

 

    ./nifi-core/bin/nifi.sh start

    ./nifi-edge/bin/nifi.sh start

* Open the UI for both instances in a browser

 

    http://localhost:7080/nifi/

    http://localhost:8080/nifi/

* Setup initial dictionary files



    mkdir nifi-edge/data

    mkdir nifi-edge/data/dictionary

    mkdir nifi-core/data

    mkdir nifi-core/data/dictionary

* In each of the above dictionary directories, create a file called levels.txt with the content:



    ERROR

    WARN

* Import nifi-streaming-examples/templates/nifi-log-example-edge.xml into the the edge instance (http://localhost:7080/nifi)

* Import nifi-streaming-examples/templates/nifi-log-example-core.xml into the the core instance (http://localhost:8080/nifi)

* Start everything on the core instance (http://localhost:8080/nifi)
![Image](https://github.com/bbende/nifi-streaming-examples/blob/master/screens/nifi-core.png?raw=true)

* To start sending logs, starting everything on the edge instance (http://localhost:8080/nifi) EXCEPT the TailFile processor, the "Generate Test Logs" process group will send fake log messages
![Image](https://github.com/bbende/nifi-streaming-examples/blob/master/screens/nifi-edge.png?raw=true)

* To tail a real file, stop the "Generate Test Logs" process group, configure TailFile to point to your log file of choice, and start the TailFile processor

## Flink - WindowLogLevelCount - Setup
* For local testing, run a standalone Flink streaming job



  cd nifi-flink-examples

  mvn clean package -PWindowLogLevelCount

  java -jar target/nifi-flink-examples-0.0.1-SNAPSHOT.jar

## Apex - LogLevelApplicationRunner - Setup

* For local testing, run LogLevelApplicationRunner from your favorite IDE:



  nifi-apex-examples/src/test/java/nifi/apex/examples/logs/LogLevelApplicationRunner.java

## Storm - LogLevelCountTopology - Setup

* For local testing, run a standalone local Storm topology



  cd nifi-storm-examples

  mvn clean package -PLogLevelCountTopology

  java -jar target/nifi-storm-examples-0.0.1-SNAPSHOT.jar