https://github.com/javiroman/dfsadmin-inotify

Simple Java example for testing the DFSAdmin API used in Apache NiFi GetHDFSEvents Processor
https://github.com/javiroman/dfsadmin-inotify

hadoop hdfs nifi nifi-processors

Last synced: about 1 month ago
JSON representation

Simple Java example for testing the DFSAdmin API used in Apache NiFi GetHDFSEvents Processor

• Host: GitHub
• URL: https://github.com/javiroman/dfsadmin-inotify
• Owner: javiroman
• License: apache-2.0
• Created: 2020-08-29T06:34:33.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2021-07-13T06:38:53.000Z (over 3 years ago)
• Last Synced: 2024-11-08T09:47:49.437Z (3 months ago)
• Topics: hadoop, hdfs, nifi, nifi-processors
• Language: Java
• Homepage:
• Size: 4.62 MB
• Stars: 1
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# HDFS Inotify Java Program

Gets iNotify HDFS events and print out in JSON to stdout.

# HDFS iNotify API

Clients obtain a DFSInotifyEventInputStream, from which they can read events, by calling

HdfsAdmin.getInotifyEventStream(). Filesystem events that happen after the client makes

this call will be available in the stream. DFSInotifyEventInputStream provides the following

three methods:

1. public Event **poll()** – a non-blocking call that returns the next event if it is immediately available, otherwise null

2. public Event **poll(long time, TimeUnit tu)** – waits up to the given amount of time for a new event, otherwise returns null

3. public Event **take()** – blocks indefinitely for a new event

## Packages: 

### Package: org.apache.hadoop.hdfs.inotify

Classes: [Event](https://hadoop.apache.org/docs/r3.2.1/api/org/apache/hadoop/hdfs/inotify/Event.html), [EventBatch](https://hadoop.apache.org/docs/r3.2.1/api/org/apache/hadoop/hdfs/inotify/EventBatch.html)

### Package: org.apache.hadoop.hdfs.DFSInotifyEventInputStream

Class: [DFSInotifyEventInputStream](https://hadoop.apache.org/docs/r3.2.1/api/index.html?org/apache/hadoop/hdfs/DFSInotifyEventInputStream.html)

# iNotify Protocol

Protocol buffers used to communicate edits to clients: [inotify.proto](https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/inotify.proto)

# HDFS iNotify Documentation

Docs folder:

inotify-design.4.pdf

Large-Scale-Multi-Tenant-Inotify-Service.pdf

2015-03-05_keep_me_in_the_loop_introducing_HDFS_inotify.pdf

# Usage example

You must run this tool as the hdfs user.

    Usage: $ java -jar hdfs-inotify-example-uber.jar   []

This is a quick and dirty example.  If you omit the TxId arg,like this:

    $ sudo -u hdfs java -jar hdfs-inotify-example-uber.jar hdfs://brooklyn.onefoursix.com:8020

    

... the output might be quite verbose, as you will get all tx's

So you might want to start with a large TxId and then work backwards if you don't get any events.

(If the TxId is larger than the number of tx's then you will simply get no data back)

For my test on a new HDFS I will start will a TxId of 0:

    $ sudo -u hdfs java -jar hdfs-inotify-example-uber.jar hdfs://brooklyn.onefoursix.com:8020 0

I see output that ends like this:

    ...

    TxId = 351352

    event type = CREATE

      path = /tmp/.cloudera_health_monitoring_canary_files/.canary_file_2015_07_10-20_29_11

      owner = hdfs

      ctime = 1436585351213

ctrl-c to kill the app

From that you can see the last TxId was 351352

You can then call the app like this to get all subsequent tx's:

    $ sudo -u hdfs java -jar hdfs-inotify-example-uber.jar hdfs://brooklyn.onefoursix.com:8020 351352

While that is still running, in another session, create a couple of files in HDFS, then delete one (without using -skipTrash) and delete the other with -skipTrash.

You should see a couple of CREATE events, a RENAME and an UNLINK, like this:

    

    TxId = 351411

    event type = CREATE

      path = /user/mark/data106.txt._COPYING_

      owner = mark

      ctime = 1436585999907

    TxId = 351412

    event type = CLOSE

    TxId = 351413

    event type = RENAME

      src = /user/mark/data106.txt._COPYING_

      dst = /user/mark/data106.txt

      timestamp = 1436586000137

    TxId = 351420

    event type = RENAME

      src = /user/mark/data106.txt

      dst = /user/mark/.Trash/Current/user/mark/data106.txt

      timestamp = 1436586013973

    TxId = 351421

    event type = CREATE

      path = /user/mark/data107.txt._COPYING_

      owner = mark

      ctime = 1436586067256

    TxId = 351425

    event type = CLOSE

    TxId = 351426

    event type = RENAME

      src = /user/mark/data107.txt._COPYING_

      dst = /user/mark/data107.txt

      timestamp = 1436586067489

    TxId = 351427

    event type = UNLINK

      path = /user/mark/data107.txt

      timestamp = 1436586074079