Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mrcsparker/nifi-sqllookup-services-bundle
NIFI controllers for SQL record and attributes lookups with built-in caching
https://github.com/mrcsparker/nifi-sqllookup-services-bundle
java jdbc nifi nifi-processor nifi-processors sql
Last synced: 19 days ago
JSON representation
NIFI controllers for SQL record and attributes lookups with built-in caching
- Host: GitHub
- URL: https://github.com/mrcsparker/nifi-sqllookup-services-bundle
- Owner: mrcsparker
- License: apache-2.0
- Created: 2018-03-21T22:52:52.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2021-11-27T05:49:24.000Z (almost 3 years ago)
- Last Synced: 2024-10-11T12:26:18.321Z (about 1 month ago)
- Topics: java, jdbc, nifi, nifi-processor, nifi-processors, sql
- Language: Java
- Homepage: https://mrcsparker.github.io/nifi-sqllookup-services-bundle
- Size: 601 KB
- Stars: 18
- Watchers: 3
- Forks: 11
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# Apache NiFi SQL Lookup Service
[](https://travis-ci.org/mrcsparker/nifi-sqllookup-services-bundle)
- [Apache NiFi SQL Lookup Service](#apache-nifi-sql-lookup-service)
- [About](#about)
- [Simple Setup](#simple-setup)
- [SQL Query Support](#sql-query-support)
- [Caching](#caching)
- [Supported caches](#supported-caches)
- [Latest release](#latest-release)
- [Articles on using NiFi lookup services](#articles-on-using-nifi-lookup-services)
## About
**NiFI SQL Lookup Service** is a SQL-based lookup service for [Apache NiFi](https://nifi.apache.org). It allows you to enrich your flowfiles with any jdbc-compliant data store, including:
- PostgreSQL
- Oracle
- MySQL
- MS SQL Server
- SQLite
- ... and many more
It includes both [LookupRecord](http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.LookupRecord/) and [LookupAttribute](https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.LookupAttribute/) controllers.
These controllers were designed to be _flexible_ and _fast_.
## Simple Setup
- [Download Apache NiFi](https://nifi.apache.org/download.html)
- Compile and install `nifi-sqllookup-bundle`
```bash
> cd nifi-sqllookup-bundle
> mvn package
> cp ./nifi-sqllookup-services-nar/target/nifi-sqllookup-services-nar-1.15.0.nar /NIFI_INSTALL/lib/
> cp ./nifi-sqllookup-services-api-nar/target/nifi-sqllookup-services-api-nar-1.15.0.nar /NIFI_INSTALL/lib/
```
- Start NiFi
## SQL Query Support
This service supports multiple query types:
- _Named Parameters_: `SELECT name FROM foo WHERE value = :value`
- _SQL IN queries_: `SELECT name FROM foo WHERE value IN(:values)`
- _Multiple lookup values_: `SELECT name FROM foo WHERE value IN(:values) AND sequence = :sequence AND catalog = :catalog`
## Caching
The goal of this service is to return values quickly. It has a built-in cache so that your database doesn't get overwhelmed.
This is configurable in the controller settings.
### Supported caches
These caches are all built-in to this service. Select your preferable cache in the controller settings.
- [Caffeine](https://github.com/ben-manes/caffeine) - default cache.
- [Cache2k](https://cache2k.org)
- [Guava](https://github.com/google/guava/wiki/CachesExplained)
You can also select the number of items that you want to cache. The caches all keep the most accessed items available by default.
If you don't know which to choose, just go with the default.
## Latest release
[https://github.com/mrcsparker/nifi-sqllookup-services-bundle/releases/latest](https://github.com/mrcsparker/nifi-sqllookup-services-bundle/releases/latest)
## Articles on using NiFi lookup services
* [Data flow enrichment with NiFi part 1 : LookupRecord processor](https://community.hortonworks.com/articles/138632/data-flow-enrichment-with-nifi-lookuprecord-proces.html)
* [Data flow enrichment with NiFi part 2 : LookupAttribute processor](https://community.hortonworks.com/articles/140231/data-flow-enrichment-with-nifi-part-2-lookupattrib.html)
* [Data flow enrichment with NiFi part 3: LookupRecord with MongoDB](https://community.hortonworks.com/articles/146198/data-flow-enrichment-with-nifi-part-3-lookuprecord.html)