{"id":15056425,"url":"https://github.com/joaohf/nifi_s2s","last_synced_at":"2026-01-30T07:12:02.575Z","repository":{"id":57530052,"uuid":"274762016","full_name":"joaohf/nifi_s2s","owner":"joaohf","description":"NiFi Site-to-Site Erlang client implementation","archived":false,"fork":false,"pushed_at":"2020-06-24T20:48:25.000Z","size":111,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-20T19:54:28.126Z","etag":null,"topics":["beam","dataflow","elixir","erlang","flowfiles","nifi"],"latest_commit_sha":null,"homepage":"","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joaohf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-24T20:21:47.000Z","updated_at":"2024-11-01T10:43:08.000Z","dependencies_parsed_at":"2022-09-26T18:11:27.252Z","dependency_job_id":null,"html_url":"https://github.com/joaohf/nifi_s2s","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/joaohf/nifi_s2s","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joaohf%2Fnifi_s2s","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joaohf%2Fnifi_s2s/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joaohf%2Fnifi_s2s/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joaohf%2Fnifi_s2s/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joaohf","download_url":"https://codeload.github.com/joaohf/nifi_s2s/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joaohf%2Fnifi_s2s/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260082361,"owners_count":22956279,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beam","dataflow","elixir","erlang","flowfiles","nifi"],"created_at":"2024-09-24T21:51:16.310Z","updated_at":"2026-01-30T07:11:57.552Z","avatar_url":"https://github.com/joaohf.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nifi_s2s\n\nnifi_s2s is an Erlang implementation of NiFi Site-to-Site flowfile transfer protocol.\n\nIf you are not aware about what does NiFi mean, please take a look in the official [Apache NiFi project](https://nifi.apache.org).\n\nThis library only makes sense if you need to transfer data from and to NiFi node or cluster.\n\n## Overview\n\n[Apache NiFi](https://nifi.apache.org) is a tool aim to automate the flow of data between systems. The tool brings valuable features that allows moving data around secondary systems.\n\n[NiFi Site to Site Protocol](https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#site-to-site) is a native NiFi protocol which can be used in order to transfer flowfiles between systems.\n\nThe intention of this application library (nifi_s2s) is to provide a site-to-site Erlang client implementation where flowfiles can be transferred or received.\n\nThe figure below shows a common use case where a device transfers data to a central NiFi node:\n\n\u003c!---\n@startuml\nnode \"IoT Device\" as d\nnode \"Nifi Node\" as n\nnode \"Server\" as s\n\nd -r-\u003e n : Site to Site\\nraw\nn .r.\u003e s : Site to Site\\nhttp\n@enduml\n--\u003e\n\n![Network deployment](http://www.plantuml.com/plantuml/png/PSwX3i9G38LXtwSucTyU0B5CO30pGs9kQ0atWPQqPRm-P5mtTPBlc7-AB9xlvuD8ZGNToSuuobglqg44kF5SRnMpyL-rwIA-YcyKH8pU-n6A0vQQWhHjB-hbGuh1Xn6nE--PBw99b7yLNm00 \"Network deployment\")\n\n\nCurrently, there are a few NiFi Site-to-Site implementations, none targeting BEAM VM. The most important are:\n\n * [Java](https://github.com/apache/nifi/tree/master/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/client)\n * [C++](https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/src/sitetosite)\n * [C](https://github.com/apache/nifi-minifi-cpp/tree/master/nanofi/src/sitetosite)\n\nAll of the above implementations are from NiFi official project.\n\nThis application library provides an Erlang site-to-site client which could be used to connect and send data to and from NiFi nodes or clusters.\n\nThe figure below shows a typical NiFi flowfile which there are three processors:\n\n * receiving flowfiles using a port\n * writing each flowfiles to file\n * sending each flowfile to remote port\n\n![Nifi Site to Site Template](./doc/nifi.png \"Site to Site example\")\n\n## Design\n\nThis library follows the common site-to-site protocol sequence as described in [NiFi System Administrator’s Guide](https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#site-to-site-protocol-sequence).\n\nThere are two state machines which encapsulates all the site-to-site protocol:\n\n* `nifi_s2s_raw_protocol_statem`: in charge of controlling the initial connection to NiFi instance and also establishing the Peer connection\n* `nifi_s2s_transaction_statem`: responsible for creating and control the transaction state when sending or receiving flowfiles.\n\nThe main user level interface is implemented by the module `nifi_s2s`. The rest of the code are just about help functions and encode/decode helpers.\n\n\n## How to use\n\nThis client exports to main functions to transfer and receive flowfiles. \n\nThe follow example show how to create a client which uses the raw protocol and transfer\na flowfile to a Peer (that is a NiFi node):\n\n```\n    S2SConfig = #{hostname =\u003e \"localhost\",\n                  port =\u003e 8080,\n                  transport_protocol =\u003e raw,\n                  local_network_interface =\u003e \"lo0\",\n                  port_id =\u003e ?config(input_port, Config)},\n\n    {ok, Pid} = nifi_s2s:create_client(S2SConfig),\n\n    Content = \u003c\u003c\"Test Nifi Content\"\u003e\u003e,\n    Attributes = #{ \u003c\u003c\"TEST1\"\u003e\u003e =\u003e \u003c\u003c\"Test\"\u003e\u003e},\n\n    Flowfiles = nifi_flowfile:add(Attributes, Content, nifi_flowfile:new())\n\n    ok = nifi_s2s:transfer_flowfiles(Pid, Flowfiles),\n\n    ok = nifi_s2s:close(Pid).\n```\n\nNow, in order to receive flowfiles from the remote Peer:\n\n```\n    {ok, _Flowfiles} = nifi_s2s:receive_flowfiles(Pid).\n```\n\n## Build\n\n```\nrebar3 compile\n```\n\n## Test\n\nIn order to run unit tests:\n\n```\nrebar3 eunit\n```\n\nHowever, to run integration tests the approach is to have a running NiFi instance on the developer machine and import the template `samples/Input_and_Output_Flowfiles.xml`. Having it imported and started is necessary when running integration tests. After that, run:\n\n```\nrebar3 ct\n```\n\n## TODO\n\n- [x] Sending and receiving flowfiles\n- [] Add internal metrics\n- [] Add SSL context\n- [] Add suport to compress flowfile\n- [] Add http protocol\n- [] Improve the state machine and use push and pop callbacks from statem\n- [] Improve test code coverage to get edge cases\n- [] Add supervisor tree\n- [] Implement peer work checks and reconnect to the less busy peer\n- [] Add Peer list persistent file\n- [] Add async receive and transfer flowfiles API\n\n## License\n\n[MIT](https://spdx.org/licenses/MIT.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoaohf%2Fnifi_s2s","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoaohf%2Fnifi_s2s","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoaohf%2Fnifi_s2s/lists"}