{"id":31917873,"url":"https://github.com/kikejimenez/nifi_api","last_synced_at":"2026-06-16T16:31:50.996Z","repository":{"id":131346052,"uuid":"339869164","full_name":"kikejimenez/nifi_api","owner":"kikejimenez","description":"NIFI API for a Cloudera Project","archived":false,"fork":false,"pushed_at":"2021-06-04T18:13:35.000Z","size":759,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-22T07:48:06.992Z","etag":null,"topics":["cloudera","nifi","nifi-processors","python","rest-api"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kikejimenez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-02-17T22:11:07.000Z","updated_at":"2024-12-11T04:29:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"f7355678-3556-482e-a97d-47aed5cdfc9e","html_url":"https://github.com/kikejimenez/nifi_api","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kikejimenez/nifi_api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kikejimenez%2Fnifi_api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kikejimenez%2Fnifi_api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kikejimenez%2Fnifi_api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kikejimenez%2Fnifi_api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kikejimenez","download_url":"https://codeload.github.com/kikejimenez/nifi_api/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kikejimenez%2Fnifi_api/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34415240,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-16T02:00:06.860Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudera","nifi","nifi-processors","python","rest-api"],"created_at":"2025-10-13T20:50:22.933Z","updated_at":"2026-06-16T16:31:50.986Z","avatar_url":"https://github.com/kikejimenez.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NIFI REST API\n\u003e Automates the Nifi dataflows \n\n\nDataflows can be fully automated using the Nifi API. This project taps and un-taps a dataflow by tracking the incoming/outgoing flowfiles. \n\nMore specifically, it turns on an *initial processor* and records the flowfiles generated by inspecting an *incoming connection*. After this, a *middle processor* is turned on. When all the flowfiles are registered in a *outcoming connection*, a *final processor* is turned on. Before finishing, it returns the pipeline to its initial state.\n\n## What is automated?\n\nThe following example represents  the actions on the dataflow that get automated by the **nifi_api** library.\n\n1. The initial state of the dataflow:\n\n![image.png](nbs/images/1_dataflow.png \" initial state\")\n\n2. Turn on the \"Initial\" processor and turn off the \"Final\" processor:\n\n![Initial State](nbs/images/2_dataflow.png \" middle state\")\n\n3. Turn off the \"Initial\" processor and record the flowfiles in the \"Initial\" connection, then turn on the \"Middle\" processor:\n\n![Initial State](nbs/images/3_dataflow.png \" final state\")\n\n4. Record the flowfiles in the \"Final\" Connection, when they coincide with the \"Initial\" flowfile turn off the \"Middle\" processor and turn on the \"Final\" processor.\n5. The flowfiles in the \"Final\" connection get consumed by the \"Final\" processor and the initial state of the dataflow is recovered.\n\nHere is a recording of the Nifi UI when the tool is being executed on this dataflow:\n\n![image.gif](nbs/images/automated_dataflow.gif \"animation\")\n\n## Environment\n\nThe Nifi cluster used for testing is in the *Cloudera Public Cloud* and needs basic authentication credentials for accessing. The following environmental variables are necessary to access the cluster:\n\n      - CLOUDERA_USER=user\n      - CLOUDERA_PASS=password\n      - CLOUDERA_CLUSTER=https://\u003curl_clustername\u003e.cloudera.site/\u003cclustername\u003e\n      - CLOUDERA_NIFI_REST=/cdp-proxy-api/nifi-app/nifi-api/\n\n## Install\n\nFor a `pip` installation run:\n\n`pip install nifi-api`\n\n## How to use\n\nConsider the template `Test_API.json` in the root folder, this is the template used in `What automates?` section. \n\nWrite the data structure with the Nifi Ids (located in *view configuration -\u003e settings -\u003e Id*) of the processors and connections:\n\n```python\nfrom nifi_api.environment import DataFlowIds\nids = {\n    \"in_connection\": {\n        \"Id\": \"cc549c6e-0177-1000-ffff-ffffb5d2aba2\",\n        \"name\": \"First\"\n    },\n    \"out_connection\": {\n        \"Id\": \"51ab3b24-084f-1309-0000-00001946f2c7\",\n        \"name\": \"Final\"\n    },\n    \"in_processor\": {\n        \"Id\": \"36c62ad6-d606-3b04-9743-d77b6249608c\",\n        \"name\": \"First\"\n    },\n    \"middle_processor\": {\n        \"Id\": \"cc54862f-0177-1000-ffff-ffffe7325a20\",\n        \"name\": \"Middle\"\n    },\n    \"out_processor\": {\n        \"Id\": \"51ab3b1e-084f-1309-a135-aa0100d7186b\",\n        \"name\": \"Final\"\n    },\n}\ndata_ids = DataFlowIds(ids)\n```\n\nInstantiate and run:\n\n```python\nfrom nifi_api.dataflow import DataFlow\ndataflow = DataFlow(\n    dataflow_ids=data_ids,\n    delay_seconds_after_start=5,\n    delay_seconds_between_checks=5,\n)\n```\n\n```python\ndataflow.run()\n```\n\n    pipeline watching has started..\n    Pipeline watching has finished ...\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkikejimenez%2Fnifi_api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkikejimenez%2Fnifi_api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkikejimenez%2Fnifi_api/lists"}