{"id":24736380,"url":"https://github.com/engineering-research-and-development/opr4aa-platform","last_synced_at":"2026-04-04T22:31:22.229Z","repository":{"id":222806420,"uuid":"627768761","full_name":"Engineering-Research-and-Development/opr4aa-platform","owner":"Engineering-Research-and-Development","description":"FIWARE/Apache Platform aiming to cover the entire industrial data value chain","archived":false,"fork":false,"pushed_at":"2023-04-21T13:03:25.000Z","size":1511,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-16T10:49:19.284Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Engineering-Research-and-Development.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-04-14T06:53:17.000Z","updated_at":"2024-04-12T09:30:52.000Z","dependencies_parsed_at":"2024-02-16T11:06:02.035Z","dependency_job_id":null,"html_url":"https://github.com/Engineering-Research-and-Development/opr4aa-platform","commit_stats":null,"previous_names":["engineering-research-and-development/opr4aa-platform"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Engineering-Research-and-Development/opr4aa-platform","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Engineering-Research-and-Development%2Fopr4aa-platform","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Engineering-Research-and-Development%2Fopr4aa-platform/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Engineering-Research-and-Development%2Fopr4aa-platform/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Engineering-Research-and-Development%2Fopr4aa-platform/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Engineering-Research-and-Development","download_url":"https://codeload.github.com/Engineering-Research-and-Development/opr4aa-platform/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Engineering-Research-and-Development%2Fopr4aa-platform/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31416770,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T20:09:54.854Z","status":"ssl_error","status_checked_at":"2026-04-04T20:09:44.350Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-27T21:25:00.693Z","updated_at":"2026-04-04T22:31:22.203Z","avatar_url":"https://github.com/Engineering-Research-and-Development.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1\u003eOn-The-Edge Processing Component for Acquisition and Actuation (OPR4AA)\u003c/h1\u003e\n\u003cimg align=\"right\" src=\"documentation/logo.png\" width=\"150\" alt=\"opr4aa logo\"\u003e\n\nOPR4AA stands for On-the-edge PRocessing for Acquisition and Actuation: it is a platform based on open source FIWARE/Apache components aiming to cover the entire industrial data value chain. Built on top of the \u003ca href=\"https://github.com/Engineering-Research-and-Development/dida\"\u003eDIDA\u003c/a\u003e (Digital Industries Data Analytics) platform, it can be deployed for edge processing, collecting data, process them and communicate results with other external cloud modules\n\nThe component allows you to bring input data call some external Rest API, process them using AI algorithms (e.g. Python Tensorflow + Keras) persist them in the internal persistence layer or export them to external modules through Rest API.\n\nThe OPR4AA platform is composed by:\n- \u003cb\u003eFIWARE Draco (deprecated)\u003c/b\u003e: based on Apache NiFi. NiFi is a data-flow system based on the flow-based concept programming designed to automate the flow of data in systems and support direct and scalable graphics.\n- \u003cb\u003eApache Airflow\u003c/b\u003e:  an open-source platform that allows users to programmatically author, schedule, and monitor workflows. It is designed to be highly scalable, extensible, and modular, making it ideal for creating complex data processing pipelines.\n- \u003cb\u003eApache Hadoop Distributed File System (HDFS)\u003c/b\u003e: designed to reliably store very large files across machines in a large cluster\n- \u003cb\u003eApache Spark\u003c/b\u003e: is an open-source parallel processing framework for running largescale data analytics applications across clustered computers. It can handle both batch and real-time analytics and data processing workloads.\n- \u003cb\u003eApache Livy\u003c/b\u003e:  is a service allowing easy interaction with a Spark cluster over a REST interface. Through it, can be easily submitted Spark jobs or snippets of Spark code, synchronous or asynchronous result retrieval, as well as Spark Context management, everything via a simple REST interface or an RPC client library. Apache Livy also simplifies the interaction between Spark and application servers.  \n\n\u003cbr\u003e\n\n\u003ch2\u003eRequirements\u003c/h2\u003e\n\u003cul\u003e\n    \u003cli\u003eDocker Engine\u003c/li\u003e\n    \u003cli\u003eMinimum 8GB RAM\u003c/li\u003e\n    \u003cli\u003eDocker Compose \u003e= 1.29\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003cbr\u003e\n\n\u003ch2\u003eHow to run\u003c/h2\u003e\n\u003ch3\u003eBuild \u0026 Run containers:\u003c/h3\u003e\n\u003cb\u003eBefore starting, chose wether to run AirFlow or NiFi as ingestion module. Comment/Uncomment docker-compose.yml accordingly.\u003c/b\u003e\n\n\u003ccode\u003edocker network create -d bridge network-bridge\u003c/code\u003e\n\n\u003ccode\u003edocker-compose up --build -d\u003c/code\u003e\n\n\u003ccode\u003edocker-compose -f airflow.yml up --build  -d\u003c/code\u003e\n\n\u003cbr\u003e\n\n\u003ch3\u003eAccess the UIs\u003c/h3\u003e\n\n1. Airflow at (https://localhost:8087/)\n1. NiFi at (https://localhost:8443/nifi)\n2. HDFS at (http://localhost:9870/explorer.html)\n3. Spark Master at (http://localhost:8080/)\n4. Spark Worker at (http://localhost:8081/)\n5. Livy UI at (http://localhost:8998/)\n\n\u003cbr\u003e\n\n\n\u003ch2\u003eConfiguration\u003c/h2\u003e\n\n\u003ch3\u003eUsing Airflow\u003c/h3\u003e\nIn Airflow you can configure your own data flow (see Airflow doc \u003ca  href=\"https://airflow.apache.org/docs\"\u003ehere\u003c/a\u003e) in a more lightweight way.\nThe solution provides processing on Spark. \n\n\u003ch4\u003eAirflow credentials\u003c/h4\u003e\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003eUser\u003c/th\u003e\n\u003cth\u003ePassword\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eairflow\u003c/td\u003e\n\u003ctd\u003eairflow\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n\u003ch4\u003eHow to run a DAG programmatically\u003c/h4\u003e\n\n```bash\ncurl --location --request POST 'http://localhost:8087/api/v1/dags/test-pipeline/dagRuns' \\\n    --header 'Authorization: Basic YWlyZmxvdzphaXJmbG93' \\\n    --header 'Content-Type: application/json' \\\n    --data-raw '{\n        \"conf\": {\n            \"host\": \"api.host.cloud\",\n            \"username\": \"*****\",\n            \"password\": \"*****\",\n            \"source_entity\": {\n                \"entity_id\": \"OPR4AA-Execution-Test\",\n                \"entity_type\": \"Entity-Type-Test\",\n                \"attribute\":\"image\"\n            },\n            \"sink_entity\": {\n                \"entity_id\": \"OPR4AA-Execution-Test\",\n                \"entity_type\": \"Entity-Type-Test\",\n                \"attribute\":\"evaluation\"\n            }\n        }\n    }'\n```\n\nBasic Authorization header is generated by base64 encoding of string \"airflow:airflow\"\n\nA [Postman collection](OPR4AA-V2.postman_collection.json) for DAG Run is provided.\n\n\n\u003cbr\u003e\n\n\u003ch3\u003eUsing Draco\u003c/h3\u003e\nIn Draco you can configure your own data flow (see NiFi doc \u003ca  href=\"https://nifi.apache.org/docs/nifi-docs/html/getting-started.html\"\u003ehere\u003c/a\u003e).\nThe solution provides processing on Draco or Spark. Algorithms \u0026 data ingestion can be done by calling the provided APIs.\nInside the template pre-loaded on Draco you can activate the flows you prefer to use and can configure each NiFi processor following the notes on the UI.\n\n\u003ch4\u003eDraco/NiFi credentials\u003c/h4\u003e\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003eUser\u003c/th\u003e\n\u003cth\u003ePassword\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003eadmin\u003c/td\u003e\n\u003ctd\u003ectsBtRBKHRAx69EqUghvvgEvjnaLjFEB\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\nDraco will start with pre-uploaded template:\n\n- [template.xml](orchestration/templates/template.xml)\n\n\n\u003cbr\u003e\n\n\u003ch4\u003eAPI (available only using Draco)\u003c/h4\u003e\n\n\u003ctable role=\"table\"\u003e\n    \u003cthead\u003e\n        \u003ctr align=\"center\"\u003e\n            \u003cth\u003eHTTP Method\u003c/th\u003e\n            \u003cth\u003ePort\u003c/th\u003e\n            \u003cth\u003eService\u003c/th\u003e\n            \u003cth\u003eDescription\u003c/th\u003e\n        \u003c/tr\u003e\n    \u003c/thead\u003e\n    \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003ePOST\u003c/td\u003e\n          \u003ctd\u003e8085\u003c/td\u003e\n          \u003ctd\u003e/ingest-algorithm\u003c/td\u003e\n          \u003ctd\u003eAlgorithm data ingestion route. Accept a .zip file that contains algorithm files.\u003c/td\u003e\n      \u003c/tr\u003e\n        \u003ctr\u003e\n          \u003ctd\u003ePOST\u003c/td\u003e\n          \u003ctd\u003e8086\u003c/td\u003e\n          \u003ctd\u003e/ingest-data\u003c/td\u003e\n          \u003ctd\u003eInput data ingestion route. Accept a file.\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\nA [Postman collection](OPR4AA.postman_collection.json) for algorithms \u0026 data ingestion is provided.\n\n\u003cbr\u003e\n\n\u003ch2\u003eTest AI classifier algorithm\u003c/h2\u003e \nThe solution is provided with an example you can run to test the solution.\nAn AI Python image classifier is provided, including a pre-trained ready-to-use neural network model based on Tensorflow and Keras (see \u003ca  href=\"https://www.tensorflow.org/tutorials/keras/classification\"\u003ehere\u003c/a\u003e).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fengineering-research-and-development%2Fopr4aa-platform","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fengineering-research-and-development%2Fopr4aa-platform","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fengineering-research-and-development%2Fopr4aa-platform/lists"}