{"id":27968581,"url":"https://github.com/ging/fiware-usage-control","last_synced_at":"2025-05-07T21:04:59.942Z","repository":{"id":33878596,"uuid":"156740487","full_name":"ging/fiware-usage-control","owner":"ging","description":"Ensuring data usage control on real-time analytics in the FIWARE context","archived":false,"fork":false,"pushed_at":"2024-12-25T07:23:18.000Z","size":4409,"stargazers_count":5,"open_issues_count":82,"forks_count":2,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-05-07T21:04:53.959Z","etag":null,"topics":["access-control","data-sovereignty","data-usage","fiware","odrl","policy-enforcement","punishments","usage-control"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ging.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/roadmap.md","authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-08T17:01:36.000Z","updated_at":"2024-08-21T22:17:31.000Z","dependencies_parsed_at":"2023-02-15T04:31:37.761Z","dependency_job_id":"443a519e-6d49-43a3-82f1-edabd7509e13","html_url":"https://github.com/ging/fiware-usage-control","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ging%2Ffiware-usage-control","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ging%2Ffiware-usage-control/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ging%2Ffiware-usage-control/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ging%2Ffiware-usage-control/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ging","download_url":"https://codeload.github.com/ging/fiware-usage-control/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252954432,"owners_count":21830903,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["access-control","data-sovereignty","data-usage","fiware","odrl","policy-enforcement","punishments","usage-control"],"created_at":"2025-05-07T21:04:59.464Z","updated_at":"2025-05-07T21:04:59.931Z","avatar_url":"https://github.com/ging.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FIWARE Usage Control\n\n\n[![FIWARE Security](https://nexus.lab.fiware.org/repository/raw/public/badges/chapters/security.svg)](https://www.fiware.org/developers/catalogue/)\n![License](https://img.shields.io/github/license/ging/fiware-usage-control.svg)\n[![](https://img.shields.io/badge/tag-fiware-orange.svg?logo=stackoverflow)](http://stackoverflow.com/questions/tagged/fiware)\n\u003cbr/\u003e\n\u003c!--[![Known Vulnerabilities](https://snyk.io/test/github/ging/fiware-usage-control/badge.svg?targetFile=pom.xml)](https://snyk.io/test/github/ging/fiware-usage-control?targetFile=pom.xml)--\u003e\n\nUsage control is a promising approach for access control in open, distributed, heterogeneous and network-connected computer environments. \nIt encompasses and enhances traditional access control models, Trust Management (TM) and Digital Rights Management (DRM), and its main novelties are mutability of attributes and continuity of access decision evaluation.\n\nUsage control encompasses Data Access control and Data Usage Control. A good representation of this concepts is shown in the next figure:\n\n![usage-control-concept](docs/images/usage-concept.png)\n\n**Data Access Control:**\n * Specifies who can access what resource\n * Also the rights to access it (actions)\n\n**Data Usage Control:**\n * Ensures data sovereignty\n * Regulates what is allowed to happen with data (future use).\n * Related with data ingestion and processing\n * Context of intellectual property protection, privacy protection, compliance with regulations and digital rights management\n\nThis repository includes a set of components and operations for providing usage control capabilities over data coming from the Orion Context Broker, processed by a data streaming processing engine (Apache Flink) through the [FIWARE Cosmos Orion Flink Connector](https://github.com/ging/fiware-cosmos-orion-flink-connector). \nFirst, the architecture and scenario are presented, followed by the instructions and resources of how you can replicate the use case presented.\n\n\n## Architecture\n\nThe next figure presents an abstract representation of the proposed architecture for usage control.\nA general overview of the architecture is presented in the next figure. \nThis scheme is derived from a hybrid model based on the *[Data Privacy Directive 95/46/EC](https://eur-lex.europa.eu/legal-content/en/TXT/?uri=CELEX%3A31995L0046)* and the *[IDS reference architecture](https://www.fraunhofer.de/content/dam/zv/de/Forschungsfelder/industrial-data-space/IDS_Referenz_Architecture.pdf)* \nand it is divided in three essential parts: Data Provider, Data Consumer and Data Controller.\n\n### Three stakeholders\n![usage-architecture-1](docs/images/usage-architecture-1.png)\n\n### Two stakeholders \nIn some cases, the Data Provider and Data Controller can be integrated in a single stakeholder inside the architecture. This is represented in the next figure:\n\n![usage-architecture-2](docs/images/usage-architecture-2.png)\n\nThe different components that make up this architecture are described in detail below:\n\n**Data Consumer:**\n\n * **Apache Flink Cluster**: Big Data Processing Engine in which client jobs are run. The data consumer may write real-time data processing jobs using Flink for Scala and the [FIWARE Cosmos Orion Flink Connector](https://github.com/ging/fiware-cosmos-orion-flink-connector) in order to have a direct ingestion of data from Orion in the processing engine. \n\n**Data Provider/Controller:**\n\n * **Orion Context Broker**: Component that allows to manage the entire lifecycle of context information\n * **IdM Keyrock**: Component for defining Access and Usage Control Policies\n * **PEP (Policy Enforcement Point) proxy**: Component for enforcing Access Control Policies\n * **PTP (Policy Translation Point)**: Component for translating the FI-ODRL Policies into a program that checks compliance in real time\n * **PXP/PDP (Policy Execution/Decision Point)**: Component with complex event processing capabilities (CEP) for analyzing the logs in order to verify the compliance of the obligations defined in the IDM and enforce the punishments\n \n \n## Example use case: Supermarket Store\n\nA fully working scenario is provided in this repository, which can be easily modified in order to fit a different use case.\n\n### Use case description\n\nThe use case proposed is based on a supermarket scenario. It consists on a series of stores that post data from each purchase to Orion. The data provider would like to make these data available to customers, only if their use of these data complies with a series of policies that both parties have agreed upon.\n\n#### Data definition\nThe data involved in this scenario is represented by a Ticket Entity available in Orion. This entity contains purchase information on the specific store (`mall`), the `client`, the `date`, and a list with the purchased `items`. \nEach of these items includes the number of units purchased (`net_am`), the unit price (`net_am`) and the description of the item (`desc`).\nA sample entity is presented below:\n\n```json\n{\n  \"id\":\"ticket\",\n  \"type\":\"ticket\",\n  \"attrs\":{\n    \"_id\":{\n      \"type\":\"String\",\n      \"value\":1027,\n      \"metadata\":{}\n    },\n    \"items\":{\n      \"type\":\"object\",\n      \"value\":[{\n        \"net_am\":3.9,\n        \"n_unit\":6,\n        \"desc\":\"GOURMET  85GR\"\n      }],\n    \"metadata\":{}\n  },\n  \"mall\":{\n    \"type\":\"String\",\n    \"value\":2,\n    \"metadata\":{}\n  },\n  \"date\":{\n    \"type\":\"date\",\n    \"value\":\"01/14/2016\",\n    \"metadata\":{}\n  },\n  \"client\":{\n    \"type\":\"int\",\n    \"value\":77021708271,\n    \"metadata\":{}\n    }\n  }\n}\n```\n\n\n\n#### Defining policies\n\nThe policies that the data provider wants to enforce on the data are the following:\n\n * The user shall **NOT** save the data without aggregating them every 15 seconds first or else the processing job will be terminated\n \n * The user shall **NOT** receive more than 200 notifications from Orion in a minute or else the subscription to the entity will be deleted\n \n \nThe data provider has to define these policies using the web interface that KeyRock provides. \n\n![Keyrock creating policies](docs/images/usage-idm.png)\n\n\nWhen the data provider creates these policies in KeyRock and applies them to a certain user, KeyRock translates them into the FI-ODRL language.\n ```xml\n // TODO\n ```\n \nKeyRock notifies the PTP that a new policy has to be enforced. A CEP program is generated from the FI-ODRL policy definition through an extended automata.\nThe policies defined in this example would turn into the following CEP code excerpt:\n\n```scala\n// First pattern: At least N events in T.    \nval countPattern2 = Pattern.begin[Entity](\"events\" )\n    .timesOrMore(200).within(Time.seconds(15))\n      \nCEP.pattern(entityStream, countPattern2).select(events =\u003e\n    Signals.createAlert(Policy.COUNT_POLICY, events, Punishment.UNSUBSCRIBE))\n\n// Second pattern: Source -\u003e Sink. Aggregation TimeWindow\nval aggregatePattern = Pattern.begin[ExecutionGraph](\"start\",    \n    AfterMatchSkipStrategy.skipPastLastEvent())\n      .where(Policies.executionGraphChecker(_, \"source\"))\n      .notFollowedBy(\"middle\").where(Policies.executionGraphChecker(_, \"aggregation\", 15000))\n      .followedBy(\"end\").where(Policies.executionGraphChecker(_, \"sink\")).timesOrMore(1)\n\nCEP.pattern(operationStream, aggregatePattern).select(events =\u003e\n   Signals.createAlert(Policy.AGGREGATION_POLICY, events, Punishment.KILL_JOB))\n\n```\n\nThe generated CEP program is deployed and receives the logs from the user processing engine:\n * **Execution Graph Logs**: Chain of operations performed by the data user\n * **Event Logs**: NGSI Events received by the data user coming from Orion\n \n#### The data user program\n\nThe data user wants to extract value in real-time from the data received. Specifically, he/she is interested in knowing what the average purchase price is for the supermarket clients every hour. \nIn order to achieve this, he/she may write a job like such:\n\n```scala\nval env = StreamExecutionEnvironment.getExecutionEnvironment\n\n// Create Orion Source. Receive notifications on port 9001\nval eventStream = env.addSource(new OrionSource(9001))\n\n// Process event stream\nval processedDataStream = eventStream\n  .flatMap(event =\u003e event.entities)\n  .map(entity =\u003e {\n    val id = entity.attrs(\"_id\").value.toString\n    val items =   entity.attrs(\"items\").value.asInstanceOf[List[Map[String,Any]]]\n        items.map(product =\u003e {\n          val productName = product(\"desc\").asInstanceOf[String]\n          val unitPrice =  product(\"net_am\").asInstanceOf[Number].floatValue()\n          val unitNumber = product(\"n_unit\").asInstanceOf[Number].floatValue()\n          SupermarketProduct(id, productName, unitPrice * unitNumber)\n        })\n      })\n      .map(_.map(_.price).sum)\n      .timeWindowAll(Time.minutes(60))\n      .aggregate(new AverageAggregate)\n    // Print the results with a single thread, rather than in parallel\n    processedDataStream.print().setParallelism(1)\n    env.execute(\"Supermarket Job\")\n\n```\n\nThe Flink job must be compiled into a JAR file. Maven will download all the necessary dependencies to build the JAR, except for the Cosmos connector. You need to [download](https://github.com/ging/fiware-cosmos-orion-flink-connector/releases/latest) it and install it manually:\n\n```\nmvn install:install-file -Dfile=$(PATH_DOWNLOAD)/orion.flink.connector-1.2.2.jar -DgroupId=org.fiware.cosmos -DartifactId=orion.flink.connector -Dversion=1.2.2 -Dpackaging=jar\n```\nOnce compiled, the job can be deployed on the Flink Client Cluster using the provided web UI. As soon as the job is deployed, the Execution Graph logs and the NGSI Event logs start to be sent to the PDP/PXP, who verifies that policies are being complied with an enforces punishments if they are not.\n  \n\n#### Monitoring policy enforcement\n\nThe data provider has to be aware of when data consumers are not complying with the established policies. \nFor this task, a control panel is provided in which all the events regarding policies can be checked in real-time, as well as a series of statistics on data usage.\n\n![FIWARE Data Usage Control Panel](docs/images/usage-panel.png)\n\n### Deployment\n#### Agents involved\nThe scenario presented in this repository is composed by a series of building blocks which can be easily replicated using the provided docker-compose file. It consists of the following containers:\n\n**Data Consumer:**\n\n * An **Apache Flink** Cluster (1 Job Manager and 1 Task Manager) \n * A **Streaming Job** for making the aggregations and operations of some values of a notified Entity created in the Orion Context Broker\n\n**Data Provider/Controller:**\n \n * One **Orion** (with MongoDB) instance \n * One **IdM Keyrock** instance\n * One **PEP proxy** instance \n * One **PTP (Policy Translation Point)** instance\n * One **Data Usage Control Panel** web application instance for monitoring the usage control rules and punishments in real-time\n * One **PXP/PDP (Policy Execution/Decision Point)** instance based on Apache Flink \n * One container with a **supermarket tickets database** posting data to the Orion Context Broker\n\n![usage-scenario](docs/images/usage-scenario.png) \n\n\nFor deploying and running this scenario you need to have docker and docker-compose\n\n1. Clone the repository\n```bash\ngit clone https://github.com/ging/fiware-usage-control.git\n```\n2. Access the root directory\n```bash\ncd fiware-usage-control\n```\n\n#### Data Provider/Controller\nFor deploying the Data Usage Control components of the Data Provider-Controller side run containers \ndefined in the `docker-compose.yml` file with their respective ENV variables\n\n3. Run containers\n```bash\nsudo docker-compose up -d\n```\n4. Check if all the containers are running\n```bash\nsudo docker ps\n```\n6. Check the orion entities\n```bash\ncurl localhost:1026/v2/entities -s -S --header 'Accept: application/json' | python -mjson.tool\n```\n#### Data Consumer\n\nNow, for deploying the component on the Data Consumer side, follow the next steps:\n\n1. Go to the `flink` folder\n```bash\ncd flink\n```\n2. Deploy the Flink Cluster\n```bash\nsudo docker-compose up -d\n```\n3. Check if all the containers are running\n```bash\nsudo docker ps\n```\n\n\nOnce you have everything up and running, you can go on to follow the demo video for the next steps.\n\n**[Demo Video](https://drive.google.com/file/d/1o_4KPLG026xG67lXitQeAj98rbZjCGx7/view?usp=sharing)**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fging%2Ffiware-usage-control","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fging%2Ffiware-usage-control","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fging%2Ffiware-usage-control/lists"}