{"id":15208915,"url":"https://github.com/o2-czech-republic/proxima-platform","last_synced_at":"2026-01-17T12:59:24.217Z","repository":{"id":26446939,"uuid":"108869877","full_name":"O2-Czech-Republic/proxima-platform","owner":"O2-Czech-Republic","description":"The Proxima platform.","archived":false,"fork":false,"pushed_at":"2025-07-10T09:25:15.000Z","size":9875,"stargazers_count":21,"open_issues_count":52,"forks_count":7,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-07-10T17:05:21.452Z","etag":null,"topics":["analytical-platform","apache-beam","apache-flink","apache-spark","batch-processing","data-mesh","iot-platform","stream-processing","unified-data-processing"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/O2-Czech-Republic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-10-30T15:26:01.000Z","updated_at":"2025-07-10T09:25:19.000Z","dependencies_parsed_at":"2023-02-13T20:30:56.372Z","dependency_job_id":"5310a7bc-e849-4c2c-9bb1-199c2c3a6539","html_url":"https://github.com/O2-Czech-Republic/proxima-platform","commit_stats":{"total_commits":1590,"total_committers":12,"mean_commits":132.5,"dds":"0.15094339622641506","last_synced_commit":"f0385606fa7ec6683a3f9290e16191dec12d0d76"},"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"purl":"pkg:github/O2-Czech-Republic/proxima-platform","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/O2-Czech-Republic%2Fproxima-platform","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/O2-Czech-Republic%2Fproxima-platform/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/O2-Czech-Republic%2Fproxima-platform/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/O2-Czech-Republic%2Fproxima-platform/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/O2-Czech-Republic","download_url":"https://codeload.github.com/O2-Czech-Republic/proxima-platform/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/O2-Czech-Republic%2Fproxima-platform/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265711402,"owners_count":23815530,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytical-platform","apache-beam","apache-flink","apache-spark","batch-processing","data-mesh","iot-platform","stream-processing","unified-data-processing"],"created_at":"2024-09-28T07:04:42.497Z","updated_at":"2026-01-17T12:59:24.180Z","avatar_url":"https://github.com/O2-Czech-Republic.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://github.com/datadrivencz/proxima-platform/actions/workflows/build.yml/badge.svg?branch=master)](https://github.com/datadrivencz/proxima-platform/actions/workflows/build.yml)\n[![sonar](https://sonarcloud.io/api/project_badges/measure?project=cz.o2.proxima%3Aplatform-parent\u0026metric=coverage)](https://sonarcloud.io/dashboard?id=cz.o2.proxima%3Aplatform-parent)\n[![sonar](https://sonarcloud.io/api/project_badges/measure?project=cz.o2.proxima%3Aplatform-parent\u0026metric=sqale_rating)](https://sonarcloud.io/dashboard?id=cz.o2.proxima%3Aplatform-parent)\n[![Maven Version](https://maven-badges.herokuapp.com/maven-central/cz.o2.proxima/proxima-core/badge.svg)](http://search.maven.org/#search|gav|1|g:\"cz.o2.proxima\")\n\n# The Proxima platform\n\nThe platform is a generic data ingestion, manipulation and retrieval framework.\nHigh level can be described by following scheme:\n\n![high-level scheme](docs/design-platform.png)\n\n## Design document\nHigh level design document can be found [here](https://docs.google.com/document/d/1s6UFrEaFldjuGhLX9IosTnfNcMhMpvGjCWfI_5Lqqp4/edit?usp=sharing).\n\n## Incomplete (under construction) documentation\nNot yet complete documentation can be found [here](https://datadrivencz.github.io/proxima-platform/book/). The documentation should grow over over time to cover all the aspects of the platform. PRs welcome!\n\n## Scheme definition\nFirst, let's introduce some glossary:\n * _entity_: a named dictionary consisting of string **key** and one or more _attributes_\n * _attribute_: an atomic field of entity with string **name** and **scheme** definining its data-type\n * _attribute family_: a logical grouping of attributes of the same entity into a named group\n * _storage_: a physical store for data\n\n### Example scheme definition\n The scheme definition uses [HOCON](https://github.com/typesafehub/config/blob/master/HOCON.md).\n As a short example we will show definition of data processing of a hypothetic e-commerce site. The site has some goods,\n some users and generates some events which describe how users interact with the goods. We will use\n [protocol buffers](https://developers.google.com/protocol-buffers/) for serialization.\n\n First, let's define our data model. We will model the system which processes events coming from some source in given\n format and based on these events creates a model of user preferences.\n ```\n entities {\n   # user entity, let's make this really simple\n   user {\n     attributes {\n\n       # some details of user - e.g. name, email, ...\n       details { scheme: \"proto:cz.o2.proxima.example.Example.UserDetails\" }\n\n       # model of preferences based on events\n       preferences { scheme: \"proto:cz.o2.proxima.example.Example.UserPreferences\" }\n\n       # selected events are stored to user's history\n       \"event.*\" { scheme: \"proto:cz.o2.proxima.example.Example.BaseEvent\" }\n\n     }\n   }\n   # entity describing a single good we want to sell\n   product {\n     # note: we have to split to separate attributes each attribute that we want to be able\n     # to update *independently*\n     attributes {\n\n       # price, with some possible additional information, like VAT and other stuff\n       price { scheme: \"proto:cz.o2.proxima.example.Example.Price\" }\n\n       # some general details of the product\n       details { scheme: \"proto:cz.o2.proxima.example.Example.ProductDetails\" }\n\n       # list of associated categories\n       \"category.*\" { scheme: \"proto:cz.o2.proxima.example.Example.ProductCategory\" }\n\n     }\n   }\n\n   # the events which link users to goods\n   event {\n     attributes {\n\n       # the event is atomic entity with just a single attribute\n       data { scheme: \"proto:cz.o2.proxima.example.Example.BaseEvent\" }\n\n     }\n   }\n\n }\n ```\n Next, after defining our data model, we need to specify attribute families for our entities. This definition\n is highly dependent on the **access pattern** to the data. Mostly, we have to worry about how are we going to\n read our data. Relevant questions are:\n  * are we going to need a random access (get or list request) for data by entity key and attribute name?\n  * are we going to be reading the data as continuously updated stream?\n  * do we want to be able to read all historical updates, or are we interested only in the last updated value for each attribute?\n  * are we going to process the data in batch fashion to build some sort of model?\n\n Let's describe our intentions as follows:\n  * we need to be able to batch reprocess all events (maybe limited by some global time window, say two years back), in order to build a model that will be used to update user's preferences with incoming events\n  * we need random acccess to data stored per user and per product\n  * we need access to stream of events to be able to do real-time updates to user preferences\n  * we want to be able to select some events to be stored in user's history and then list this history by time from newest to oldest\n\n  To be able to fulfill these requirements, we have chosen the following storages:\n   * [Apache Kafka](https://kafka.apache.org/) (for streaming)\n   * [Apache Cassandra](http://cassandra.apache.org/) (for random access to data)\n   * [Apache Hadoop HDFS](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) (for batch procesing)\n\n   This will yield us the following setup for attribute families (some details are ommitted for simplicity):\n   ```\n    attributeFamilies {\n\n       # we need this to be able to read user attributes 'details' and 'preferences' by user's key\n       user-random-access {\n         entity: user\n         attributes: [ \"details\", \"preferences\" ]\n         storage: \"cassandra://\"${cassandra.seed}/${cassandra.user-table}\"?primary=user\"\n         type: primary\n         access: random-access\n       }\n\n       # store incoming events to user's history\n       user-event-history-store {\n         entity: event\n         attributes: [ \"data\" ]\n         storage: \"cassandra://\"${cassandra.seed}/${cassandra.user-event-table}/\n         # this class defines how we transform incoming event to CQL\n         cqlFactory: cz.o2.proxima.example.EventHistoryCqlFactory\n         # this is filtering condition, we want to select only some events\n         filter: cz.o2.proxima.example.EventHistoryFilter\n         type: replica\n         access: write-only\n       }\n\n       # this family defines read access to the stored event history\n       user-event-history-read {\n         entity: user\n         attributes: [ \"event.*\" ]\n         storage: \"cassandra://\"${cassandra.seed}/${cassandra.user-event-table}\"?primary=user\u0026secondary=stamp\u0026data=event\u0026reversed=true\"\n         # ignore this for now\n         converter: cz.o2.proxima.core.storage.cassandra.DateToLongConverter\n         type: replica\n         # we will not explicitly modify this, it will be updated automatically by incoming events\n         access: read-only\n       }\n\n       # random access to products\n       product-random-acesss {\n         entity: product\n         attributes: [ \"*\" ]\n         storage: \"cassandra://\"${cassandra.seed}/${cassandra.product-table}\n         type: primary\n         access: [ random-access, batch-snapshot ]\n       }\n\n       # event stream storage\n       event-commit-log {\n         entity: event\n         attributes: [ \"*\" ]\n         storage: \"kafka://\"${kafka.brokers}/${kafka.events-topic}\n         # this is our commit log\n         type: primary\n         access: commit-log\n       }\n       # store events for batch analytics\n       event-batch-storage {\n         entity: event\n         attributes: [ \"*\" ]\n         storage: \"hdfs://\"${hdfs.authority}/${hdfs.event-path}\n         type: replica\n         access: batch-updates\n       }\n\n     }\n\n     cassandra {\n       seed = \"cassandra:9042\"\n       user-table = \"user\"\n       product-table = \"product\"\n       user-event-table = \"user_event\"\n     }\n\n     kafka {\n       brokers = \"kafka1:9092,kafka2:9092,kafka3:9092\"\n       events-topic = \"events\"\n     }\n\n     hdfs {\n       authority = \"hdfs-master\"\n       event-path = \"/events\"\n     }\n\n   ```\n   By this definition, we have (somewhat simplified) working description of Proxima platform scheme for data manipulation,\n   that can be fed into the ingestion/retrieval service and will start working as described above.\n\n## Platform's data model\n Generally, data are modelled as unbounded stream of _updates_ to attributes of entities. Each update consists of the following:\n  * name of entity\n  * name of attribute\n  * value of attribute (or flag representing _delete_)\n  * timestamp of the update\n  * UUID of the update\n\n Each stream can then be represented as a _table_ (a.k.a [table-stream duality](https://docs.confluent.io/current/streams/concepts.html#duality-of-streams-and-tables)), which is essentially a snapshot of a stream at a certain time (in terms of Proxima platform called _batch snapshot_).\n\n## Compiling scheme definition to access classes\n The platform contains maven compiler of scheme specification to java access classes as follows:\n ```xml\n      \u003cplugin\u003e\n        \u003cgroupId\u003ecz.o2.proxima\u003c/groupId\u003e\n        \u003cartifactId\u003eproxima-compiler-java-maven-plugin\u003c/artifactId\u003e\n        \u003cversion\u003e0.15.0\u003c/version\u003e\n        \u003cconfiguration\u003e\n          \u003coutputDir\u003e${project.build.directory}/generated-sources/model\u003c/outputDir\u003e\n          \u003cjavaPackage\u003ecz.o2.proxima.testing.model\u003c/javaPackage\u003e\n          \u003cclassName\u003eModel\u003c/className\u003e\n          \u003cconfig\u003e${basedir}/src/main/resources/test-readme.conf\u003c/config\u003e\n        \u003c/configuration\u003e\n        \u003cexecutions\u003e\n          \u003cexecution\u003e\n            \u003cphase\u003egenerate-sources\u003c/phase\u003e\n            \u003cgoals\u003e\n              \u003cgoal\u003ecompile\u003c/goal\u003e\n            \u003c/goals\u003e\n          \u003c/execution\u003e\n        \u003c/executions\u003e\n        \u003cdependencies\u003e\n          \u003c!--\n            Use direct data operator access, see later\n          --\u003e\n          \u003cdependency\u003e\n            \u003cgroupId\u003e${project.groupId}\u003c/groupId\u003e\n            \u003cartifactId\u003eproxima-direct-compiler-plugin\u003c/artifactId\u003e\n            \u003cversion\u003e0.15.0\u003c/version\u003e\n          \u003c/dependency\u003e\n          \u003c!--\n            The following dependencies define additional\n            dependencies for this example\n          --\u003e\n          \u003cdependency\u003e\n            \u003cgroupId\u003e${project.groupId}\u003c/groupId\u003e\n            \u003cartifactId\u003eproxima-core\u003c/artifactId\u003e\n            \u003cversion\u003e${project.version}\u003c/version\u003e\n            \u003cclassifier\u003etests\u003c/classifier\u003e\n          \u003c/dependency\u003e\n          \u003cdependency\u003e\n            \u003cgroupId\u003e${project.groupId}\u003c/groupId\u003e\n            \u003cartifactId\u003eproxima-scheme-proto\u003c/artifactId\u003e\n            \u003cversion\u003e0.15.0\u003c/version\u003e\n          \u003c/dependency\u003e\n          \u003cdependency\u003e\n            \u003cgroupId\u003e${project.groupId}\u003c/groupId\u003e\n            \u003cartifactId\u003eproxima-scheme-proto-testing\u003c/artifactId\u003e\n            \u003cversion\u003e0.15.0\u003c/version\u003e\n          \u003c/dependency\u003e\n        \u003c/dependencies\u003e\n      \u003c/plugin\u003e\n ```\n\n This plugin then generates class `cz.o2.proxima.testing.model.Model` into `target/generated-sources/model`.\n The class can be instantiated via\n ```java\n   Model model = Model.of(ConfigFactory.defaultApplication());\n  ```\n  or (in case of tests, where some validations and initializations are skipped)\n  ```\n   Model model = Model.ofTest(ConfigFactory.defaultApplication());\n  ```\n\n## Platform's _DataOperators_\n The platform offers various modes of access to data. As of version 0.15.0, these types are:\n  * direct\n  * Apache Beam\n  * Apache Flink\n### Direct access to data\n This operator is used when accessing data from inside single JVM (or potentially multiple JVMs, e.g. coordinated via distributed consumption of commit log). The operator is constructed as follows:\n ```java\n    private DirectDataOperator createDataOperator(Model model) {\n      Repository repo = model.getRepo();\n      return repo.getOrCreateOperator(DirectDataOperator.class);\n    }\n ```\n Next, we can use the operator to create instances of data accessors, namely:\n  * CommitLogReader\n  * BatchLogReader\n  * RandomAccessReader\n\n For instance, observing commit log can be done by\n ```java\n    DirectDataOperator operator = model.getRepo().getOrCreateOperator(DirectDataOperator.class);\n    CommitLogReader commitLog = operator.getCommitLogReader(\n        model.getEvent().getDataDescriptor())\n        .orElseThrow(() -\u003e new IllegalArgumentException(\"Missing commit log for \"\n            + model.getEvent().getDataDescriptor()));\n    commitLog.observe(\"MyObservationProcess\", new LogObserver() {\n\n      @Override\n      public boolean onError(Throwable error) {\n        throw new RuntimeException(error);\n      }\n\n      @Override\n      public boolean onNext(StreamElement elem, OnNextContext context) {\n        log.info(\"Consumed element {}\", elem);\n        // commit processing, so that it is not redelivered\n        context.confirm();\n        // continue processing\n        return true;\n      }\n\n    });\n ```\n Creating BatchLogReader or RandomAccessReader is analogous.\n\n### [Apache Beam](https://beam.apache.org/) access to data\n First, create BeamDataOperator as follows:\n ```java\n   BeamDataOperator operator = model.getRepo().getOrCreateOperator(BeamDataOperator.class);\n ```\n Next, use this operator to create PCollection from Model.\n ```java\n   // some imports omitted, including these for clarity\n   import org.apache.beam.sdk.Pipeline;\n   import org.apache.beam.sdk.transforms.Count;\n   import org.apache.beam.sdk.transforms.WithKeys;\n   import org.apache.beam.sdk.transforms.windowing.AfterWatermark;\n   import org.apache.beam.sdk.transforms.windowing.FixedWindows;\n   import org.apache.beam.sdk.transforms.windowing.Window;\n   import org.apache.beam.sdk.values.KV;\n   import org.apache.beam.sdk.values.PCollection;\n   import org.joda.time.Duration;\n\n   Pipeline pipeline = Pipeline.create();\n   PCollection\u003cStreamElement\u003e input = operator.getStream(\n       pipeline, Position.OLDEST, false, true,\n       model.getEvent().getDataDescriptor());\n   PCollection\u003cKV\u003cString, Long\u003e\u003e counted =\n       input\n           .apply(\n               Window.\u003cStreamElement\u003einto(FixedWindows.of(Duration.standardMinutes(1)))\n                   .triggering(AfterWatermark.pastEndOfWindow())\n                   .discardingFiredPanes())\n           .apply(\n               WithKeys.of(\n                   el -\u003e\n                       model\n                           .getEvent()\n                           .getDataDescriptor()\n                           .valueOf(el)\n                           .map(BaseEvent::getProductId)\n                           .orElse(\"\")))\n           .apply(Count.perKey());\n\n   // do something with the output\n ```\n\n## Online Java docs\n  * [Latest](https://datadrivencz.github.io/proxima-platform/apidocs/)\n\n## Build notes\n CI is run only against changed modules (and its dependents) in pull requests. To completely rebuild the whole project in a PR push a commit with commit message 'rebuild'. After the build, you can squash and remove the commit.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fo2-czech-republic%2Fproxima-platform","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fo2-czech-republic%2Fproxima-platform","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fo2-czech-republic%2Fproxima-platform/lists"}