{"id":19565753,"url":"https://github.com/pravega/pravega-samples","last_synced_at":"2025-04-27T01:31:14.092Z","repository":{"id":20775875,"uuid":"82957400","full_name":"pravega/pravega-samples","owner":"pravega","description":"Sample Applications for Pravega. ","archived":false,"fork":false,"pushed_at":"2023-11-15T08:12:22.000Z","size":54367,"stargazers_count":54,"open_issues_count":20,"forks_count":61,"subscribers_count":24,"default_branch":"master","last_synced_at":"2024-04-18T17:13:18.492Z","etag":null,"topics":["data-streaming","pravega","sample-app","streaming-data"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pravega.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-02-23T18:17:08.000Z","updated_at":"2024-01-08T17:47:11.000Z","dependencies_parsed_at":"2022-08-27T12:50:46.001Z","dependency_job_id":null,"html_url":"https://github.com/pravega/pravega-samples","commit_stats":null,"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pravega%2Fpravega-samples","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pravega%2Fpravega-samples/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pravega%2Fpravega-samples/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pravega%2Fpravega-samples/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pravega","download_url":"https://codeload.github.com/pravega/pravega-samples/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224053274,"owners_count":17247858,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-streaming","pravega","sample-app","streaming-data"],"created_at":"2024-11-11T05:28:19.384Z","updated_at":"2024-11-11T05:28:20.611Z","avatar_url":"https://github.com/pravega.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pravega and Analytics Connectors Examples\n\nThis repository contains code samples to demonstrate how developers can work with \n[Pravega](http://pravega.io). We also provide code samples to connect analytics \nengines such as [Flink](https://flink.apache.org/) and\n[Spark](http://spark.apache.org/) with Pravega as a storage substrate for data \nstreams. \nWe also provide samples for using new pravega schema registry with pravega applications. \n\nFor more information on Pravega, we recommend to read the [documentation and the\ndeveloper guide](http://pravega.io).\n\n# Repository Structure\n\nThis repository is divided into sub-projects (`pravega-client-examples`, `flink-connector-examples`, \n`schema-registry-examples`, and `spark-connector-examples`), each one addressed to demonstrate a specific component. In these sub-projects, \nwe provide a battery of simple code examples aimed at illustrating how a particular \nfeature or API works. Moreover, we also include a `scenarios` folder that contains \nmore complex applications as sub-projects, which show use-cases exploiting one or multiple components.\n\n\u003e Hint: Have a look to the [terminology and concepts](http://pravega.io/docs/latest/terminology/) in Pravega.\n\n## Pravega Client Examples\n| Example Name  | Description  | Language |\n| ------------- |:-----| :-----|\n| `gettingstarted` | Simple example of how to read/write from/to a Pravega `Stream`. | [Java](pravega-client-examples/src/main/java/io/pravega/example/gettingstarted)\n| `consolerw` | Application that allows users to work with `Stream`, `Transaction` and `StreamCut` APIs via CLI. | [Java](pravega-client-examples/src/main/java/io/pravega/example/consolerw)\n| `noop` | Example of how to add a simple callback executed upon a read event. | [Java](pravega-client-examples/src/main/java/io/pravega/example/noop)\n| `statesynchronizer` | Application that allows users to work with `StateSynchronizer` API via CLI. | [Java](pravega-client-examples/src/main/java/io/pravega/example/statesynchronizer)\n| `streamcuts` | Application examples demonstrating the use of `StreamCut`s via CLI. | [Java](pravega-client-examples/src/main/java/io/pravega/example/streamcuts) \n| `tables` | Application examples demonstrating the use of `KeyValueTable`s via CLI. | [Java](pravega-client-examples/src/main/java/io/pravega/example/tables) \n\nThe related documentation and instructions are [here](pravega-client-examples).\n\n## Flink Connector Examples\n| Example Name  | Description  | Language |\n| ------------- |:-----| :-----|\n| `wordcount` | Counting the words continuously from a Pravega `Stream` to demonstrate the usage of Flink connector for Pravega. | [Java](flink-connector-examples/src/main/java/io/pravega/example/flink/wordcount)\n| `primer` | This sample demonstrates Pravega \"exactly-once\" feature jointly with Flink checkpointing and exactly-once mode. | [Java](flink-connector-examples/src/main/java/io/pravega/example/flink/primer)\n| `streamcuts` | This sample demonstrates the use of Pravega StreamCuts in Flink applications. | [Java](flink-connector-examples/src/main/java/io/pravega/example/flink/streamcuts)\n\nThe related documentation and instructions are [here](flink-connector-examples).\n\n## Scenarios\n| Example Name                                                                           | Description                                                                                                                                                                                                                         | Language |\n|----------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :-----|\n| [`turbineheatsensor`](scenarios/turbine-heat-sensor)                                   | It emulates parallel sensors producing temperature values (writers) and parallel consumers performing real-time statistics (readers) via Pravega client.                                                                            | [Java](scenarios/turbine-heat-sensor/src/main/java/io/pravega/turbineheatsensor)\n| [`turbineheatprocessor`](scenarios/turbine-heat-processor)                             | A Flink streaming application for processing temperature data from a Pravega stream produced by the `turbineheatsensor` app. The application computes a daily summary of the temperature range observed on that day by each sensor. | [Java](scenarios/turbine-heat-processor/src/main/java/io/pravega/turbineheatprocessor), [Scala](scenarios/turbine-heat-processor/src/main/scala/io/pravega/turbineheatprocessor)\n| [`anomaly-detection`](scenarios/anomaly-detection)                                     | A Flink streaming application for detecting anomalous input patterns using a finite-state machine.                                                                                                                                  | [Java](scenarios/anomaly-detection/src/main/java/io/pravega/anomalydetection)\n| [`pravega-flink-connector-sql-samples`](scenarios/pravega-flink-connector-sql-samples) | Flink connector table api/sql samples.                                                                                                                                                                                              | [Java](scenarios/pravega-flink-connector-sql-samples/src/main/java/io/pravega/connectors.nytaxi)\n| [`pravega-flink-autoscaling`](scenarios/pravega-flink-autoscaling) | Demonstrates coordinated auto-scaling of Pravega streams and Flink task managers. This sample is build independently in its own folder.                                                                                             | [Java](scenarios/pravega-flink-autoscaling/)\n\n## Schema Registry Examples\nThe prerequisite for running Schema Registry Examples is to deploy Pravega and Schema Registry Service. For instructions to run pravga schema registry, please see instructions [here](https://github.com/pravega/schema-registry) \n\n| Example Name  | Description  | Language |\n| ------------- |:-----| :-----|\n| `Avro` | Samples for registering schema in avro format with registry service. Samples demonstrate how to use avro schemas and serializers for writing and reading data from pravega streams. | [Java](schema-registry-examples/src/main/java/io/pravega/schemaregistry/samples/avro)\n| `Protobuf` | Samples for registering schema in protobuf format with registry service. Samples demonstrate how to use protobuf schemas and serializers for writing and reading data from pravega streams. | [Java](schema-registry-examples/src/main/java/io/pravega/schemaregistry/samples/protobuf)\n| `Json` | Samples for registering schema in json format with registry service. Samples demonstrate how to use json schemas and serializers for writing and reading data from pravega streams. | [Java](schema-registry-examples/src/main/java/io/pravega/schemaregistry/samples/json)\n| `Multiple Formats` | Samples that demonstrate how to serialize data in different formats and write into same pravega stream. | [Java](schema-registry-examples/src/main/java/io/pravega/schemaregistry/samples/multiformatdemo)\n| `Codec` | Samples that demonstrate how to use additional codecs and share encoding information using schema registry service. This sample demonstrates using compression codecs for snappy and gzip. | [Java](schema-registry-examples/src/main/java/io/pravega/schemaregistry/samples/codec)\n\nThe related documentation and instructions are [here](schema-registry-examples).\n\n## Spark Connector Examples\n| Example Name  | Description  | Language |\n| ------------- |:-----| :-----|\n| `batch_file_to_pravega` | PySpark batch job that reads events from the file and writes to a Pravega stream | [Python](spark-connector-examples/src/main/python/batch_file_to_pravega.py)\n| `batch_pravega_to_console` | PySpark batch job that reads from a Pravega stream and writes to the console | [Python](spark-connector-examples/src/main/python/batch_pravega_to_console.py)\n| `stream_generated_data_to_pravega` | PySpark Streaming job that writes generated data to a Pravega stream | [Python](spark-connector-examples/src/main/python/stream_generated_data_to_pravega.py)\n| `stream_pravega_to_console` | PySpark Streaming job that reads from a Pravega stream and writes to the console | [Python](spark-connector-examples/src/main/python/stream_pravega_to_console.py)\n| `stream_bounded_pravega_to_console` | PySpark Streaming job that reads from a bounded Pravega stream and writes to the console | [Python](spark-connector-examples/src/main/python/stream_bounded_pravega_to_console.py)\n| `stream_pravega_to_pravega` | PySpark Streaming job that reads from a Pravega stream and writes to another Pravega stream | [Python](spark-connector-examples/src/main/python/stream_pravega_to_pravega.py)\n| `StreamPravegaToConsole` | Scala Spark Streaming job that reads from a Pravega stream and writes to the console | [Scala](spark-connector-examples/src/main/scala/io/pravega/example/spark/StreamPravegaToConsole.scala)\n| `StreamPravegaToPravega` | Scala Spark Streaming job that reads from a Pravega stream and writes to another Pravega stream | [Scala](spark-connector-examples/src/main/scala/io/pravega/example/spark/StreamPravegaToPravega.scala)\n\nThe related documentation and instructions are [here](spark-connector-examples).\n\n## Hadoop Connector Examples (_Retired: available up to Pravega Samples release 0.10_)\n| Example Name  | Description  | Language |\n| ------------- |:-----| :-----|\n| `wordcount` | Counts the words from a Pravega `Stream` filled with random text to demonstrate the usage of Hadoop connector for Pravega. | [Java](https://github.com/pravega/pravega-samples/tree/r0.10/hadoop-connector-examples/src/main/java/io/pravega/example/hadoop/wordcount)\n| `terasort` | Sort events from an input Pravega `Stream` and then write sorted events to one or more streams. | [Java](https://github.com/pravega/pravega-samples/tree/r0.10/hadoop-connector-examples/src/main/java/io/pravega/example/hadoop/terasort)\n\nThe related documentation and instructions are [here](https://github.com/pravega/pravega-samples/tree/r0.10/hadoop-connector-examples).\n\n# Build Instructions\n\nNext, we provide instructions for building the `pravega-samples` repository. There are two main options: \n- _Out-of-the-box_: If you want a quick start, run the samples by building `pravega-samples` out-of-the-box\n(go straight to section `Pravega Samples Build Instructions`). \n- _Build from source_: If you want to have fun building the different projects from source, please read\nsection `Building Pravega Components from Source (Optional)` before building `pravega-samples`. \n\n## Pre-requisites\n\n* Java 11\n* Python 3.8 (if you wish to run the python examples)\n\n## Building Pravega Components from Source (Optional)\n\n### Pravega Build Instructions \n\nIf you want to build Pravega from source, you may need to generate the latest Pravega `jar` files and install them to \nyour local Maven repository. To build Pravega from sources and use it here, please run the following commands:\n\n```\n$ git clone https://github.com/pravega/pravega.git\n$ cd pravega\n$ ./gradlew install\n```\n\nThe above command should generate the required `jar` files into your local Maven repository.\n\n\u003e Hint: For using in the sample applications the Pravega version you just built, you need to update the \n`pravegaVersion=\u003clocal_maven_pravega_version\u003e` property in `gradle.properties` file \nof `pravega-samples`.\n\nFor more information, please visit [Pravega](https://github.com/pravega/pravega).\n\n### Flink Connector Build Instructions\n\nTo build the Flink connector from source, follow the below steps to build and publish artifacts from \nsource to local Maven repository:\n\n```\n$ git clone --recursive https://github.com/pravega/flink-connectors.git\n$ cd flink-connectors\n$ ./gradlew install\n```\n\n\u003e Hint: For using in the sample applications the Flink connector version you just built, you need to update the \n`flinkConnectorVersion=\u003clocal_maven_flink_connector_version\u003e` property in `gradle.properties` file \nof `pravega-samples`.\n\n\nFor more information, please visit [Flink Connectors](https://github.com/pravega/flink-connectors). \n\n### Schema Registry Build Instructions\n\nSchema registry uses pravega, so make sure pravega is installed and running before installing schema registry. \nTo build Schema Registry from source, follow the below steps to build and publish artifacts from \nsource to local Maven repository:\n\n```\n$ git clone https://github.com/pravega/schema-registry.git\n$ cd schema-registry\n$ ./gradlew install\n$ cd server/build/install/schema-registry\n$ # edit conf/schema-registry.config.properties to point to pravega URI (hint: if you are running pravega standalone, it would be tcp://localhost:9090) \n$ ./bin/schema-registry\n```\n\nFor more information, please visit [Schema Registry](https://github.com/pravega/schema-registry). \n\n### Configuring Pravega Samples for Running with Source Builds\n\nIn the previous instructions, we noted that you will need to change the `gradle.properties` file in\n`pravega-samples` for using the Pravega components built from source. Here we provide an example of how to do so:\n\n1) Imagine that we want to build Pravega from source. Let us assume that we \nexecuted `git clone https://github.com/pravega/pravega.git` and the last commit of \n`master` branch is `2990193xxx`. \n\n2) After executing `./gradlew install`, we will see in our local Maven repository \n(e.g., `~/.m2/repository/io/pravega/*`) artifacts that contain in their names that commit version \nsuch as `0.3.0-1889.2990193-SNAPSHOT`. These artifacts are the result from building Pravega from source. \n\n3) The only thing you have to do is to set `pravegaVersion=0.3.0-1889.2990193-SNAPSHOT` in the `gradle.properties`\nfile of `pravega-samples`.\n\nWhile this example is for Pravega, the same procedure applies for Flink and Spark connectors.\n\n\n## Pravega Samples Build Instructions\n\nThe `pravega-samples` project is prepared for working out-of-the-box with \n[release artifacts](https://github.com/pravega/pravega/releases) of Pravega components, which are already \navailable in Maven central. To build `pravega-samples` from source, use the built-in gradle wrapper as follows:\n\n```\n$ git clone https://github.com/pravega/pravega-samples.git\n$ cd pravega-samples\n$ ./gradlew clean installDist\n```\nThat's it! You are good to go and execute the examples :) \n\nTo ease their execution, most examples can be run either using the gradle wrapper (gradlew) or scripts. \nThe above gradle command automatically creates the execution scripts that can be found under:\n\n```\npravega-samples/pravega-client-examples/build/install/pravega-client-examples/bin\n```\n\nThere is a Linux/Mac script and a Windows (.bat) script for each separate executable.\n\n_Working with `dev` branch_: If you are curious about the most recent sample applications, \nyou may like to try the `dev` version of `pravega-samples` as well. To do so, just clone the \n`dev` branch instead of `master` (default): \n\n```\n$ git clone -b dev https://github.com/pravega/pravega-samples.git\n$ cd pravega-samples\n$ ./gradlew clean installDist\n```\n\nThe `dev` branch works with Pravega snapshots artifacts published in \nour [JFrog repository](https://oss.jfrog.org/artifactory/jfrog-dependencies/io/pravega/) instead of \nusing release versions.\n\n\n# Proposed Roadmap\n\nWe propose a roadmap to proceed with the execution of examples based on their complexity:\n1. [Pravega client examples](pravega-client-examples): \nFirst step to understand the basics of Pravega and exercise the concepts presented in the documentation. \n2. [Flink connector examples](flink-connector-examples): \nThese examples show the basic functionality of the Flink connector for Pravega.\n3. [Spark connector examples](spark-connector-examples): \nThese examples show the basic functionality of the Spark connector for Pravega.\n4. [Scenarios](scenarios): Applications that go beyond the basic usage of Pravega APIs, which may include complex interactions \nbetween Pravega and analytics engines (e.g., Flink, Spark) to demonstrate analytics use cases.\n\n# Where to Find Help\n\nDocumentation on Pravega and Analytics Connectors:\n* [Pravega.io](http://pravega.io/), [Pravega Wiki](https://github.com/pravega/pravega/wiki).\n* [Flink Connectors Wiki](https://github.com/pravega/flink-connectors/wiki).\n\nDid you find a problem or bug?\n* First, check our [FAQ](http://pravega.io/docs/latest/faq/).\n* If the FAQ does not help you, create a [new GitHub issue](https://github.com/pravega/pravega-samples/issues).\n\nDo you want to contribute a new example application?\n* Follow the [guidelines for contributors](https://github.com/pravega/pravega/wiki/Contributing).\n\nHave fun!!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpravega%2Fpravega-samples","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpravega%2Fpravega-samples","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpravega%2Fpravega-samples/lists"}