{"id":22096900,"url":"https://github.com/streamnative/flink-example","last_synced_at":"2025-07-24T22:32:41.026Z","repository":{"id":42052663,"uuid":"324091303","full_name":"streamnative/flink-example","owner":"streamnative","description":"Flink Pulsar Integration Related Examples","archived":false,"fork":false,"pushed_at":"2022-12-06T01:55:27.000Z","size":264,"stargazers_count":9,"open_issues_count":1,"forks_count":7,"subscribers_count":25,"default_branch":"main","last_synced_at":"2023-03-03T02:46:13.859Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/streamnative.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null}},"created_at":"2020-12-24T07:08:56.000Z","updated_at":"2022-12-14T23:31:25.000Z","dependencies_parsed_at":"2023-01-23T05:30:29.003Z","dependency_job_id":null,"html_url":"https://github.com/streamnative/flink-example","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fflink-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fflink-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fflink-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fflink-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/streamnative","download_url":"https://codeload.github.com/streamnative/flink-example/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227482461,"owners_count":17779968,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-01T04:13:18.937Z","updated_at":"2024-12-01T04:13:19.556Z","avatar_url":"https://github.com/streamnative.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Flink Pulsar Integration Related Examples\n\nWe provided a set of examples on using the latest `flink-connector-pulsar` in the Flink repository.\nShowing the user how to use this connector.\n\n## The example list\n\n1. `SimpleSource`: Consuming the message from Pulsar by using Flink's `StringSchema`.\n2. `SimpleSink`: Write the message into Pulsar by using Flink's `StringSchema`.\n3. `LoadEventSink`: Write load event messages with multiple sub-classes into Pulsar by `JSONSchema`.\n4. `LoadEventSource`: Consuming load event messages from Pulsar by `JSONSchema` and filter the `LoadCreateEvent` class.\n\n## How to use\n\n### Prepare the Pulsar instance\n\n#### Docker Compose (Recommend)\n\nDocker compose is quite easy to use. Simply execute `docker compose run pulsar` in project root directory.\n\n#### Docker command\n\nWe use docker to run an operable Pulsar environment. All the thing you need to do is just one command.\n\n```shell\ncd \"${this flink-example project directory}\"\n\nsudo docker run -it \\\n  -p 6650:6650 \\\n  -p 8080:8080 \\\n  --mount type=bind,source=${PWD}/docker/data,target=/pulsar/data \\\n  --mount type=bind,source=${PWD}/docker/bootstrap.sh,target=/pulsar/bin/bootstrap.sh \\\n  apachepulsar/pulsar:2.10.2 \\\n  /pulsar/bin/bootstrap.sh\n```\n\n### Install `pulsarctl`\n\nAfter install and setup the Pulsar standalone, we need some management tools for operating on the Pulsar cluster.\nWe prefer to use [pulsarctl](https://github.com/streamnative/pulsarctl) because it supports shell auto-completion.\nYou can skip this section if you want to use the scripts bundled in Pulsar distribution.\n\n#### Mac operating system\n\nUse [homebrew](https://brew.sh/) to install `pulsarctl` on the Mac operating system.\n\n```bash\nbrew tap streamnative/streamnative\nbrew install pulsarctl\n```\n\nWe would auto install zsh-completion and bash-completion when you use Homebrew.\n\n#### Linux operating system\n\nUse this command to install `pulsarctl` on the Linux operating system.\n\n```bash\nsh -c \"$(curl -fsSL https://raw.githubusercontent.com/streamnative/pulsarctl/master/install.sh)\"\n```\n\n#### Windows operating system\n\nTo install `pulsarctl` on the Windows operating system, follow these steps:\n\n1. Download the package from [here](https://github.com/streamnative/pulsarctl/releases).\n2. Add the `pulsarctl` directory to your system PATH.\n3. Execute `pulsarctl -h`  to verify that `pulsarctl` is work.\n\n### Prepare the test dataset\n\nAll the code snippet shown below was using `pulsarctl`. You can convert it to Pulsar scripts\nby reading [the documentation](https://pulsar.apache.org/docs/en/admin-api-overview/) for the Pulsar admin interface.\n\n1. Create and use pulsarctl's context for connecting to a Pulsar standalone instance.\n\n```shell\n## Change the 192.168.50.8 to your Pulsar standalone address. \npulsarctl context set development --admin-service-url=\"http://192.168.50.8:8080\"\n\n## Use the created context\npulsarctl context use development\n```\n\n2. Execute the initialization scripts. `docker/create-topics.sh`\n\n### Execute the program in IntelliJ IDEA for DataStream Connectors\n\nAll the required configurations are defined in `configs.yml` file. Change the `serviceUrl` and `adminUrl` to your\npulsar standalone address.\n\nWe share the IDEA run configuration in `.run` directory. You can choose the example case in IDEA's `Run Configuration` and execute it.\n\n\n### Try the SQL Connector in docker container\n\nPlease go to [SQL examples](docs/sql-example.md)\n\n## Reference\n\n* [Apache Pulsar Connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/pulsar/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamnative%2Fflink-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstreamnative%2Fflink-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamnative%2Fflink-example/lists"}