https://github.com/swimos/swim-flink-connector
https://github.com/swimos/swim-flink-connector
Last synced: 5 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/swimos/swim-flink-connector
- Owner: swimos
- License: apache-2.0
- Created: 2023-06-01T17:16:54.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-10-04T13:15:15.000Z (about 2 years ago)
- Last Synced: 2025-05-13T00:45:34.724Z (5 months ago)
- Language: Java
- Size: 76.2 KB
- Stars: 1
- Watchers: 7
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# 
Swim Flink Connector Library  [](https://github.com/swimos/swim/blob/main/LICENSE)
This is a [Flink Connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/) library. The Swim Flink Connector library
acts as a bridge between Flink and Swim applications. This library should be configured to send messages generated from
Flink jobs to a specified Swim application.
## Apache Flink
Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities.
Learn more about Flink at [https://flink.apache.org/](https://flink.apache.org/)
## Building the Swim Flink Connector from Source
### Prerequisites
- Install JDK 11
- Ensure that your `JAVA_HOME` environment variable points to the Java installation.
- Ensure that your `PATH` includes `$JAVA_HOME`.
### On Windows
```bat
$ .\gradlew.bat build
```
### On Linux or MacOS
```bash
$ ./gradlew build
```
The resulting jars can be found in the `build/libs`.
## Design and Implementation Details
### Swim Application Overview
Swim applications consist of interconnected, distributed objects, called Web Agents. Each Web Agent has URI address, like a REST endpoint.
But unlike RESTful Web Services, Web Agents are stateful, and accessed via streaming APIs. Each Web Agent has a set of named lanes,
representing the properties and methods of the Web Agent. Lanes come in several varieties, corresponding to common data structures and access patterns.
To send a message to a specific Web Agent, external systems (like the Swim Flink Connector) need the following:
1. Host URI : A Swim Application has a Host URI (similar to a web server's host URI).
2. Web Agent URI: Web Agents are hosted in a Swim Application and have unique URIs (similar to a REST endpoint).
3. Lane URI: The name of the lane in the Web Agent.
For a detailed overview of Swim concepts please refer to [swimos.org](https://www.swimos.org/concepts/)
## Validating with the Test Application and Example Job
The Swim Flink Connector library can be validated by running the test application ([test-app](test-app)) alongside the
([example-job](example-job)).
See the [test-app](test-app/README.md) and [example-job](example-job/README.md) documentation to view how to run each respectively.
### Access the data from the Application
Data sent to the Web Agents in the Test Application from the Swim Flink Connector library can be accessed using HTTP APIs.
For example, if the `id` extracted from the message is "User_5" then the Swim Flink Connector Library will compute the agent URI to
be: `/agent/User_5`. The HTTP API to get the message from this agent will be
`http://:9001/agent/User_5?lane=latestData`
If the `id` extracted from the message is "User_7", then the HTTP API to get the message from this agent will be:
`http://:9001/agent/User_7?lane=latestData`
Here the `` will be "localhost" if the Test Application is running on the same machine as the Swim Flink Connector library.
If the Test Application is running on a different machine then the `` will be the Fully Qualified Domain Name of the machine.
## Repository Structure
### Key files
- [gradlew](gradlew)/[gradlew.bat](gradlew.bat) — gradle wrapper
- [build.gradle](build.gradle) — gradle file to build the library
- [gradle.properties](gradle.properties) — configuration variables
### Key directories
- [src](src) — source code
- [main/java](src/main/java) — java source code
- [main/resources](src/main/resources) — configuration files
- [test](src/test)-unit test code
- [test-app](test-app) — test Swim application which may be used to validate the swim-flink-connector library
- [example-job](example-job) — example Flink job which may be used to validate the swim-flink-connector library