Swim Flink Connector Library 

This is a Flink Connector library. The Swim Flink Connector library acts as a bridge between Flink and Swim applications. This library should be configured to send messages generated from Flink jobs to a specified Swim application.

Apache Flink

Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities.

Learn more about Flink at https://flink.apache.org/

Building the Swim Flink Connector from Source

Prerequisites

• Install JDK 11
• Ensure that your JAVA_HOME environment variable points to the Java installation.
• Ensure that your PATH includes $JAVA_HOME.

On Windows

$ .\gradlew.bat build

On Linux or MacOS

$ ./gradlew build

The resulting jars can be found in the build/libs.

Design and Implementation Details

Swim Application Overview

Swim applications consist of interconnected, distributed objects, called Web Agents. Each Web Agent has URI address, like a REST endpoint. But unlike RESTful Web Services, Web Agents are stateful, and accessed via streaming APIs. Each Web Agent has a set of named lanes, representing the properties and methods of the Web Agent. Lanes come in several varieties, corresponding to common data structures and access patterns.

To send a message to a specific Web Agent, external systems (like the Swim Flink Connector) need the following:

1. Host URI : A Swim Application has a Host URI (similar to a web server's host URI).
2. Web Agent URI: Web Agents are hosted in a Swim Application and have unique URIs (similar to a REST endpoint).
3. Lane URI: The name of the lane in the Web Agent.

For a detailed overview of Swim concepts please refer to swimos.org

Validating with the Test Application and Example Job

The Swim Flink Connector library can be validated by running the test application (test-app) alongside the (example-job). See the test-app and example-job documentation to view how to run each respectively.

Access the data from the Application

Data sent to the Web Agents in the Test Application from the Swim Flink Connector library can be accessed using HTTP APIs. For example, if the id extracted from the message is "User_5" then the Swim Flink Connector Library will compute the agent URI to be: /agent/User_5. The HTTP API to get the message from this agent will be
http://<host-name>:9001/agent/User_5?lane=latestData

If the id extracted from the message is "User_7", then the HTTP API to get the message from this agent will be:
http://<host-name>:9001/agent/User_7?lane=latestData

Here the <host-name> will be "localhost" if the Test Application is running on the same machine as the Swim Flink Connector library. If the Test Application is running on a different machine then the <host-name> will be the Fully Qualified Domain Name of the machine.

Repository Structure

Key files

• gradlew/gradlew.bat — gradle wrapper
• build.gradle — gradle file to build the library
• gradle.properties — configuration variables

Key directories

• src — source code
  • main/java — java source code
  • main/resources — configuration files
  • test-unit test code
• test-app — test Swim application which may be used to validate the swim-flink-connector library
• example-job — example Flink job which may be used to validate the swim-flink-connector library