-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
20 changed files
with
778 additions
and
37 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| # User Delete | ||
|
|
||
| User Delete job | ||
|
|
||
| ## Getting Started | ||
|
|
||
| These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a yarn or kubernetes. | ||
| ### Prerequisites | ||
|
|
||
| 1. Download flink-1.13.6-scala_2.12 from [apache-flink-downloads](https://www.apache.org/dyn/closer.lua/flink/flink-1.13.6/flink-1.13.6-bin-scala_2.12.tgz). | ||
| 2. Download [hadoop dependencies](https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar) (only for running on Yarn). Copy the hadoop dependency jar under lib folder of the flink download. | ||
| 3. export HADOOP_CLASSPATH=`<hadoop-executable-dir>/hadoop classpath` either in .bashrc or current execution shell. | ||
| 4. Docker installed. | ||
| 5. A running yarn cluster or a kubernetes cluster. | ||
|
|
||
| ### Build | ||
|
|
||
| mvn clean install | ||
|
|
||
| ## Deployment | ||
|
|
||
| ### Yarn | ||
|
|
||
| Flink requires memory to be allocated for both job-manager and task manager. -yjm parameter assigns job-manager memory and -ytm assigns task-manager memory. | ||
|
|
||
| ``` | ||
| ./bin/flink run -m yarn-cluster -p 2 -yjm 1024m -ytm 1024m <cokreat-jobs>/user-delete/target/user-delete-0.0.1.jar | ||
| ``` | ||
|
|
||
| ### Kubernetes | ||
|
|
||
| ``` | ||
| # Create a single node cluster | ||
| k3d create --server-arg --no-deploy --server-arg traefik --name flink-cluster --image rancher/k3s:v1.0.0 | ||
| # Export the single node cluster into KUBECONFIG in the current shell or in ~/.bashrc. | ||
| export KUBECONFIG="$(k3d get-kubeconfig --name='flink-cluster')" | ||
| # Only for Mac OSX | ||
| # /usr/local/bin/kubectl -> /Applications/Docker.app/Contents/Resources/bin/kubectl | ||
| rm /usr/local/bin/kubectl | ||
| brew link --overwrite kubernetes-cli | ||
| # Create a configmap using the flink-configuration-configmap.yaml | ||
| kubectl create -f knowledge-platform-job/kubernetes/flink-configuration-configmap.yaml | ||
| # Create pods for jobmanager-service, job-manager and task-manager using the yaml files | ||
| kubectl create -f knowledge-platform-job/kubernetes/jobmanager-service.yaml | ||
| kubectl create -f knowledge-platform-job/kubernetes/jobmanager-deployment.yaml | ||
| kubectl create -f knowledge-platform-job/kubernetes/taskmanager-deployment.yaml | ||
| # Create a port-forwarding for accessing the job-manager UI on localhost:8081 | ||
| kubectl port-forward deployment/flink-jobmanager 8081:8081 | ||
| # Submit the job to the Kubernetes single node cluster flink-cluster | ||
| ./bin/flink run -m localhost:8081 <cokreat-jobs>/user-delete/target/user-delete-0.0.1.jar | ||
| # Commands to delete the pods created in the cluster | ||
| kubectl delete deployment/flink-jobmanager | ||
| kubectl delete deployment/flink-taskmanager | ||
| kubectl delete service/flink-jobmanager | ||
| kubectl delete configmaps/flink-config | ||
| # Command to stop the single-node cluster | ||
| k3d stop --name="flink-cluster" | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,215 @@ | ||
| <?xml version="1.0" encoding="UTF-8"?> | ||
|
|
||
| <project xmlns="http://maven.apache.org/POM/4.0.0" | ||
| xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
| <modelVersion>4.0.0</modelVersion> | ||
| <parent> | ||
| <groupId>org.sunbird</groupId> | ||
| <artifactId>cokreat-jobs</artifactId> | ||
| <version>1.0</version> | ||
| </parent> | ||
| <artifactId>user-delete</artifactId> | ||
| <version>1.0.0</version> | ||
| <packaging>jar</packaging> | ||
| <name>user-delete</name> | ||
| <description> | ||
| User Delete Flink Job | ||
| </description> | ||
|
|
||
| <properties> | ||
| <encoding>UTF-8</encoding> | ||
| <scoverage.plugin.version>1.4.0</scoverage.plugin.version> | ||
| </properties> | ||
|
|
||
| <dependencies> | ||
| <dependency> | ||
| <groupId>org.apache.flink</groupId> | ||
| <artifactId>flink-clients_${scala.version}</artifactId> | ||
| <version>${flink.version}</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.flink</groupId> | ||
| <artifactId>flink-streaming-scala_${scala.version}</artifactId> | ||
| <version>${flink.version}</version> | ||
| <!-- <scope>provided</scope>--> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.sunbird</groupId> | ||
| <artifactId>jobs-core</artifactId> | ||
| <version>1.0.0</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>joda-time</groupId> | ||
| <artifactId>joda-time</artifactId> | ||
| <version>2.10.6</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.sunbird</groupId> | ||
| <artifactId>jobs-core</artifactId> | ||
| <version>1.0.0</version> | ||
| <type>test-jar</type> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.flink</groupId> | ||
| <artifactId>flink-test-utils_${scala.version}</artifactId> | ||
| <version>${flink.version}</version> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.flink</groupId> | ||
| <artifactId>flink-runtime_${scala.version}</artifactId> | ||
| <version>${flink.version}</version> | ||
| <scope>test</scope> | ||
| <classifier>tests</classifier> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.flink</groupId> | ||
| <artifactId>flink-streaming-java_${scala.version}</artifactId> | ||
| <version>${flink.version}</version> | ||
| <scope>test</scope> | ||
| <classifier>tests</classifier> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.scalatest</groupId> | ||
| <artifactId>scalatest_${scala.version}</artifactId> | ||
| <version>3.0.6</version> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.mockito</groupId> | ||
| <artifactId>mockito-core</artifactId> | ||
| <version>3.3.3</version> | ||
| <scope>test</scope> | ||
| </dependency> | ||
|
|
||
| </dependencies> | ||
|
|
||
| <build> | ||
| <sourceDirectory>src/main/scala</sourceDirectory> | ||
| <testSourceDirectory>src/test/scala</testSourceDirectory> | ||
| <plugins> | ||
| <plugin> | ||
| <groupId>org.apache.maven.plugins</groupId> | ||
| <artifactId>maven-compiler-plugin</artifactId> | ||
| <version>3.8.1</version> | ||
| <configuration> | ||
| <release>11</release> | ||
| </configuration> | ||
| </plugin> | ||
| <plugin> | ||
| <groupId>org.apache.maven.plugins</groupId> | ||
| <artifactId>maven-shade-plugin</artifactId> | ||
| <version>3.2.1</version> | ||
| <executions> | ||
| <!-- Run shade goal on package phase --> | ||
| <execution> | ||
| <phase>package</phase> | ||
| <goals> | ||
| <goal>shade</goal> | ||
| </goals> | ||
| <configuration> | ||
| <artifactSet> | ||
| <excludes> | ||
| <exclude>com.google.code.findbugs:jsr305</exclude> | ||
| </excludes> | ||
| </artifactSet> | ||
| <filters> | ||
| <filter> | ||
| <!-- Do not copy the signatures in the META-INF folder. | ||
| Otherwise, this might cause SecurityExceptions when using the JAR. --> | ||
| <artifact>*:*</artifact> | ||
| <excludes> | ||
| <exclude>META-INF/*.SF</exclude> | ||
| <exclude>META-INF/*.DSA</exclude> | ||
| <exclude>META-INF/*.RSA</exclude> | ||
| </excludes> | ||
| </filter> | ||
| </filters> | ||
| <transformers> | ||
| <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/> | ||
| <transformer | ||
| implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> | ||
| <mainClass>org.sunbird.job.task.UserDeleteStreamTask</mainClass> | ||
| </transformer> | ||
| <!-- append default configs --> | ||
| <transformer | ||
| implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> | ||
| <resource>reference.conf</resource> | ||
| </transformer> | ||
| </transformers> | ||
| </configuration> | ||
| </execution> | ||
| </executions> | ||
| </plugin> | ||
|
|
||
| <plugin> | ||
| <groupId>net.alchim31.maven</groupId> | ||
| <artifactId>scala-maven-plugin</artifactId> | ||
| <version>4.4.0</version> | ||
| <configuration> | ||
| <source>11</source> | ||
| <target>11</target> | ||
| <scalaVersion>${scala.maj.version}</scalaVersion> | ||
| <checkMultipleScalaVersions>false</checkMultipleScalaVersions> | ||
| </configuration> | ||
| <executions> | ||
| <execution> | ||
| <id>scala-compile-first</id> | ||
| <phase>process-resources</phase> | ||
| <goals> | ||
| <goal>add-source</goal> | ||
| <goal>compile</goal> | ||
| </goals> | ||
| </execution> | ||
| <execution> | ||
| <id>scala-test-compile</id> | ||
| <phase>process-test-resources</phase> | ||
| <goals> | ||
| <goal>testCompile</goal> | ||
| </goals> | ||
| </execution> | ||
| </executions> | ||
| </plugin> | ||
|
|
||
| <plugin> | ||
| <artifactId>maven-surefire-plugin</artifactId> | ||
| <version>2.22.2</version> | ||
| <configuration> | ||
| <skipTests>true</skipTests> | ||
| </configuration> | ||
| </plugin> | ||
|
|
||
| <plugin> | ||
| <groupId>org.scalatest</groupId> | ||
| <artifactId>scalatest-maven-plugin</artifactId> | ||
| <version>1.0</version> | ||
| <configuration> | ||
| <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory> | ||
| <junitxml>.</junitxml> | ||
| <filereports>user-delete-testsuite.txt</filereports> | ||
| </configuration> | ||
| <executions> | ||
| <execution> | ||
| <id>test</id> | ||
| <goals> | ||
| <goal>test</goal> | ||
| </goals> | ||
| </execution> | ||
| </executions> | ||
| </plugin> | ||
| <plugin> | ||
| <groupId>org.scoverage</groupId> | ||
| <artifactId>scoverage-maven-plugin</artifactId> | ||
| <version>${scoverage.plugin.version}</version> | ||
| <configuration> | ||
| <scalaVersion>${scala.version}</scalaVersion> | ||
| <aggregate>true</aggregate> | ||
| <highlighting>true</highlighting> | ||
| </configuration> | ||
| </plugin> | ||
| </plugins> | ||
| </build> | ||
|
|
||
| </project> |
Oops, something went wrong.