我这里搭建了3节点的集群
操作系统:centos7
配置静态ip
配置三节点的免密通讯
配置时钟同步
安装jdk1.8
以上的具体步骤我就不多说了。
下载flink安装包:https://archive.apache.org/dist/flink/flink-1.8.0/
上传安装包到集群
解压安装包
直接启动flink
./start-cluster.sh
这两个进程在的话证明启动成功
打开浏览器访问:http://192.168.215.161:8081/ 大家根据自己的实际ip地址来
可以看到我们通过8081端口访问Local模式的flink集群
通过idea的maven把之前的项目打包
D:\MyCode\FlinkStudy\flink>mvn clean package
D:\MyCode\FlinkStudy\flink>set JAVA_HOME=D:\SoftWares\JDK
[INFO] Scanning for projects...
[INFO]
[INFO] ---------------------------< com.gong:flink >---------------------------
[INFO] Building Flink Quickstart Job 1.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ flink ---
[INFO] Deleting D:\MyCode\FlinkStudy\flink\target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ flink ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ flink ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 4 source files to D:\MyCode\FlinkStudy\flink\target\classes
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (default) @ flink ---
[WARNING] Expected all dependencies to require Scala version: 2.11.12
[WARNING] org.apache.flink:flink-runtime_2.11:1.6.2 requires scala version: 2.11.12
[WARNING] org.scala-lang:scala-reflect:2.11.12 requires scala version: 2.11.12
[WARNING] org.apache.flink:flink-scala_2.11:1.6.2 requires scala version: 2.11.12
[WARNING] org.apache.flink:flink-scala_2.11:1.6.2 requires scala version: 2.11.12
[WARNING] org.scala-lang:scala-compiler:2.11.12 requires scala version: 2.11.12
[WARNING] org.scala-lang.modules:scala-xml_2.11:1.0.5 requires scala version: 2.11.7
[WARNING] Multiple versions of scala libraries detected!
[INFO] Using incremental compilation
[INFO] Compiling 2 Scala sources and 4 Java sources to D:\MyCode\FlinkStudy\flink\target\classes...
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ flink ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\MyCode\FlinkStudy\flink\src\test\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ flink ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (default) @ flink ---
[WARNING] Expected all dependencies to require Scala version: 2.11.12
[WARNING] org.apache.flink:flink-runtime_2.11:1.6.2 requires scala version: 2.11.12
[WARNING] org.scala-lang:scala-reflect:2.11.12 requires scala version: 2.11.12
[WARNING] org.apache.flink:flink-scala_2.11:1.6.2 requires scala version: 2.11.12
[WARNING] org.apache.flink:flink-scala_2.11:1.6.2 requires scala version: 2.11.12
[WARNING] org.scala-lang:scala-compiler:2.11.12 requires scala version: 2.11.12
[WARNING] org.scala-lang.modules:scala-xml_2.11:1.0.5 requires scala version: 2.11.7
[WARNING] Multiple versions of scala libraries detected!
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ flink ---
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ flink ---
[INFO] Building jar: D:\MyCode\FlinkStudy\flink\target\flink-1.0.jar
[INFO]
[INFO] --- maven-shade-plugin:3.0.0:shade (default) @ flink ---
[INFO] Including org.apache.flink:flink-java:jar:1.6.2 in the shaded jar.
[INFO] Including org.apache.flink:flink-core:jar:1.6.2 in the shaded jar.
[INFO] Including org.apache.flink:flink-annotations:jar:1.6.2 in the shaded jar.
[INFO] Including org.apache.flink:flink-metrics-core:jar:1.6.2 in the shaded jar.
[INFO] Including com.esotericsoftware.kryo:kryo:jar:2.24.0 in the shaded jar.
[INFO] Including com.esotericsoftware.minlog:minlog:jar:1.2 in the shaded jar.
[INFO] Including org.objenesis:objenesis:jar:2.1 in the shaded jar.
[INFO] Including commons-collections:commons-collections:jar:3.2.2 in the shaded jar.
[INFO] Including org.apache.commons:commons-compress:jar:1.4.1 in the shaded jar.
[INFO] Including org.tukaani:xz:jar:1.0 in the shaded jar.
[INFO] Including org.apache.flink:flink-shaded-asm:jar:5.0.4-4.0 in the shaded jar.
[INFO] Including org.apache.commons:commons-lang3:jar:3.3.2 in the shaded jar.
[INFO] Including org.apache.commons:commons-math3:jar:3.5 in the shaded jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.7 from the shaded jar.
[INFO] Excluding com.google.code.findbugs:jsr305:jar:1.3.9 from the shaded jar.
[INFO] Excluding org.apache.flink:force-shading:jar:1.6.2 from the shaded jar.
[INFO] Including org.apache.flink:flink-streaming-java_2.11:jar:1.6.2 in the shaded jar.
[INFO] Including org.apache.flink:flink-runtime_2.11:jar:1.6.2 in the shaded jar.
[INFO] Including org.apache.flink:flink-queryable-state-client-java_2.11:jar:1.6.2 in the shaded jar.
[INFO] Including org.apache.flink:flink-hadoop-fs:jar:1.6.2 in the shaded jar.
[INFO] Including commons-io:commons-io:jar:2.4 in the shaded jar.
[INFO] Including org.apache.flink:flink-shaded-netty:jar:4.1.24.Final-4.0 in the shaded jar.
[INFO] Including org.apache.flink:flink-shaded-jackson:jar:2.7.9-4.0 in the shaded jar.
[INFO] Including commons-cli:commons-cli:jar:1.3.1 in the shaded jar.
[INFO] Including org.javassist:javassist:jar:3.19.0-GA in the shaded jar.
[INFO] Including com.typesafe.akka:akka-actor_2.11:jar:2.4.20 in the shaded jar.
[INFO] Including com.typesafe:config:jar:1.3.0 in the shaded jar.
[INFO] Including org.scala-lang.modules:scala-java8-compat_2.11:jar:0.7.0 in the shaded jar.
[INFO] Including com.typesafe.akka:akka-stream_2.11:jar:2.4.20 in the shaded jar.
[INFO] Including org.reactivestreams:reactive-streams:jar:1.0.0 in the shaded jar.
[INFO] Including com.typesafe:ssl-config-core_2.11:jar:0.2.1 in the shaded jar.
[INFO] Including com.typesafe.akka:akka-protobuf_2.11:jar:2.4.20 in the shaded jar.
[INFO] Including com.typesafe.akka:akka-slf4j_2.11:jar:2.4.20 in the shaded jar.
[INFO] Including org.clapper:grizzled-slf4j_2.11:jar:1.0.2 in the shaded jar.
[INFO] Including com.github.scopt:scopt_2.11:jar:3.5.0 in the shaded jar.
[INFO] Including org.xerial.snappy:snappy-java:jar:1.1.4 in the shaded jar.
[INFO] Including com.twitter:chill_2.11:jar:0.7.4 in the shaded jar.
[INFO] Including com.twitter:chill-java:jar:0.7.4 in the shaded jar.
[INFO] Including org.apache.flink:flink-clients_2.11:jar:1.6.2 in the shaded jar.
[INFO] Including org.apache.flink:flink-optimizer_2.11:jar:1.6.2 in the shaded jar.
[INFO] Including org.apache.flink:flink-shaded-guava:jar:18.0-4.0 in the shaded jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.7 from the shaded jar.
[INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar.
[INFO] Including org.apache.flink:flink-scala_2.11:jar:1.6.2 in the shaded jar.
[INFO] Including org.scala-lang:scala-reflect:jar:2.11.12 in the shaded jar.
[INFO] Including org.scala-lang:scala-library:jar:2.11.12 in the shaded jar.
[INFO] Including org.scala-lang:scala-compiler:jar:2.11.12 in the shaded jar.
[INFO] Including org.scala-lang.modules:scala-xml_2.11:jar:1.0.5 in the shaded jar.
[INFO] Including org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4 in the shaded jar.
[INFO] Including org.apache.flink:flink-streaming-scala_2.11:jar:1.6.2 in the shaded jar.
[INFO] Including org.apache.flink:flink-table_2.10:jar:1.3.2 in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing D:\MyCode\FlinkStudy\flink\target\flink-1.0.jar with D:\MyCode\FlinkStudy\flink\target\flink-1.0-shaded.jar
[INFO] Dependency-reduced POM written at: D:\MyCode\FlinkStudy\flink\dependency-reduced-pom.xml
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.482 s
[INFO] Finished at: 2020-09-22T20:03:08+08:00
[INFO] ------------------------------------------------------------------------
D:\MyCode\FlinkStudy\flink>
将这个包上传到集群
在集群运行
bin/flink run -c com.gong.flink.batch.WordCount flink-1.0.jar --input /opt/modules/flink-1.8.0/test.txt --output /opt/modules/flink-1.8.0/out.txt
这里我们知道输入路径和输出路径
下面的是一个官方的例子
先启动nc
重新打开一个终端启动官方的例子
bin/flink run examples/streaming/SocketWindowWordCount.jar --hostname master --port 9999
在nc端输入一些字符
通过浏览器的界面,我们可以看到流处理的结果
通过界面把作业停止掉
可以看到已经停止了