Flink1.11内存模型与参数调整

原创

来一块提拉米苏 2022-11-18 15:57:00 博主文章分类：大数据 ©著作权

©著作权归作者所有：来自51CTO博客作者来一块提拉米苏的原创作品，请联系作者获取转载授权，否则将追究法律责任

Flink TaskManager启动日志

INFO  [] - Loading configuration property: taskmanager.memory.process.size, 2048m
INFO  [] - Loading configuration property: taskmanager.memory.managed.fraction, 0.3
INFO  [] - Loading configuration property: taskmanager.memory.jvm-overhead.fraction, 0.2
INFO  [] - Loading configuration property: taskmanager.memory.network.fraction, 0.1
INFO  [] - Loading configuration property: parallelism.default, 2
INFO  [] - Loading configuration property: task.cancellation.timeout, 0
INFO  [] - Final TaskExecutor Memory configuration:
INFO  [] -   Total Process Memory:          2.000gb (2147483648 bytes)
INFO  [] -     Total Flink Memory:          1.350gb (1449551456 bytes)
INFO  [] -       Total JVM Heap Memory:     701.440mb (735513127 bytes)
INFO  [] -         Framework:               128.000mb (134217728 bytes)
INFO  [] -         Task:                    573.440mb (601295399 bytes)
INFO  [] -       Total Off-heap Memory:     680.960mb (714038329 bytes)
INFO  [] -         Managed:                 414.720mb (434865454 bytes)
INFO  [] -         Total JVM Direct Memory: 266.240mb (279172875 bytes)
INFO  [] -           Framework:             128.000mb (134217728 bytes)
INFO  [] -           Task:                  0 bytes
INFO  [] -           Network:               138.240mb (144955147 bytes)
INFO  [] -     JVM Metaspace:               256.000mb (268435456 bytes)
INFO  [] -     JVM Overhead:                409.600mb (429496736 bytes)

Flink1.11内存模型与参数调整_flink

Total Process Memory：flink总资源数 2048m，参数：taskmanager.memory.process.size
JVM Metaspace：JVM元空间，参数：taskmanager.memory.jvm-metaspace.size，计算：默认值256m
JVM Overhead：JVM额外开销，参数：taskmanager.memory.jvm-overhead.fraction，计算：2048 x 0.2=409.6m
Total Flink Memory：flink可用资源数，计算：由1-2-3=1.35g
Total JVM Heap Memory：JVM堆内存，计算：由4-6=701.44m
Total Off-heap Memory：JVM堆外内存，计算：由7+8=680.96m
Total JVM Direct Memory：计算：由9+10(默认为0)+11=266.240m
Managed：托管内存，参数：taskmanager.memory.flink.size，计算：Total Flink Memory x 0.3=1.35 x 0.3= 414.72m
Network：网络缓存，参数：taskmanager.memory.network.fraction，计算：Total Flink Memory x 0.1=1.35 x 0.1= 138.240m
Framework：Flink框架内存，参数：taskmanager.memory.framework.heap.size，计算：默认128m
Task：任务内存，计算：5-10=573.440m