Prometheus JMX Exporter 微服务 JVM


0x00 概述

本文实现微服务JVM监控的方法为,使用volume HostPath挂载的JMX Exporter的方式在容器内以in-process的方式实现对微服务的JMV监控。

主要坑点为HostPath挂载JMX Exporter javaagent导致微服务无法启动。

 

0x02 部署

2.1 下载JMX Exporter javaagent包

$ mkdir -p /tmp/jmx_exporter
$ wget -O /tmp/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.17.0/jmx_prometheus_javaagent-0.17.0.jar

2.2 配置YAML文件

此处镜像采用Deployment+Tomcat的镜像作为样例,自测的时候注意及时更换镜像;

'''
1. 采用HostPath的方式,将下载的JMX Exporter的jar包挂载到服务器容内启动;
2. JMX Exporter的配置文件以configmap的形式挂载到服务容器内;
'''

 

Tomcat服务的Deployment文件:

apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-with-jvmExporter
namespace: default
labels:
app: tomcat-with-jvmExporter
spec:
replicas: 1
selector:
matchLabels:
app: tomcat-with-jvmExporter
template:
metadata:
labels:
app: tomcat-with-jvmExporter
spec:
nodeSelector:
kubernetes.io/hostname: docker49
restartPolicy: Always
containers:
- name: tomcat-with-jvmExporter
image: tomcat:9.0.54-jdk8-openjdk
imagePullPolicy: Always
env:
# JVM env
- name: JAVA_OPTS
value: "-Xms500m -Xmx500m -javaagent:/jmx_prometheus_javaagent-0.17.0.jar=5555:/jmx/prometheus-jmx-config.yaml"
#value: "-Xms500m -Xmx500m"
volumeMounts:
- mountPath: /jmx_prometheus_javaagent-0.17.0.jar
name: jmx-prometheus-javaagent
- mountPath: /jmx
name: prometheus-jmx-config
resources:
limits:
cpu: 2000m
memory: "4096Mi"
requests:
cpu: 2000m
memory: "4096Mi"
volumes:
- name: jmx-prometheus-javaagent
hostPath:
path: /tmp/jmx_prometheus_javaagent-0.17.0.jar
- configMap:
name: prometheus-jmx-config
name: prometheus-jmx-config
'''
1. 通过hostpath挂载的方式,对于pod来说,不一定会调度到这个节点服务器上;

2. 如果pod没调度到有这个jar包的服务器上,在启动pod的时候,会发现pod挂载了jar包和配置文件,但是无法执行这个jar包;

3. 所以需要在yaml中通过nodeSelector指定pod调度到固定节点上,这样hostPath才能取到文件,pod才会成功挂载这个文件并正常运行

4. 上面我们将jar包下载到了docker49这台机器上,所以指定将tomcat的pod调度到docker49,注意红字部分

'''

未配置nodeSelector的话,会出现如下报错:

'''
Error opening zip file or JAR manifest missing : /jmx_prometheus_javaagent-0.17.0.jar
Error occurred during initialization of VM
agent library failed to init: instrument
'''

直接搜索以上报错日志,StackOverFlow上大部分都会说是权限问题,但是无论你改成755还是777都一样,配置nodeSelector,将pod调度到JMX Exporter jar包的机器上即可。

 

Tomcat服务的Service文件:

apiVersion: v1
kind: Service
metadata:
name: tomcat-with-jvmExporter-svc
namespace: default
labels:
app: tomcat-with-jvmExporter-svc
annotations:
prometheus.io/port: "5555"
prometheus.io/jvm: "true"
spec:
type: NodePort
ports:
- port: 8080
name: tomcat-with-jvmExporter-svc
targetPort: 30899
nodePort: 30899
- name: jmx-metrics
port: 5555
protocol: TCP
selector:
app: tomcat-with-jvmExporter

 

Tomcat服务的configmap文件(此处为方便测试采用最小化配置文件,​​官方配置文件样例​​):

apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-jmx-config
namespace: default
data:
prometheus-jmx-config.yaml: |
# lowercaseOutputLabelNames: true
# lowercaseOutputName: true
rules:
- pattern: ".*"
# - pattern: 'Catalina<type=GlobalRequestProcessor, name=\"(\w+-\w+)-(\d+)\"><>(\w+):'
# name: tomcat_$3_total
# labels:
# port: "$2"
# protocol: "$1"
# help: Tomcat global $3
# type: COUNTER
# - pattern: 'Catalina<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), name=([-a-zA-Z0-9+/$%~_-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|maxTime|processingTime|errorCount):'
# name: tomcat_servlet_$3_total
# labels:
# module: "$1"
# servlet: "$2"
# help: Tomcat servlet $3 total
# type: COUNTER
# - pattern: 'Catalina<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount):'
# name: tomcat_threadpool_$3
# labels:
# port: "$2"
# protocol: "$1"
# help: Tomcat threadpool $3
# type: GAUGE
# - pattern: 'Catalina<type=Manager, host=([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), context=([-a-zA-Z0-9+/$%~_-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
# name: tomcat_session_$3_total
# labels:
# context: "$2"
# host: "$1"
# help: Tomcat session $3 total
# type: COUNTER

 

2.3 All in One yaml文件

tomcat-with-jvmExporter-AllinOne.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-wtih-jvmExporter
namespace: default
labels:
app: tomcat-with-jvmExporter
spec:
replicas: 1
selector:
matchLabels:
app: tomcat-with-jvmExporter
template:
metadata:
labels:
app: tomcat-with-jvmExporter
spec:
nodeSelector:
kubernetes.io/hostname: docker49
restartPolicy: Always
containers:
- name: tomcat-with-jvmExporter
image: tomcat:9.0.54-jdk8-openjdk
imagePullPolicy: Always
env:
# JVM env
- name: JAVA_OPTS
value: "-Xms500m -Xmx500m -javaagent:/jmx_prometheus_javaagent-0.17.0.jar=5555:/jmx/prometheus-jmx-config.yaml"
#value: "-Xms500m -Xmx500m"
volumeMounts:
- mountPath: /jmx_prometheus_javaagent-0.17.0.jar
name: jmx-prometheus-javaagent
- mountPath: /jmx
name: prometheus-jmx-config
resources:
limits:
cpu: 2000m
memory: "4096Mi"
requests:
cpu: 2000m
memory: "4096Mi"
volumes:
- name: jmx-prometheus-javaagent
hostPath:
path: /tmp/jmx_prometheus_javaagent-0.17.0.jar
- configMap:
name: prometheus-jmx-config
name: prometheus-jmx-config
---
apiVersion: v1
kind: Service
metadata:
name: tomcat-with-jvmExporter-svc
namespace: default
labels:
app: tomcat-with-jvmExporter-svc
annotations:
prometheus.io/port: "5555"
prometheus.io/jvm: "true"
spec:
type: NodePort
ports:
- port: 8080
name: tomcat-with-jvmExporter-svc
targetPort: 30899
nodePort: 30899
- name: jmx-metrics
port: 5555
protocol: TCP
selector:
app: tomcat-with-jvmExporter
---
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-jmx-config
namespace: default
data:
prometheus-jmx-config.yaml: |
#lowercaseOutputLabelNames: true
#lowercaseOutputName: true
rules:
- pattern: ".*"
# - pattern: 'Catalina<type=GlobalRequestProcessor, name=\"(\w+-\w+)-(\d+)\"><>(\w+):'
# name: tomcat_$3_total
# labels:
# port: "$2"
# protocol: "$1"
# help: Tomcat global $3
# type: COUNTER
# - pattern: 'Catalina<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), name=([-a-zA-Z0-9+/$%~_-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|maxTime|processingTime|errorCount):'
# name: tomcat_servlet_$3_total
# labels:
# module: "$1"
# servlet: "$2"
# help: Tomcat servlet $3 total
# type: COUNTER
# - pattern: 'Catalina<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount):'
# name: tomcat_threadpool_$3
# labels:
# port: "$2"
# protocol: "$1"
# help: Tomcat threadpool $3
# type: GAUGE
# - pattern: 'Catalina<type=Manager, host=([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), context=([-a-zA-Z0-9+/$%~_-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
# name: tomcat_session_$3_total
# labels:
# context: "$2"
# host: "$1"
# help: Tomcat session $3 total
# type: COUNTER

2.4 启动并检查服务

启动服务:

kubectl apply -f tomcat-with-jvmExporter-AllinOne.yaml

 检查服务:

kubectl get svc -n default

 检查JVM Exporter是否已获取JVM监控数据,此处docker49 ip:192.168.6.6,JVM Exporter对外nodeExporter为30566

curl 192.168.6.6:30566/metrics

 如果返回以下数据,说明已成功获取监控数据:

jvm_threads_daemon 36.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
jvm_threads_peak 80.0
# HELP jvm_threads_started_total Started thread count of a JVM
# TYPE jvm_threads_started_total counter
jvm_threads_started_total 155322.0
# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
# TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
# HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
# TYPE jvm_threads_deadlocked_monitor gauge
jvm_threads_deadlocked_monitor 0.0
# HELP jvm_threads_state Current count of threads by state
# TYPE jvm_threads_state gauge
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="TIMED_WAITING",} 22.0
jvm_threads_state{state="TERMINATED",} 0.0
jvm_threads_state{state="RUNNABLE",} 12.0
jvm_threads_state{state="WAITING",} 41.0
# HELP jvm_classes_currently_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_currently_loaded gauge
jvm_classes_currently_loaded 18332.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 18478.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 146.0
# HELP jmx_config_reload_failure_created Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_created gauge
jmx_config_reload_failure_created 1.655778604558E9
# HELP jmx_config_reload_success_created Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_created gauge
jmx_config_reload_success_created 1.655778604539E9
# HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_created gauge
jvm_memory_pool_allocated_bytes_created{pool="Code Cache",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="PS Eden Space",} 1.655778606951E9
jvm_memory_pool_allocated_bytes_created{pool="PS Old Gen",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="PS Survivor Space",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.655778606954E9

 

0x03 配置Prometheus获取监控个数据

修改prometheus.yml配置文件,新增JVM Exporter数据

- job_name: "jvm-ucm-config"
honor_labels: true
static_configs:
- targets: ["192.168.6.6:30566"]

 重新加载Prometheus配置文件

kill -HUP $pid

 

0x04 Grafana导入JVM监控面板

推荐面板id:11278

需要修改面板变量,根据以上配置,变量设置为2,job+instance

Prometheus监控学习笔记之使用JMX Exporter监控微服务JVM_jar

 

0x05 参考

​prometheus/jmx_exporter官方Github​

​K8S 中使用 Prometheus 监控 JVM (一)​