jmx exporter整合prometheus

引文

Jmx_exporter是以代理的形式收集目标应用的jmx指标，这样做的好处在于无需对目标应用做任何的改动。

使用方法：

-javaagent:/path/to/JavaAgent.jar=[host:]:

• JavaAgent.jar 就是jmx_exporter的jar包
• [host:] 就是目标应用所在的地址和端口
• <yaml configuration file> 是配置文件

源码分析

下面开始进入源码分析

入口位于：
io.prometheus.jmx.JavaAgent#agentmain

public static void agentmain(String agentArgument, Instrumentation instrumentation) throws Exception {
     premain(agentArgument, instrumentation);
   }

   public static void premain(String agentArgument, Instrumentation instrumentation) throws Exception {
     // Bind to all interfaces by default (this includes IPv6).
     String host = "0.0.0.0";

     // If we have IPv6 address in square brackets, extract it first and then
     // remove it from arguments to prevent confusion from too many colons.
     Integer indexOfClosingSquareBracket = agentArgument.indexOf("]:");
     if (indexOfClosingSquareBracket >= 0) {
       host = agentArgument.substring(0, indexOfClosingSquareBracket + 1);
       agentArgument = agentArgument.substring(indexOfClosingSquareBracket + 2);
     }

     String[] args = agentArgument.split(":");
     if (args.length < 2 || args.length > 3) {
       System.err.println("Usage: -javaagent:/path/to/JavaAgent.jar=[host:]<port>:<yaml configuration file>");
       System.exit(1);
     }

     int port;
     String file;
     InetSocketAddress socket;

     if (args.length == 3) {
       port = Integer.parseInt(args[1]);
       socket = new InetSocketAddress(args[0], port);
       file = args[2];
     } else {
       port = Integer.parseInt(args[0]);
       socket = new InetSocketAddress(host, port);
       file = args[1];
     }

     new BuildInfoCollector().register();
     new JmxCollector(new File(file)).register();
     DefaultExports.initialize();
     server = new HTTPServer(socket, CollectorRegistry.defaultRegistry, true);
   }

参数解析

agentArgument 是通过命令行传入的参数，从中解析出IP地址host，端口port，配置文件file

收集器的注册

收集标量信息的结构

收集的信息会添加到MetricFamilySamples 的列表中，首先看看MetricFamilySamples的结构

public static class MetricFamilySamples {
        public final String name;
        public final Collector.Type type;
        public final String help;
        public final List<Collector.MetricFamilySamples.Sample> samples;
。。。。。。。。

这个类主要由以下几个属性：

• 1、 Name 指标名称
• 2、 Collector.Type 指标的类型：

• a) COUNTER
• b) GAUGE
• c) SUMMARY
• d) HISTOGRAM
• e) UNTYPED
  BuildInfoCollector收集的指标类型是GAUGE

• 3、 Help 指标的说明
• 4、 List<Collector.MetricFamilySamples.Sample> samples 指标的样本集合

再进一步看看样本Sample的结构
io.prometheus.client.Collector.MetricFamilySamples.Sample

public static class Sample {
            public final String name;
            public final List<String> labelNames;
            public final List<String> labelValues;
            public final double value;
            public final Long timestampMs;

属性介绍：

• 1、 String name 样本名称
• 2、 List labelNames 标签名称
• 3、 List labelValues 标签值
• 4、 double value 样本值
• 5、 Long timestampMs 时间戳

（1）BuildInfoCollector

用于收集 jxm_exporter的版本信息，让我们看看这个收集器是如何收集编译版本信息的
io.prometheus.jmx.BuildInfoCollector#collect

public List<Collector.MetricFamilySamples> collect() {
    List<Collector.MetricFamilySamples> mfs = new ArrayList<Collector.MetricFamilySamples>();

    GaugeMetricFamily artifactInfo = new GaugeMetricFamily(
            "jmx_exporter_build_info",
            "A metric with a constant '1' value labeled with the version of the JMX exporter.",
            asList("version", "name"));

    Package pkg = this.getClass().getPackage();
    String version = pkg.getImplementationVersion();
    String name = pkg.getImplementationTitle();

    artifactInfo.addMetric(asList(
            version != null ? version : "unknown",
            name != null ? name : "unknown"
    ), 1L);
    mfs.add(artifactInfo);

    return mfs;
  }

1、 获得包名和版本号：name,version
2、 将name,version封装成Sample，然后添加进MetricFamilySamples中的样本集合中

（2）JmxCollector

主要收集jmx_exporter.yaml文件中指定的对象的jmx指标。

public JmxCollector(File in) throws IOException, MalformedObjectNameException {
        configFile = in;
        config = loadConfig((Map<String, Object>)new Yaml().load(new FileReader(in)));
        config.lastUpdate = configFile.lastModified();
}

首先通过构造函数把配置文件加载进来到config中，下面看如何收集信息

io.prometheus.jmx.JmxCollector#collect

public List<MetricFamilySamples> collect() {
      if (configFile != null) {
        long mtime = configFile.lastModified();
        if (mtime > config.lastUpdate) {
          LOGGER.fine("Configuration file changed, reloading...");
          reloadConfig();
        }
      }

      Receiver receiver = new Receiver();
      JmxScraper scraper = new JmxScraper(config.jmxUrl, config.username, config.password, config.ssl,
              config.whitelistObjectNames, config.blacklistObjectNames, receiver, jmxMBeanPropertyCache);
      long start = System.nanoTime();
      double error = 0;
      if ((config.startDelaySeconds > 0) &&
        ((start - createTimeNanoSecs) / 1000000000L < config.startDelaySeconds)) {
        throw new IllegalStateException("JMXCollector waiting for startDelaySeconds");
      }
      try {
        scraper.doScrape();
      } catch (Exception e) {
        error = 1;
        StringWriter sw = new StringWriter();
        e.printStackTrace(new PrintWriter(sw));
        LOGGER.severe("JMX scrape failed: " + sw.toString());
      }
      List<MetricFamilySamples> mfsList = new ArrayList<MetricFamilySamples>();
      mfsList.addAll(receiver.metricFamilySamplesMap.values());
      List<MetricFamilySamples.Sample> samples = new ArrayList<MetricFamilySamples.Sample>();
      samples.add(new MetricFamilySamples.Sample(
          "jmx_scrape_duration_seconds", new ArrayList<String>(), new ArrayList<String>(), (System.nanoTime() - start) / 1.0E9));
      mfsList.add(new MetricFamilySamples("jmx_scrape_duration_seconds", Type.GAUGE, "Time this JMX scrape took, in seconds.", samples));

      samples = new ArrayList<MetricFamilySamples.Sample>();
      samples.add(new MetricFamilySamples.Sample(
          "jmx_scrape_error", new ArrayList<String>(), new ArrayList<String>(), error));
      mfsList.add(new MetricFamilySamples("jmx_scrape_error", Type.GAUGE, "Non-zero if this scrape failed.", samples));
      return mfsList;
    }

1、 首先判断配置文件是否有被修改，如果被修改就重新加载，可以看出jmx_exporter的配置是热加载的。

2、 等待开始收集jmx指标，等待的时间由配置startDelaySeconds指定。

3、 开始收集jmx指标，scraper.doScrape(); 这个方法会收集配置文件中指定的对象的信息，并且根据配置的正则表达式进行过滤

public void doScrape() throws Exception {
        MBeanServerConnection beanConn;
        JMXConnector jmxc = null;
        if (jmxUrl.isEmpty()) {
          beanConn = ManagementFactory.getPlatformMBeanServer();
        } else {
          Map<String, Object> environment = new HashMap<String, Object>();
          if (username != null && username.length() != 0 && password != null && password.length() != 0) {
            String[] credent = new String[] {username, password};
            environment.put(javax.management.remote.JMXConnector.CREDENTIALS, credent);
          }
          if (ssl) {
              environment.put(Context.SECURITY_PROTOCOL, "ssl");
              SslRMIClientSocketFactory clientSocketFactory = new SslRMIClientSocketFactory();
              environment.put(RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, clientSocketFactory);
              environment.put("com.sun.jndi.rmi.factory.socket", clientSocketFactory);
          }

          jmxc = JMXConnectorFactory.connect(new JMXServiceURL(jmxUrl), environment);
          beanConn = jmxc.getMBeanServerConnection();
        }
        try {
            // Query MBean names, see #89 for reasons queryMBeans() is used instead of queryNames()
            Set<ObjectName> mBeanNames = new HashSet<ObjectName>();
            for (ObjectName name : whitelistObjectNames) {
                for (ObjectInstance instance : beanConn.queryMBeans(name, null)) {
                    mBeanNames.add(instance.getObjectName());
                }
            }

            for (ObjectName name : blacklistObjectNames) {
                for (ObjectInstance instance : beanConn.queryMBeans(name, null)) {
                    mBeanNames.remove(instance.getObjectName());
                }
            }

            // Now that we have *only* the whitelisted mBeans, remove any old ones from the cache:
            jmxMBeanPropertyCache.onlyKeepMBeans(mBeanNames);

            for (ObjectName objectName : mBeanNames) {
                long start = System.nanoTime();
                scrapeBean(beanConn, objectName);
                logger.fine("TIME: " + (System.nanoTime() - start) + " ns for " + objectName.toString());
            }
        } finally {
          if (jmxc != null) {
            jmxc.close();
          }
        }
}

A、 首先根据有户名，密码等认证信息获取jmx的链接beanConn

B、 根据配置中的白名单whitelistObjectNames，黑名单blacklistObjectNames确定需要获取的对象名单

C、 挨个去获取名单中的对象信息scrapeBean(beanConn, objectName);

private void scrapeBean(MBeanServerConnection beanConn, ObjectName mbeanName) {
        MBeanInfo info;
        try {
          info = beanConn.getMBeanInfo(mbeanName);
        } catch (IOException e) {
          logScrape(mbeanName.toString(), "getMBeanInfo Fail: " + e);
          return;
        } catch (JMException e) {
          logScrape(mbeanName.toString(), "getMBeanInfo Fail: " + e);
          return;
        }
        MBeanAttributeInfo[] attrInfos = info.getAttributes();

        Map<String, MBeanAttributeInfo> name2AttrInfo = new LinkedHashMap<String, MBeanAttributeInfo>();
        for (int idx = 0; idx < attrInfos.length; ++idx) {
            MBeanAttributeInfo attr = attrInfos[idx];
            if (!attr.isReadable()) {
                logScrape(mbeanName, attr, "not readable");
                continue;
            }
            name2AttrInfo.put(attr.getName(), attr);
        }
        final AttributeList attributes;
        try {
            attributes = beanConn.getAttributes(mbeanName, name2AttrInfo.keySet().toArray(new String[0]));
        } catch (Exception e) {
            logScrape(mbeanName, name2AttrInfo.keySet(), "Fail: " + e);
            return;
        }
        for (Attribute attribute : attributes.asList()) {
            MBeanAttributeInfo attr = name2AttrInfo.get(attribute.getName());
            logScrape(mbeanName, attr, "process");
            processBeanValue(
                    mbeanName.getDomain(),
                    jmxMBeanPropertyCache.getKeyPropertyList(mbeanName),
                    new LinkedList<String>(),
                    attr.getName(),
                    attr.getType(),
                    attr.getDescription(),
                    attribute.getValue()
            );
        }
}

scrapeBean方法会变量对象的每个属性，将属性信息交给processBeanValue方法处理，processBeanValue是一个递归方法，如果属性不是基本类型，可进一步解析出下一层属性。属性信息最终是通过receiver.recordBean方法转化为Sample对象添加入MetricFamilySamples中。

public void recordBean(
          String domain,
          LinkedHashMap<String, String> beanProperties,
          LinkedList<String> attrKeys,
          String attrName,
          String attrType,
          String attrDescription,
          Object beanValue) {

        String beanName = domain + angleBrackets(beanProperties.toString()) + angleBrackets(attrKeys.toString());
        // attrDescription tends not to be useful, so give the fully qualified name too.
        String help = attrDescription + " (" + beanName + attrName + ")";
        String attrNameSnakeCase = toSnakeAndLowerCase(attrName);

        for (Rule rule : config.rules) {
          Matcher matcher = null;
          String matchName = beanName + (rule.attrNameSnakeCase ? attrNameSnakeCase : attrName);
          if (rule.pattern != null) {
            matcher = rule.pattern.matcher(matchName + ": " + beanValue);
            if (!matcher.matches()) {
              continue;
            }
          }

          Number value;
          if (rule.value != null && !rule.value.isEmpty()) {
            String val = matcher.replaceAll(rule.value);

            try {
              beanValue = Double.valueOf(val);
            } catch (NumberFormatException e) {
              LOGGER.fine("Unable to parse configured value '" + val + "' to number for bean: " + beanName + attrName + ": " + beanValue);
              return;
            }
          }
          if (beanValue instanceof Number) {
            value = ((Number)beanValue).doubleValue() * rule.valueFactor;
          } else if (beanValue instanceof Boolean) {
            value = (Boolean)beanValue ? 1 : 0;
          } else {
            LOGGER.fine("Ignoring unsupported bean: " + beanName + attrName + ": " + beanValue);
            return;
          }

          // If there's no name provided, use default export format.
          if (rule.name == null) {
            defaultExport(domain, beanProperties, attrKeys, rule.attrNameSnakeCase ? attrNameSnakeCase : attrName, help, value, rule.type);
            return;
          }

          // Matcher is set below here due to validation in the constructor.
          String name = safeName(matcher.replaceAll(rule.name));
          if (name.isEmpty()) {
            return;
          }
          if (config.lowercaseOutputName) {
            name = name.toLowerCase();
          }

          // Set the help.
          if (rule.help != null) {
            help = matcher.replaceAll(rule.help);
          }

          // Set the labels.
          ArrayList<String> labelNames = new ArrayList<String>();
          ArrayList<String> labelValues = new ArrayList<String>();
          if (rule.labelNames != null) {
            for (int i = 0; i < rule.labelNames.size(); i++) {
              final String unsafeLabelName = rule.labelNames.get(i);
              final String labelValReplacement = rule.labelValues.get(i);
              try {
                String labelName = safeName(matcher.replaceAll(unsafeLabelName));
                String labelValue = matcher.replaceAll(labelValReplacement);
                if (config.lowercaseOutputLabelNames) {
                  labelName = labelName.toLowerCase();
                }
                if (!labelName.isEmpty() && !labelValue.isEmpty()) {
                  labelNames.add(labelName);
                  labelValues.add(labelValue);
                }
              } catch (Exception e) {
                throw new RuntimeException(
                  format("Matcher '%s' unable to use: '%s' value: '%s'", matcher, unsafeLabelName, labelValReplacement), e);
              }
            }
          }

          // Add to samples.
          LOGGER.fine("add metric sample: " + name + " " + labelNames + " " + labelValues + " " + value.doubleValue());
          addSample(new MetricFamilySamples.Sample(name, labelNames, labelValues, value.doubleValue()), rule.type, help);
          return;
        }
      }

在处理属性信息的过程中会根据配置中rule定义的正则表达式去过滤不需要的信息。

4、 在添加mx_scrape_duration_seconds（采集时间），jmx_scrape_error（是否采集成功）两个标量。

（3）默认收集器

注册一些系统级别的采集器，这些采集器用于采集一些通用指标

DefaultExports.initialize();
public static synchronized void initialize() {
        if (!initialized) {
            (new StandardExports()).register();
            (new MemoryPoolsExports()).register();
            (new BufferPoolsExports()).register();
            (new GarbageCollectorExports()).register();
            (new ThreadExports()).register();
            (new ClassLoadingExports()).register();
            (new VersionInfoExports()).register();
            initialized = true;
        }

}

默认收集器及其指标

StandardExports：

• process_cpu_seconds_total
• process_start_time_seconds
• process_open_fds
• process_max_fds

MemoryPoolsExports：

• jvm_memory_bytes_used
• jvm_memory_bytes_committed
• jvm_memory_bytes_max
• jvm_memory_bytes_init
• jvm_memory_pool_bytes_used
• jvm_memory_pool_bytes_committed
• jvm_memory_pool_bytes_max
• jvm_memory_pool_bytes_init

BufferPoolsExports：

• jvm_buffer_pool_used_bytes
• jvm_buffer_pool_capacity_bytes
• jvm_buffer_pool_used_buffers

GarbageCollectorExports：

• jvm_gc_collection_seconds

ThreadExports：

• jvm_threads_current
• jvm_threads_daemon
• jvm_threads_peak
• jvm_threads_started_total
• jvm_threads_deadlocked
• jvm_threads_deadlocked_monitor

ClassLoadingExports：

• jvm_classes_loaded
• jvm_classes_loaded_total
• jvm_classes_unloaded_total

VersionInfoExports：

• jvm_info

收集器注册流程

截止目前，已经介绍了jmx_exporter中所有的收集器了，我们接着看这些收集器如何注册的，因此需要进入register方法中：
io.prometheus.client.Collector#register()

public <T extends Collector> T register() {
        return this.register(CollectorRegistry.defaultRegistry);
}
		public <T extends Collector> T register(CollectorRegistry registry) {
        registry.register(this);
        return this;
}

	public void register(Collector m) {
        List<String> names = this.collectorNames(m);
        Map var3 = this.collectorsToNames;
        synchronized(this.collectorsToNames) {
            Iterator var4 = names.iterator();

            String name;
            while(var4.hasNext()) {
                name = (String)var4.next();
                if (this.namesToCollectors.containsKey(name)) {
                    throw new IllegalArgumentException("Collector already registered that provides name: " + name);
                }
            }

            var4 = names.iterator();

            while(var4.hasNext()) {
                name = (String)var4.next();
                this.namesToCollectors.put(name, m);
            }

            this.collectorsToNames.put(m, names);
        }
    }

收集器的最终是注册到

• io.prometheus.client.CollectorRegistry#collectorsToNames
• io.prometheus.client.CollectorRegistry#namesToCollectors
  中的，collectorsToNames维护了收集器到指标名称的映射，namesToCollectors维护了指标名称到收集器的映射。

1、 首先通过collectorNames去获取这个收集器收集标量的名称列表 names

2、 看namesToCollectors中是否已经有names的标量名称了，如果有说明这个收集器已经注册过了，抛异常。

3、 将收集器信息及其对应的标量信息添加到collectorsToNames，namesToCollectors中。

启动服务

server = new HTTPServer(socket, CollectorRegistry.defaultRegistry, true);

public HTTPServer(InetSocketAddress addr, CollectorRegistry registry, boolean daemon) throws IOException {
        this.server = HttpServer.create();
        this.server.bind(addr, 3);
        HttpHandler mHandler = new HTTPServer.HTTPMetricHandler(registry);
        this.server.createContext("/", mHandler);
        this.server.createContext("/metrics", mHandler);
        this.executorService = Executors.newFixedThreadPool(5, HTTPServer.DaemonThreadFactory.defaultThreadFactory(daemon));
        this.server.setExecutor(this.executorService);
        HttpHandler mHandler = new HTTPServer.HTTPMetricHandler(registry);
}

这里主要注意两点：
1、 注册器registry被处理http请求的handler持有了HttpHandler mHandler = new HTTPServer.HTTPMetricHandler(registry);
2、 服务通过HttpHandler mHandler = new HTTPServer.HTTPMetricHandler(registry);启动对外提供服务。

服务启动之后，就可以通过http协议去查询指标信息了。而服务的请求则是由mHandler处理的，下面我们看看请求是如何获取指标信息的

public void handle(HttpExchange t) throws IOException {
            String query = t.getRequestURI().getRawQuery();
            ByteArrayOutputStream response = (ByteArrayOutputStream)this.response.get();
            response.reset();
            OutputStreamWriter osw = new OutputStreamWriter(response);
            TextFormat.write004(osw, this.registry.filteredMetricFamilySamples(HTTPServer.parseQuery(query)));
            osw.flush();
            osw.close();
            response.flush();
            response.close();
            t.getResponseHeaders().set("Content-Type", "text/plain; version=0.0.4; charset=utf-8");
            t.getResponseHeaders().set("Content-Length", String.valueOf(response.size()));
            if (HTTPServer.shouldUseCompression(t)) {
                t.getResponseHeaders().set("Content-Encoding", "gzip");
                t.sendResponseHeaders(200, 0L);
                GZIPOutputStream os = new GZIPOutputStream(t.getResponseBody());
                response.writeTo(os);
                os.finish();
            } else {
                t.sendResponseHeaders(200, (long)response.size());
                response.writeTo(t.getResponseBody());
            }

            t.close();
        }

可以看到这里调用了收集器注册器的方法registry.filteredMetricFamilySamples(HTTPServer.parseQuery(query))

public Enumeration<MetricFamilySamples> filteredMetricFamilySamples(Set<String> includedNames) {
        return new CollectorRegistry.MetricFamilySamplesEnumeration(includedNames);
}

MetricFamilySamplesEnumeration(Set<String> includedNames) {
            this.includedNames = includedNames;
            this.collectorIter = this.includedCollectorIterator(includedNames);
            this.findNextElement();
        }

this.includedCollectorIterator(includedNames)根据查询条件获取收集器，然后进入下一步this.findNextElement()

private void findNextElement() {
            this.next = null;

            while(this.metricFamilySamples != null && this.metricFamilySamples.hasNext()) {
                this.next = this.filter((MetricFamilySamples)this.metricFamilySamples.next());
                if (this.next != null) {
                    return;
                }
            }

            if (this.next == null) {
                while(this.collectorIter.hasNext()) {
                    this.metricFamilySamples = ((Collector)this.collectorIter.next()).collect().iterator();

                    while(this.metricFamilySamples.hasNext()) {
                        this.next = this.filter((MetricFamilySamples)this.metricFamilySamples.next());
                        if (this.next != null) {
                            return;
                        }
                    }
                }
            }

        }

1、 用collectorIter中的收集器去获取标量信息
2、 根据includedNames去找到请求需要的标量信息。

这样第三方就可以通过http的方式来获取指标信息了。

至此，jmx_exporter的源码分析完毕！