logstash java应用 logstash java插件

转载

cnolnic 2023-11-01 15:48:34

文章标签 logstash java应用 java ruby bundle 文章分类 Java 后端开发

标题中bug：在最后按照离线包处说明，给出解决方式

公司需要使用最新版logstash来开发，我们自己开放性框架

github下载对应源码：Logstash 7.10.1 github 地址

解压windows 直接解压/linux unzip xxxx.zip
生成java插件所需api 核心 jar 包文件：进入项目根目录执行-windows:gradlew.bat assemble linux：./gradlew assemble

java 示列插件git地址：logstash-input-java_input_example

plugin demo 根目录，增加java.properties 配置文件 gradle.properties，添加值：LOGSTASH_CORE_PATH=/data/logstashcpl/logstash-7.10.1/logstash-core 该核心路径指向上一步中核心jar包路径
打包插件：./gradlew gem

在引入 Logstash 官网下载地址下载了 logstash-7.10.1

解压即可使用，进入logstash根目录
安装上面打包的插件：bin/logstash-plugin install --no-verify --local 你打包出来的插件

bin/logstash-plugin install --no-verify --local /data/plugins/logstash-input-java_input_example-master/logstash-input-java_input_example-1.0.1.gem

bin/logstash -e "input {java_input_example {}} output {stdout { codec => rubydebug }}"

查看该插件在logstash中的名称

[root@VM_centos logstash-7.10.1]# bin/logstash-plugin list
Using JAVA_HOME defined java: /data/java/jdk1.8.0_271/
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
....
logstash-input-imap
logstash-input-java_input_example
logstash-input-jm
.....

#logstash-input-java_input_example 即为插件名

导出该插件用于离线环境安装（我司内网，没办法在内网编译安装）bin/logstash-plugin prepare-offline-pack --overwrite --output [包名] 插件名，上一步获取的插件名即可

#打离线包
[root@VM_centos logstash-7.10.1]# bin/logstash-plugin prepare-offline-pack --overwrite --output /data/java_input.zip logstash-input-java_input_example
Using JAVA_HOME defined java: /data/java/jdk1.8.0_271/
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
Offline package created at: /data/java_input.zip

You can install it with this command `bin/logstash-plugin install file:///data/java_input.zip`

安装该离线包：bin/logstash-plugin install file:///data/java_input.zip

我在执行上述步骤是遇到问题如下：

demo 插件问题：该错误来源：

No signature of method: org.gradle.api.internal.tasks.DefaultTaskDependency $TaskDependencySet.getAt() is applicable for argument types: (ArrayList) values:

dependsOn [downloadAndInstallJRuby, removeObsoleteJars, vendor, generateRubySupportFiles] 修改如下

dependsOn([downloadAndInstallJRuby, removeObsoleteJars, vendor, generateRubySupportFiles])

demo插件问题：

/data/plugins/logstash-input-java_input_example-master/src/main/java/org/logstashplugins/JavaInputExample.java:3: error: package co.elastic.logstash.api does not exist
import co.elastic.logstash.api.Configuration;

　　该错误原因为build.gradle 去匹配核心jar包时未匹配到，因为demo中

implementation fileTree(dir: LOGSTASH_CORE_PATH, include: "**/logstash-core-?.?.?.jar")
#logstash-core-?.?.?.jar 中 ? 匹配一个字符，一旦 x.y.z.jar包中版本超过9，达到两个字符就会匹配不到
#修改为：
implementation fileTree(dir: LOGSTASH_CORE_PATH, include: "**/logstash-core-?.??.?.jar") #或指定你生成的jar包版本

安装离线包：

ERROR: Something went wrong when installing file:///data//plugins/java-input.zip, message: Illformed requirement ["=java_input_example"]

　　该错误是由于默认命名插件时以 java开头导致：build.gradle 文件中

pluginInfo.pluginType      = "input"
pluginInfo.pluginClass     = "JavaInputExample"
pluginInfo.pluginName      = "java_input_example" //此处命名java开头导致

分析安装离线包导致的该问题：

根目录搜索该报错信息：

[root@VM_centos logstash-7.10.1]# grep Illformed -nr ./*
./vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:110:      raise BadRequirementError, "Illformed requirement [#{obj.inspect}]"
[root@VM_63_147_centos logstash-7.10.1]# vim ./vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb +110

输出堆栈信息：puts caller

def self.parse(obj)
    return ["=", obj] if Gem::Version === obj

    unless PATTERN =~ obj.to_s
      puts caller
      raise BadRequirementError, "Illformed requirement [#{obj.inspect}]"
    end

    if $1 == ">=" && $2 == "0"
      DefaultRequirement
    else
      [$1 || "=", Gem::Version.new($2)]
    end
  end

　堆栈信息如下

Installing file: /data/plugins/java-input.zip
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:140:in `block in initialize'
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:140:in `map!'
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:140:in `initialize'
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:65:in `create'
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/dependency.rb:60:in `initialize'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/bundler-1.17.3/lib/bundler/dependency.rb:82:in `initialize'
/data/logstash/logstash-7.10.1/lib/pluginmanager/bundler/logstash_injector.rb:65:in `dependency'
/data/logstash/logstash-7.10.1/lib/pluginmanager/bundler/logstash_injector.rb:52:in `collect'
/data/logstash/logstash-7.10.1/lib/pluginmanager/bundler/logstash_injector.rb:52:in `inject!'
/data/logstash/logstash-7.10.1/lib/pluginmanager/pack_installer/local.rb:58:in `execute'
/data/logstash/logstash-7.10.1/lib/pluginmanager/install.rb:47:in `execute'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:68:in `run'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/subcommand/execution.rb:11:in `execute'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:68:in `run'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:133:in `run'
/data/logstash/logstash-7.10.1/lib/pluginmanager/main.rb:64:in `<main>'
ERROR: Something went wrong when installing file:///data/plugins/java-input.zip, message: Illformed requirement ["=java_input_example"]

　通过上述信息定位到：

vim ./lib/pluginmanager/pack_installer/pack.rb +65

class Pack
    class GemInformation
      EXTENSION = ".gem"
      SPLIT_CHAR = "-"
      JAVA_PLATFORM_RE = /-java/
      DEPENDENCIES_DIR_RE = /dependencies/

      attr_reader :file, :name, :version, :platform

      def initialize(gem)
        @file = gem
        extracts_information
      end

      def dependency?
        @dependency
      end

      def plugin?
        !dependency?
      end

      private
      def extracts_information
        basename = ::File.basename(file, EXTENSION)
        parts = basename.split(SPLIT_CHAR)
        @dependency = ::File.dirname(file) =~ DEPENDENCIES_DIR_RE
        puts "basename: #{basename}"
        puts "parts: #{parts}"
        if basename.match(JAVA_PLATFORM_RE)
          @platform = parts.pop
          @version = parts.pop
          @name = parts.join(SPLIT_CHAR)
        else
          @platform = nil
          @version = parts.pop
          @name = parts.join(SPLIT_CHAR)
        end
      end
    end

　输出：basename、parts 可以分析出，因为打包名为 logstash-input-java_input_example-1.0.1 而 logstash 安装源码中，使用了

JAVA_PLATFORM_RE = /-java/

if basename.match(JAVA_PLATFORM_RE) 此处匹配到了，/-java/ 认为打包平台为 java 不是 ruby 导致

[root@VM_centos logstash-7.10.1]# bin/logstash-plugin install file:///data/plugins/java-input.zip
...
arguments: ["install", "file:///data/plugins/java-input.zip"]
subcmd_class: LogStash::PluginManager::Install
arguments: ["file:///data/plugins/java-input.zip"]
Installing file: /data/plugins/java-input.zip
basename: logstash-input-java_input_example-1.0.1

解决方案就比较多了，当然不建议修改源码，

直接修改插件名，别用 java 开头就好了

多说一点。

楼主排查的时候怀疑是不是打包就有问题，少打了 -java 参数到包名中去，捋了好久，具体过程就不说了，看结果吧

使用 bin/logstash-plugin prepare-offline-pack 去打包的话

[root@VM_centos logstash-7.10.1]# grep -rn "my collect:" ./
./vendor/bundle/jruby/2.5.0/gems/paquet-0.2.1/lib/paquet/gem.rb:32:      puts "my collect: #{collect_required_gems}"
[root@VM_centos logstash-7.10.1]# vim ./vendor/bundle/jruby/2.5.0/gems/paquet-0.2.1/lib/paquet/gem.rb +32

module Paquet
  class Gem
    RUBYGEMS_URI = "https://rubygems.org/downloads"

    attr_reader :gems, :ignores

    def initialize(target_path, cache = nil)
      @target_path = target_path
      @gems = []
      @ignores = []
      @cache = cache
    end

    def add(name)
      @gems << name
    end

    def ignore(name)
      @ignores << name
    end

    def pack
      Paquet::ui.info("Cleaning existing target path: #{@target_path}")

      FileUtils.rm_rf(@target_path)
      FileUtils.mkdir_p(@target_path)
      puts "sq collect: #{collect_required_gems}"
      package_gems(collect_required_gems)
    end

输出打包命令去收集所需gems时参数都为：

[root@VM_centos logstash-7.10.1]# bin/logstash-plugin prepare-offline-pack --overwrite --output /data/plugins/java-input.zip logstash-input-java_input_example
Using JAVA_HOME defined java: /data/java/jdk1.8.0_271/
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
arguments: ["prepare-offline-pack", "--overwrite", "--output", "/data/plugins/java-input.zip", "logstash-input-java_input_example"]
subcmd_class: LogStash::PluginManager::PrepareOfflinePack
arguments: ["--overwrite", "--output", "/data/plugins/java-input.zip", "logstash-input-java_input_example"]
my test: ["logstash-input-java_input_example"]
my collect: [#<Paquet::Dependency:0x51d76ad3 @version=#<Gem::Version "1.0.1">, @name="logstash-input-java_input_example", @platform="ruby">]

可以发现上面最后一行 collect 收集参数中 platform 就是ruby 不是 java
至此楼主也懒得研究了，研究这几天炸了。
反正 logstash 官网这插件 demo 质量太差，然后处理打包上，源码感觉也不够严谨，导致了这样的bug，分割字符串，正则匹配太SX了

最后楼主将修改后打出的包放到内网成功如下：

SIS3.0.45.0 /data/logstash-7.10.1 # bin/logstash-plugin install file:///data/java-input.zip
Using JAVA_HOME defined java: /data/share/java/jdk
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK

Installing file: /data/java-input.zip
Install successful
SIS3.0.45.0 /home/fantom/party/logstash-7.10.1 #
SIS3.0.45.0 /data/logstash-7.10.1 # bin/logstash -e "input {input_example_java {}} output {stdout { codec => rubydebug }}"
Using JAVA_HOME defined java: /data/java/jdk
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK

Sending Logstash logs to /data/logstash-7.10.1/logs which is now configured via log4j2.properties
[2020-12-21T21:14:01,451][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.10.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.131-b11 on 1.8.0_131-b11 +indy +jit [linux-x86_64]"}
[2020-12-21T21:14:01,858][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/data/logstash-7.10.1/data/queue"}
[2020-12-21T21:14:01,878][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/home/fantom/party/logstash-7.10.1/data/dead_letter_queue"}
[2020-12-21T21:14:02,485][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-12-21T21:14:02,529][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"a33c9423-70fa-4794-9995-f02bc72e2192", :path=>"/home/fantom/party/logstash-7.10.1/data/uuid"}
[2020-12-21T21:14:05,000][INFO ][org.reflections.Reflections] Reflections took 88 ms to scan 1 urls, producing 23 keys and 47 values
[2020-12-21T21:14:08,510][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x21cc9c5e run>"}
[2020-12-21T21:14:11,199][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>2.66}
[2020-12-21T21:14:11,252][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-12-21T21:14:11,432][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
    "@timestamp" => 2020-12-21T13:14:11.308Z,
       "message" => "message        1 of 3       ",
      "@version" => "1"
}
{
    "@timestamp" => 2020-12-21T13:14:11.319Z,
       "message" => "message        2 of 3       ",
      "@version" => "1"
}
{
    "@timestamp" => 2020-12-21T13:14:11.319Z,
       "message" => "message        3 of 3       ",
      "@version" => "1"
}
[2020-12-21T21:14:12,511][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2020-12-21T21:14:13,048][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2020-12-21T21:14:13,060][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-12-21T21:14:13,450][INFO ][logstash.runner          ] Logstash shut down.

人生还有意义。那一定是还在找存在的理由

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。