使用supervisor 管理logstash
supervisor和logstash此处不做介绍，直接进行操作配置
环境准备
需要创建一个普通用户，如jianxp以下均使用该用户部署服务，设用户目录为 /home/jianxp，安装目录为 /opt
准备部署包 jdk/supervisor/logstash 上传至服务器，默认在 /tools 目录下，解压到/opt目录下。
JDK安装
-
若 已安装 jdk1.8.0，则只需要建立已有JDK的软连接，可跳过后面的步骤：
mkdir -p /opt/java ln -sf ${JAVA_HOME} /opt/java/jdk -
解压安装包：
tar zxf jdk-8u191-linux-x64.tar.gz -C /opt ln -s /opt/jdk1.8.0_191 /opt/jdk -
添加环境变量：
编辑文件
vi ~/.bash_profile，追加如下环境变量：export JAVA_HOME=/opt/jdk export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar -
执行命令
source ~/.bash_profile使环境变量生效。 -
执行命令
java -version，若输出java version "1.8.0_191"，则说明jdk安装成功
Supervisor安装
-
离线安装supervisor，解压安装包
tar -zxf /tools/supervisor-3.3.4.tar -C /opt/，依次执行下列步骤cd /opt/supervisor-3.3.4 tar -zxf setuptools-32.3.1.tar.gz && cd setuptools-32.3.1 python setup.py install --user cd .. && tar -zxf meld3-1.0.2.tar.gz && cd meld3-1.0.2 python setup.py install --user cd .. && tar -zxf supervisor-3.3.4.tar.gz && cd supervisor-3.3.4 python setup.py install --user -
编辑supervisor配置文件
mkdir -p /opt/supervisor && cd /opt/supervisor && vi supervisord.confsupervisord.conf 文件内容如下
[unix_http_server] file=/opt/supervisor/supervisor.sock ; (the path to the socket file) chmod=0700 ; sockef file mode (default 0700) [supervisord] logfile=/opt/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log) pidfile=/opt/supervisor/supervisord.pid ; (supervisord pidfile;default supervisord.pid) childlogdir=/opt/supervisor ; ('AUTO' child log dir, default $TEMP) ; the below section must remain in the config file for RPC ; (supervisorctl/web interface) to work, additional interfaces may be ; added by defining them in separate rpcinterface: sections [rpcinterface:supervisor] supervisor.rpcinterface_factory=supervisor.rpcinterface:make_main_rpcinterface [supervisorctl] serverurl=unix:///opt/supervisor/supervisor.sock ; use a unix:// URL for a unix socket ; The [include] section can just contain the "files" setting. This ; setting can list multiple files (separated by whitespace or ; newlines). It can also contain wildcards. The filenames are ; interpreted as relative to this file. Included files *cannot* ; include files themselves. [include] files = /opt/supervisor/conf.d/*.conf -
创建配置文件里涉及到的目录
-
日志目录
mkdir -p /opt/supervisor && mkdir -p /opt/supervisor -
管理进程的配置文件目录：
mkdir -p /opt/supervisor/conf.d
-
-
建立软连接
mkdir -p ~/.local/etc ln -s /opt/supervisor/supervisord.conf ~/.local/etc/ -
至此，supervisor安装完毕
- 执行
supervisod即可启动supervisor - 执行
supervisorctl status，无报错则说明启动成功。
- 执行
logstash 安装
解压安装包，添加软连接
tar -zxf /tools/logstash-6.8.9-fix.tar.gz -C /opt/
ln -s /opt/logstash-6.8.9 /opt/logstash
配置logstash用supervisor管理（logstash的所有配置均在supervisor管理的配置文件中完成）
编辑配置文件，执行 cd /opt/supervisor/conf.d && vi logstash.conf
; logstash config file
[program:logstash]
command=/opt/logstash-6.8.9/bin/logstash
directory=/opt/logstash-6.8.9/
user=jianxp
numprocs=1
stdout_logfile=/opt/logstash/logs/stderr.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=1
redirect_stderr=true
autostart=true
autorestart=true
startsecs=1
stopwaitsecs=1
killasgroup=true
priority=1
说明
command=/opt/logstash-6.8.9/bin/logstash 这个没有加-f等参数，默认读config/pipelines.yaml文件，此文件也是后续要配置的。
在/opt/logstash-6.8.9/config/conf.d 创建两个config文件，作为logstash的pipeline配置
vim config/conf.d/test.conf
input{
file{
path =>"/opt/logstash_test/test.log"
start_position=>"beginning"
}
}
output {
file {
path =>"/opt/logstash_test/output.log"
}
}
vim config/conf.d/file.conf
input{
file{
path =>"/opt/logstash_test/test.log"
start_position=>"beginning"
}
}
output {
file {
path =>"/opt/logstash_test/output33.log"
}
}
上面两个pipeline时比较简单的，以文件输入和和文件输出为例子，两个pipeline都是以文件test.log为共同输入，然后输出到不同的文件中。
编辑config/pipelines.yaml 文件，添加如下内容
- pipeline.id: file
pipeline.workers: 1
pipeline.batch.size: 1
path.config: "/opt/logstash-6.8.9/config/conf.d/file.conf"
- pipeline.id: test
pipeline.workers: 1
pipeline.batch.size: 1
path.config: "/opt/logstash-6.8.9/config/conf.d/test.conf"
上面的logstash的pipeline配置完成后，并且supervisor目录下的logstash.conf也配置完成，既可以使用supervisor启动logstash
supervisorctl update
supervisorctl status
在/opt/logstash-6.8.9/logs 目录下，可以查看logstash输出的日志，通过日志可以看到，Pipelines running count=2，表示启动两个pipeline。
可以开启另外一个shell窗口，向test.log文件中增加内容，log中也会输出Opening file 等信息。
[2021-03-10T16:18:04,953][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"test", :thread=>"#<Thread:0x24bd4c10 run>"}
[2021-03-10T16:18:04,958][INFO ][logstash.runner ] Logstash shut down.
[2021-03-10T16:18:42,012][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.8.9"}
[2021-03-10T16:18:47,360][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"file", "pipeline.workers"=>1, "pipeline.batch.size"=>1, "pipeline.batch.delay"=>50}
[2021-03-10T16:18:47,721][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"test", "pipeline.workers"=>1, "pipeline.batch.size"=>1, "pipeline.batch.delay"=>50}
[2021-03-10T16:18:47,940][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/opt/logstash-6.8.9/data/plugins/inputs/file/.sincedb_513b5c3d29a45c373b60caa3e09938d7", :path=>["/opt/logstash_test/test.log"]}
[2021-03-10T16:18:47,940][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/opt/logstash-6.8.9/data/plugins/inputs/file/.sincedb_513b5c3d29a45c373b60caa3e09938d7", :path=>["/opt/logstash_test/test.log"]}
[2021-03-10T16:18:47,992][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"file", :thread=>"#<Thread:0x5c0e2cd6 run>"}
[2021-03-10T16:18:47,993][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"test", :thread=>"#<Thread:0x6626055f run>"}
[2021-03-10T16:18:48,074][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2021-03-10T16:18:48,074][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2021-03-10T16:18:48,108][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:file, :test], :non_running_pipelines=>[]}
[2021-03-10T16:18:48,531][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-03-10T16:19:11,774][INFO ][logstash.outputs.file ] Opening file {:path=>"/opt/logstash_test/output.log"}
[2021-03-10T16:19:11,776][INFO ][logstash.outputs.file ] Opening file {:path=>"/opt/logstash_test/output33.log"}
查看输出的文件
[jianxp@localhost logstash_test]$ ll
总用量 16
-rw-rw-r--. 1 jianxp jianxp 335 3月 10 15:23 aa
-rw-r--r--. 1 jianxp jianxp 3338 3月 10 16:19 output33.log
-rw-r--r--. 1 jianxp jianxp 3338 3月 10 16:19 output.log
-rw-rw-r--. 1 jianxp jianxp 1340 3月 10 16:19 test.log
