实时数仓springboot日志采集 springboot日志原理

springboot日志源码解析

1.常见日志框架

• JUL，JCL，Jboss-loging,logback,log4j,log4j2,self4j
  这里我们只介绍springboot默认的日志框架，抽象层是SLF4J，实现层是logback。当然springboot还可以使用其他众多的日志框架来实现日志的输出。那为什么springboot推荐我们使用logback呢？原因是logback是新一代的日志框架，她的执行效率更高，而且可以适应诸多的环境，还支持slf4j.

2 ,springboot使用logback

• 引入日志依赖

<!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-logging -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-logging</artifactId>
    <version>2.3.4.RELEASE</version>
</dependency>

此处注意：其实在springboot的父依赖里就已经依赖了starter-logging了，所以这里可以不用引入。

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-idavAlmI-1617503401179)(C:\Users\asus.DESKTOP-9D7DVCG\AppData\Roaming\Typora\typora-user-images\1617457033106.png)]

springboot默认的日志级别是info，也就是管理员级别的。

• 日志级别分为：TRACE, DEBUG, INFO, WARN, ERROR, FATAL, OFF
• 从上边控制台输出可以看到:时间 级别 进程ID -------线程名 ：日志内容

3，更换输出日志级别,输出路基等

• 修改application.properties文件

#修改日志的级别，默认root是info
#logging.level.root=debug

# 不指定路径在当前项目下生成springboot.log日志
#logging.file=springboot.log
# 可以指定完整的路径；
logging.file=d://springboot.log

# 在当前磁盘的根路径下创建spring文件夹和里面的log文件夹；使用spring.log 作为默认文件
logging.path=/spring/log

# 在控制台输出的日志的格式
logging.pattern.console=%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n
# 指定文件中日志输出的格式
logging.pattern.file=%d{yyyy-MM-dd} === [%thread] === %-5level === %logger{50} ==== %msg%n

• 日志输出格式

日志输出格式：
%d表示日期时间，
%thread表示线程名，
%-5level：级别从左显示5个字符宽度
%logger{50} 表示logger名字最长50个字符，否则按照句点分割。 
%msg：日志消息，
%n是换行符

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-EHcXG4mN-1617503401181)(C:\Users\asus.DESKTOP-9D7DVCG\AppData\Roaming\Typora\typora-user-images\1617457473352.png)]

4.使用代码在控制台输出业务日志

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequestMapping("/testlogging")
public class LoggingTestController {
    private Logger logger = LoggerFactory.getLogger(this.getClass());
    @GetMapping("/hello")
    public String hello() {
        logger.info("test logging...");
        return "hello";
    }
}

注意这里的logger 包都是log4j的包

5 .自定义日志输出

• 1.在全局配置文件里配置

# 日志配置
# 指定具体包的日志级别
logging.level.net.codingme=debug
# 控制台和日志文件输出格式
logging.pattern.console=%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n
logging.pattern.file=%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n
# 日志文件大小
logging.file.max-size=10MB
# 保留的日志时间
logging.file.max-history=10
# 日志输出路径，默认文件spring.log
logging.path=systemlog
#logging.file=log.log

• 2.自定义配置文件，会喝springboot原有的配置一起协同起作用

##自定义配置文件名称
logging.config=classpath:logback-black.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <contextName>SpringBoot-logBack-demo</contextName>
    <!-- 文件输出格式 -->
    <property name="PATTERN" value="%-12(%d{yyyy-MM-dd HH:mm:ss.SSS}) |-%-5level %contextName [%thread] %c [%L] -| %msg%n" />
    <!-- test文件路径 -->
    <property name="TEST_FILE_PATH" value="F:/WORKSPACE/IDEA_WORKSPACE/MICRO_COURSES/springBootLogging/target/logs" />
    <!-- pro文件路径 -->
    <property name="PRO_FILE_PATH" value="F:/WORKSPACE/IDEA_WORKSPACE/MICRO_COURSES/springBootLogging/target/logs" />

    <property name="LOG_FILE_SIZE" value="100MB"/>
    <!-- 每天产生一个文件 -->
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <charset>UTF-8</charset>
            <pattern>${PATTERN}</pattern>
        </encoder>
    </appender>
    <!--对应info级别，文件名以info-xxx.log形式命名,每天产生一个文件-->
    <appender name="INFOFILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <!--日志名称，如果没有File 属性，那么只会使用FileNamePattern的文件路径规则
            如果同时有<File>和<FileNamePattern>，那么当天日志是<File>，明天会自动把今天
            的日志改名为今天的日期。即，<File> 的日志都是当天的。
        -->
        <file>${TEST_FILE_PATH}/info.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!-- 文件名称 -->
            <fileNamePattern>${TEST_FILE_PATH}/info.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <!-- 只保留最近30天的日志 -->
            <MaxHistory>30</MaxHistory>
            <!-- 除按日志记录之外，还配置了日志文件不能超过2M，若超过2M，日志文件会以索引0开始，命名日志文件，例如info-2018-06-10.0.log -->
            <TimeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>${LOG_FILE_SIZE}</maxFileSize>
            </TimeBasedFileNamingAndTriggeringPolicy>
            <!--用来指定日志文件的上限大小，那么到了这个值，就会删除旧的日志-->
            <!--<totalSizeCap>1GB</totalSizeCap>-->
        </rollingPolicy>

        <!--<layout class="ch.qos.logback.classic.PatternLayout">
            <pattern>${PATTERN}</pattern>
        </layout>-->
        <encoder>
            <charset>UTF-8</charset>
            <pattern>${PATTERN}</pattern>
        </encoder>
        <!--如果只想要INFO级别的日志，使用如下策略即可只输出INFO级别的日志-->
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <!--过滤INFO-->
            <level>INFO</level>
            <!--匹配到就允许-->
            <onMatch>ACCEPT</onMatch>
            <!--没有匹配到就允许-->
            <onMismatch>DENY</onMismatch>
        </filter>
    </appender>
    <!--对应info级别，文件名以error-xxx.log形式命名 每天产生一个文件-->
    <appender name="ERRORFILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <!--日志名称，如果没有File 属性，那么只会使用FileNamePattern的文件路径规则
            如果同时有<File>和<FileNamePattern>，那么当天日志是<File>，明天会自动把今天
            的日志改名为今天的日期。即，<File> 的日志都是当天的。
        -->
        <file>${TEST_FILE_PATH}/error.log</file>
        <!--滚动策略，按照时间滚动 TimeBasedRollingPolicy-->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!-- 文件名称 -->
            <fileNamePattern>${TEST_FILE_PATH}/error.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <!-- 只保留最近30天的日志 -->
            <MaxHistory>30</MaxHistory>
            <!-- 除按日志记录之外，还配置了日志文件不能超过2M，若超过2M，日志文件会以索引0开始，命名日志文件，例如error-2018-06-10.0.log -->
            <TimeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>${LOG_FILE_SIZE}</maxFileSize>
            </TimeBasedFileNamingAndTriggeringPolicy>
            <!--用来指定日志文件的上限大小，那么到了这个值，就会删除旧的日志-->
            <!--<totalSizeCap>1GB</totalSizeCap>-->
        </rollingPolicy>

        <!--<layout class="ch.qos.logback.classic.PatternLayout">
            <pattern>${PATTERN}</pattern>
        </layout>-->
        <encoder>
            <charset>UTF-8</charset>
            <pattern>${PATTERN}</pattern>
        </encoder>
        <!--如果只想要ERROR级别的日志，使用如下策略即可只输出ERROR级别的日志-->
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <!--过滤INFO-->
            <level>ERROR</level>
            <!--匹配到就允许-->
            <onMatch>ACCEPT</onMatch>
            <!--没有匹配到就允许-->
            <onMismatch>DENY</onMismatch>
        </filter>
    </appender>
    <!--具体个性化某个Controller或者某个包下使用指定级别的日志输出-->
    <logger name="com.black.example.*" level="INFO"/>
    <logger name="org.springframework.web" level="DEBUG"/>
    <!--测试环境-->
    <springProfile name="test">
        <!--<root level="TRACE">-->
        <root level="INFO">
            <appender-ref ref="ERRORFILE" />
            <appender-ref ref="INFOFILE"/>
            <appender-ref ref="CONSOLE"/>
        </root>
    </springProfile>
    <!--开发环境-->
    <springProfile name="dev">
        <root level="DEBUG">
            <appender-ref ref="ERRORFILE" />
            <appender-ref ref="INFOFILE"/>
        </root>
    </springProfile>
    <!--生产环境-->
    <springProfile name="pro">
        <root level="ERROR">
            <appender-ref ref="ERRORFILE" />
            <appender-ref ref="INFOFILE"/>
        </root>
    </springProfile>

</configuration>

6。springboot日志源码解读

还记得springboot项目里我们只要在pom.xml引入spring-boot-starter-parent，父依赖也依赖于logging，所以在启动springboot时候，会自动打印info以上级别的日志，说明springboot已经办我们加载好了日志方面的配置等。

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Uon82xhD-1617503401185)(C:\Users\asus.DESKTOP-9D7DVCG\AppData\Roaming\Typora\typora-user-images\1617500605462.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-qEbGNwne-1617503401189)(C:\Users\asus.DESKTOP-9D7DVCG\AppData\Roaming\Typora\typora-user-images\1617500629549.png)]

在jar里找到springboot-》META_INF-》spring.factories

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-QBhxtsqN-1617503401191)(C:\Users\asus.DESKTOP-9D7DVCG\AppData\Roaming\Typora\typora-user-images\1617500688173.png)]

可以发现LoggingApplicationListener这个类就是帮我们加载日志方面的的罪魁祸首。从名字不难猜到其实他就是监听了SpringbootApplication的启动，然后触发的。下面我们来看看LoggingApplicationListener

public class LoggingApplicationListener implements GenericApplicationListener {

	private static final ConfigurationPropertyName LOGGING_LEVEL = ConfigurationPropertyName.of("logging.level");//默认日志级别level
	public static final String LOGGING_SYSTEM_BEAN_NAME = "springBootLoggingSystem";
private static final Map<String, List<String>> DEFAULT_GROUP_LOGGERS;
	static {//静态代码块表示可以配置输出的日志范围
		MultiValueMap<String, String> loggers = new LinkedMultiValueMap<>();
		loggers.add("web", "org.springframework.core.codec");
		loggers.add("web", "org.springframework.http");
		loggers.add("web", "org.springframework.web");
		loggers.add("web", "org.springframework.boot.actuate.endpoint.web");
		loggers.add("web", "org.springframework.boot.web.servlet.ServletContextInitializerBeans");
		loggers.add("sql", "org.springframework.jdbc.core");
		loggers.add("sql", "org.hibernate.SQL");
		loggers.add("sql", "org.jooq.tools.LoggerListener");
		DEFAULT_GROUP_LOGGERS = Collections.unmodifiableMap(loggers);
	}
	}
//进行一系列初始化环境加载操作，也就是把配置文件里配置的东西初始化
	protected void initialize(ConfigurableEnvironment environment, ClassLoader classLoader) {
		getLoggingSystemProperties(environment).apply();
		this.logFile = LogFile.get(environment);
		if (this.logFile != null) {
			this.logFile.applyToSystemProperties();
		}
		this.loggerGroups = new LoggerGroups(DEFAULT_GROUP_LOGGERS);
		initializeEarlyLoggingLevel(environment);
		initializeSystem(environment, this.loggingSystem, this.logFile);
		initializeFinalLoggingLevels(environment, this.loggingSystem);
		registerShutdownHookIfNecessary(environment, this.loggingSystem);
	}
	private void initializeSystem(ConfigurableEnvironment environment, LoggingSystem system, LogFile logFile) {
		String logConfig = StringUtils.trimWhitespace(environment.getProperty(CONFIG_PROPERTY));
		try {
			LoggingInitializationContext initializationContext = new LoggingInitializationContext(environment);
			if (ignoreLogConfig(logConfig)) {
				system.initialize(initializationContext, null, logFile);
			}
			else {
				system.initialize(initializationContext, logConfig, logFile);
			}
		}
		catch (Exception ex) {
			Throwable exceptionToReport = ex;
			while (exceptionToReport != null && !(exceptionToReport instanceof FileNotFoundException)) {
				exceptionToReport = exceptionToReport.getCause();
			}
			exceptionToReport = (exceptionToReport != null) ? exceptionToReport : ex;
			// NOTE: We can't use the logger here to report the problem
			System.err.println("Logging system failed to initialize using configuration from '" + logConfig + "'");
			exceptionToReport.printStackTrace(System.err);
			throw new IllegalStateException(ex);
		}
	}

	private boolean ignoreLogConfig(String logConfig) {
		return !StringUtils.hasLength(logConfig) || logConfig.startsWith("-D");
	}
//初始化日志级别
	private void initializeFinalLoggingLevels(ConfigurableEnvironment environment, LoggingSystem system) {
		bindLoggerGroups(environment);
		if (this.springBootLogging != null) {
			initializeSpringBootLogging(system, this.springBootLogging);
		}
		setLogLevels(system, environment);
	}

	private void bindLoggerGroups(ConfigurableEnvironment environment) {
		if (this.loggerGroups != null) {
			Binder binder = Binder.get(environment);
			binder.bind(LOGGING_GROUP, STRING_STRINGS_MAP).ifBound(this.loggerGroups::putAll);
		}
	}
	protected void initializeSpringBootLogging(LoggingSystem system, LogLevel springBootLogging) {
		BiConsumer<String, LogLevel> configurer = getLogLevelConfigurer(system);
		SPRING_BOOT_LOGGING_LOGGERS.getOrDefault(springBootLogging, Collections.emptyList())
				.forEach((name) -> configureLogLevel(name, springBootLogging, configurer));
	}

	/**
	 * Set logging levels based on relevant {@link Environment} properties.
	 * @param system the logging system
	 * @param environment the environment
	 */
	protected void setLogLevels(LoggingSystem system, ConfigurableEnvironment environment) {
		BiConsumer<String, LogLevel> customizer = getLogLevelConfigurer(system);
		Binder binder = Binder.get(environment);
		Map<String, LogLevel> levels = binder.bind(LOGGING_LEVEL, STRING_LOGLEVEL_MAP).orElseGet(Collections::emptyMap);
		levels.forEach((name, level) -> configureLogLevel(name, level, customizer));
	}

	private void configureLogLevel(String name, LogLevel level, BiConsumer<String, LogLevel> configurer) {
		if (this.loggerGroups != null) {
			LoggerGroup group = this.loggerGroups.get(name);
			if (group != null && group.hasMembers()) {
				group.configureLogLevel(level, configurer);
				return;
			}
		}
		configurer.accept(name, level);
	}

	private BiConsumer<String, LogLevel> getLogLevelConfigurer(LoggingSystem system) {
		return (name, level) -> {
			try {
				name = name.equalsIgnoreCase(LoggingSystem.ROOT_LOGGER_NAME) ? null : name;
				system.setLogLevel(name, level);
			}
			catch (RuntimeException ex) {
				this.logger.error(LogMessage.format("Cannot set level '%s' for '%s'", level, name));
			}
		};
	}

	private void registerShutdownHookIfNecessary(Environment environment, LoggingSystem loggingSystem) {
		boolean registerShutdownHook = environment.getProperty(REGISTER_SHUTDOWN_HOOK_PROPERTY, Boolean.class, false);
		if (registerShutdownHook) {
			Runnable shutdownHandler = loggingSystem.getShutdownHandler();
			if (shutdownHandler != null && shutdownHookRegistered.compareAndSet(false, true)) {
				registerShutdownHook(new Thread(shutdownHandler));
			}
		}
	}

	void registerShutdownHook(Thread shutdownHook) {
		Runtime.getRuntime().addShutdownHook(shutdownHook);
	}

	public void setOrder(int order) {
		this.order = order;
	}

	@Override
	public int getOrder() {
		return this.order;
	}

	/**
	 * Sets a custom logging level to be used for Spring Boot and related libraries.
	 * @param springBootLogging the logging level
	 */
	public void setSpringBootLogging(LogLevel springBootLogging) {
		this.springBootLogging = springBootLogging;
	}

	/**
	 * Sets if initialization arguments should be parsed for {@literal debug} and
	 * {@literal trace} properties (usually defined from {@literal --debug} or
	 * {@literal --trace} command line args). Defaults to {@code true}.
	 * @param parseArgs if arguments should be parsed
	 */
	public void setParseArgs(boolean parseArgs) {
		this.parseArgs = parseArgs;
	}

}

接下来说说LoggingApplicationListener类里面的重要属性

• LoggingSystem： 作用是根据引入的日志框架加载对应的日志文件

/**
	 * Detect and return the logging system in use. Supports Logback and Java Logging.
	 * @param classLoader the classloader
	 * @return the logging system
	 */
	public static LoggingSystem get(ClassLoader classLoader) {
        //SYSTEMS中存放了SpringBoot中Logback，Log4j2，Java Util Logging几个日志框架适配器的路径，检查这些类适配器是否存在以判断这些日志框架是否引入
		String loggingSystemClassName = System.getProperty(SYSTEM_PROPERTY);
		if (StringUtils.hasLength(loggingSystemClassName)) {
			if (NONE.equals(loggingSystemClassName)) {
				return new NoOpLoggingSystem();
			}
			return get(classLoader, loggingSystemClassName);
		}
		LoggingSystem loggingSystem = SYSTEM_FACTORY.getLoggingSystem(classLoader);
		Assert.state(loggingSystem != null, "No suitable logging system located");
		return loggingSystem;
	}

• initialize:

/**
	 * Initialize the logging system according to preferences expressed through the
	 */
	protected void initialize(ConfigurableEnvironment environment, ClassLoader classLoader) {
		getLoggingSystemProperties(environment).apply();
		this.logFile = LogFile.get(environment);//从environment读取logging.file，logging.path配置，构建LogFile
		if (this.logFile != null) {
			this.logFile.applyToSystemProperties();
		}
		this.loggerGroups = new LoggerGroups(DEFAULT_GROUP_LOGGERS);
		initializeEarlyLoggingLevel(environment);// 读取Environment中debug，trace的配置到springBootLogging属性
		initializeSystem(environment, this.loggingSystem, this.logFile);
		initializeFinalLoggingLevels(environment, this.loggingSystem);
		registerShutdownHookIfNecessary(environment, this.loggingSystem);
	}

/*
Environment代表当前应用运行环境，管理配置属性数据，并提供Profile特性，即可以根据环境得到相应配置属性数据。
*/