闲来无事想做个HTTP请求日志分析以及存档功能 因为和客户端交互数据用的是Protobuf 所以这里记录下拦截日志的实现(Json什么的其他传输协议也做了几个) 日志拦截器继 DispatcherServlet类然后找个地方注册一下Bean即可 注册代码如下 LoggableDispatcherServlet是我们自己的类
记录的日志是打印出json 因为结构比较复杂
@Bean
public ServletRegistrationBean dispatcherRegistration() {
return new ServletRegistrationBean<>(dispatcherServlet());
}
@Bean(name = DispatcherServletAutoConfiguration.DEFAULT_DISPATCHER_SERVLET_BEAN_NAME)
public DispatcherServlet dispatcherServlet() {
return new LoggableDispatcherServlet();
}
然后是拦截类 我这里用的是protobuf 不是平时的json 所以比较麻烦 为此还折腾了一会 json或者xml简单很多 如果你不是protobuf 吧那一部分删除即可 本文主要围绕protobuf做处理 其他的处理方式也都在里面弄了下 可以参考
import com.dexfun.magic.common.util.HttpMessage;
import com.dexfun.magic.protobuf.Transmission;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.google.protobuf.Message;
import com.google.protobuf.util.JsonFormat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.core.MethodParameter;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.http.HttpStatus;
import org.springframework.util.StringUtils;
import org.springframework.web.method.HandlerMethod;
import org.springframework.web.servlet.DispatcherServlet;
import org.springframework.web.servlet.HandlerExecutionChain;
import org.springframework.web.util.ContentCachingResponseWrapper;
import javax.servlet.ReadListener;
import javax.servlet.ServletInputStream;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletRequestWrapper;
import javax.servlet.http.HttpServletResponse;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.PrintWriter;
import java.lang.reflect.Method;
import java.lang.reflect.Parameter;
import java.util.Collection;
import java.util.Enumeration;
import java.util.HashMap;
import java.util.Map;
@Order(Ordered.HIGHEST_PRECEDENCE)//最高优先级 方便拦截404什么的
public class LoggableDispatcherServlet extends DispatcherServlet {
private static final Logger logger = LoggerFactory.getLogger("HttpLogger");
private static final ObjectMapper mapper = new ObjectMapper();
@Override
protected void doDispatch(HttpServletRequest request, HttpServletResponse response) throws Exception {
ContentCachingResponseWrapper responseWrapper = new ContentCachingResponseWrapper(response);
BufferedServletRequestWrapper bufferedServletRequestWrapper = new BufferedServletRequestWrapper(request);
ServletInputStream inputStream = bufferedServletRequestWrapper.getInputStream();
setThrowExceptionIfNoHandlerFound(true);
ObjectNode rootNode = mapper.createObjectNode();
ObjectNode reqNode = mapper.createObjectNode();
ObjectNode resNode = mapper.createObjectNode();
String method = request.getMethod();
rootNode.put("method", method);
rootNode.put("url", request.getRequestURL().toString());
rootNode.put("remoteAddr", request.getRemoteAddr());
rootNode.put("x-forwarded-for", request.getHeader("x-forwarded-for"));
rootNode.set("request", reqNode);
rootNode.set("response", resNode);
reqNode.set("headers", mapper.valueToTree(getRequestHeaders(request)));
if (method.equals("GET")) {
reqNode.set("body", mapper.valueToTree(request.getParameterMap()));
} else {
if (isProtoBufPost(request)) {
HandlerExecutionChain handlerExecutionChain = getHandler(request);
if (handlerExecutionChain != null) {
Object handler = handlerExecutionChain.getHandler();
if (handler instanceof HandlerMethod) {
HandlerMethod handlerMethod = (HandlerMethod) handler;
for (MethodParameter methodParameter : handlerMethod.getMethodParameters()) {
Parameter parameter = methodParameter.getParameter();
if (Message.class.isAssignableFrom(parameter.getType())) {
Class<Message> type = (Class<Message>) parameter.getType();
byte[] contentAsByteArray = inputStream.readAllBytes();
Transmission.Request parseFrom = Transmission.Request.parseFrom(HttpMessage.transform(contentAsByteArray));
String print = JsonFormat.printer().usingTypeRegistry(JsonFormat.TypeRegistry.newBuilder().add(parseFrom.getData().unpack(type).getDescriptorForType()).build()).print(parseFrom);
reqNode.set("body", mapper.readTree(print));
reqNode.put("bodyIsJson", true);
}
}
}
} else {
//走到这里基本就是404的情况了 无法预知 Any的data类型 无法格式化成json 只能转成文本消息存日志了 2333
byte[] contentAsByteArray = inputStream.readAllBytes();
Transmission.Request parseFrom = Transmission.Request.parseFrom(HttpMessage.transform(contentAsByteArray));
reqNode.put("body", parseFrom.toString());
reqNode.put("bodyIsJson", false);
}
} else if (isFormPost(request)) {
reqNode.set("body", mapper.valueToTree(request.getParameterMap()));
reqNode.put("bodyIsJson", true);
} else if (isJsonPost(request)) {
byte[] contentAsByteArray = inputStream.readAllBytes();
reqNode.set("body", mapper.readTree(contentAsByteArray));
reqNode.put("bodyIsJson", true);
} else if (isTextPost(request) || isXmlPost(request)) {
byte[] contentAsByteArray = inputStream.readAllBytes();
reqNode.put("body", new String(contentAsByteArray));
reqNode.put("bodyIsJson", false);
} else if (isMediaPost(request)) {
reqNode.put("body", "Media Request Body ContentLength = " + request.getContentLengthLong());
reqNode.put("bodyIsJson", false);
} else {
byte[] contentAsByteArray = inputStream.readAllBytes();
reqNode.put("body", "Unknown Request Body ContentLength = " + request.getContentLengthLong() + " body = " + (request.getContentLengthLong() > 2048 ? "content is too long" : new String(contentAsByteArray)));
reqNode.put("bodyIsJson", false);
}
}
HandlerExecutionChain handlerExecutionChain = getHandler(request);
if (handlerExecutionChain == null) {
//手动判断是不是404 不走系统流程 直接处理 因为会重定向/error
resNode.put("status", HttpStatus.NOT_FOUND.value());
logger.info(rootNode.toString());
response.setStatus(HttpStatus.NOT_FOUND.value());
PrintWriter writer = response.getWriter();
writer.write("Request path not found");
writer.flush();
writer.close();
return;
}
System.out.println(handlerExecutionChain);
try {
super.doDispatch(bufferedServletRequestWrapper, responseWrapper);
} finally {
byte[] responseWrapperContentAsByteArray = responseWrapper.getContentAsByteArray();
responseWrapper.copyBodyToResponse();//这里有顺序 必须先读body 然后再调用这个方法 才能继续读
resNode.put("status", response.getStatus());
Map<String, Object> responseHeaders = getResponseHeaders(response);
//这里判断错误拦截是不是吧url改成error了 如果是就做一下替换 替换的值是错误拦截器写到header里面的
String url = rootNode.get("url").asText();
if (url.endsWith("/error")) {
String path = (String) responseHeaders.get("x-error-path");
if (!StringUtils.isEmpty(path)) {
rootNode.put("url", url.replace("/error", path));
}
}
resNode.set("headers", mapper.valueToTree(responseHeaders));
if (isProtoBufPost(responseWrapper)) {
Object handler = handlerExecutionChain.getHandler();
if (handler instanceof HandlerMethod) {
HandlerMethod handlerMethod = (HandlerMethod) handler;
MethodParameter returnType = handlerMethod.getReturnType();
Method returnTypeMethod = returnType.getMethod();
if (returnTypeMethod != null) {
if (Message.class.isAssignableFrom(returnTypeMethod.getReturnType()) && response.getStatus() == HttpStatus.OK.value()) {
Class<Message> type = (Class<Message>) returnTypeMethod.getReturnType();
Transmission.Response parseFrom = Transmission.Response.parseFrom(HttpMessage.transform(responseWrapperContentAsByteArray));
if (parseFrom.hasData()) {
String print = JsonFormat.printer().usingTypeRegistry(JsonFormat.TypeRegistry.newBuilder().add(parseFrom.getData().unpack(type).getDescriptorForType()).build()).print(parseFrom);
resNode.set("body", mapper.readTree(print));
} else {
String print = JsonFormat.printer().print(parseFrom);
resNode.set("body", mapper.readTree(print));
}
resNode.put("bodyIsJson", true);
}
}
}
} else {
try {
resNode.set("body", mapper.readTree(responseWrapperContentAsByteArray));
resNode.put("bodyIsJson", true);
} catch (Exception e) {
resNode.put("body", new String(responseWrapperContentAsByteArray));
resNode.put("bodyIsJson", false);
}
}
logger.info(rootNode.toString());
}
}
private Map<String, Object> getRequestHeaders(HttpServletRequest request) {
Map<String, Object> headers = new HashMap<>();
Enumeration<String> headerNames = request.getHeaderNames();
while (headerNames.hasMoreElements()) {
String headerName = headerNames.nextElement();
headers.put(headerName, request.getHeader(headerName));
}
return headers;
}
private Map<String, Object> getResponseHeaders(HttpServletResponse response) {
Map<String, Object> headers = new HashMap<>();
Collection<String> headerNames = response.getHeaderNames();
for (String headerName : headerNames) {
headers.put(headerName, response.getHeader(headerName));
}
return headers;
}
private boolean isFormPost(HttpServletRequest request) {
String contentType = request.getContentType();
return (contentType != null && contentType.contains("x-www-form"));
}
private boolean isMediaPost(HttpServletRequest request) {
String contentType = request.getContentType();
if (contentType != null)
return contentType.contains("stream") || contentType.contains("image") || contentType.contains("video") || contentType.contains("audio");
return false;
}
private boolean isTextPost(HttpServletRequest request) {
String contentType = request.getContentType();
if (contentType != null)
return contentType.contains("text/plain");
return false;
}
private boolean isJsonPost(HttpServletRequest request) {
String contentType = request.getContentType();
if (contentType != null)
return contentType.contains("application/json");
return false;
}
private boolean isXmlPost(HttpServletRequest request) {
String contentType = request.getContentType();
if (contentType != null)
return contentType.contains("application/xml");
return false;
}
private boolean isProtoBufPost(HttpServletRequest request) {
String contentType = request.getContentType();
if (contentType != null)
return contentType.contains("application") && contentType.contains("protobuf");
return false;
}
private boolean isProtoBufPost(HttpServletResponse response) {
String contentType = response.getContentType();
if (contentType != null)
return contentType.contains("application") && contentType.contains("protobuf");
return false;
}
class BufferedServletInputStream extends ServletInputStream {
private ByteArrayInputStream inputStream;
private ServletInputStream is;
public BufferedServletInputStream(byte[] buffer, ServletInputStream is) {
this.is = is;
this.inputStream = new ByteArrayInputStream(buffer);
}
@Override
public int available() {
return inputStream.available();
}
@Override
public int read() {
return inputStream.read();
}
@Override
public int read(byte[] b, int off, int len) {
return inputStream.read(b, off, len);
}
@Override
public boolean isFinished() {
return is.isFinished();
}
@Override
public boolean isReady() {
return is.isReady();
}
@Override
public void setReadListener(ReadListener listener) {
is.setReadListener(listener);
}
}
class BufferedServletRequestWrapper extends HttpServletRequestWrapper {
private byte[] buffer;
private ServletInputStream is;
public BufferedServletRequestWrapper(HttpServletRequest request) throws IOException {
super(request);
this.is = request.getInputStream();
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byteArrayOutputStream.writeBytes(is.readAllBytes());
this.buffer = byteArrayOutputStream.toByteArray();
}
@Override
public ServletInputStream getInputStream() {
return new BufferedServletInputStream(this.buffer, this.is);
}
}
}
BufferedServletRequestWrapper 和 ContentCachingResponseWrapper类都是为了能重复读写 Stream
代码中的HttpMessage.transform是我自定义的数据加密工具类 这里就不多说了
HandlerExecutionChain handlerExecutionChain = getHandler(request);
protobuf的核心处理办法就是通过DispatcherServlet类的getHandler方法获取url对应的Controller方法中的参数类型 代码如上 然后利用JsonFormat输出入参的参数类类型 因为我这里的请求与入参都用自定义protobuf的 Transmission实体类包装了一下 并且正式data是Any类型 因为我要在外面封装一下统一参数 例如时间戳或者签名字符串来做校验 防止抓包或者重放请求 其实安全问题最好是做https证书双向校验 具体可以参考我的安全系列博文
这里贴一下我的 protobuf 消息转换类 这里面有上面提到的加密逻辑
import com.dexfun.magic.common.exception.CommonException;
import com.dexfun.magic.common.util.HttpMessage;
import com.dexfun.magic.protobuf.Transmission;
import com.google.protobuf.Message;
import org.springframework.http.HttpInputMessage;
import org.springframework.http.HttpOutputMessage;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.converter.AbstractHttpMessageConverter;
import org.springframework.http.converter.HttpMessageNotReadableException;
import org.springframework.http.converter.HttpMessageNotWritableException;
import org.springframework.lang.NonNull;
import java.io.DataInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
public class ProtoBufHttpMessageConverter extends AbstractHttpMessageConverter<Message> {
private static final MediaType PROTOBUF = new MediaType("application", "x-protobuf", StandardCharsets.UTF_8);
private static final String X_PROTOBUF_SCHEMA_HEADER = "X-Protobuf-Schema";
private static final String X_PROTOBUF_MESSAGE_HEADER = "X-Protobuf-Message";
public ProtoBufHttpMessageConverter() {
super(PROTOBUF);
}
@Override
protected boolean supports(@NonNull Class<?> clazz) {
return Message.class.isAssignableFrom(clazz);
}
@Override
protected MediaType getDefaultContentType(Message message) {
return PROTOBUF;
}
@NonNull
@Override
protected Message readInternal(@NonNull Class<? extends Message> clazz, @NonNull HttpInputMessage inputMessage)
throws IOException, HttpMessageNotReadableException {
InputStream inputStream = inputMessage.getBody();
DataInputStream dataInputStream = new DataInputStream(inputStream);
byte[] bytes = new byte[inputStream.available()];
dataInputStream.readFully(bytes);
dataInputStream.close();
inputStream.close();
Transmission.Request request = Transmission.Request.parseFrom(HttpMessage.transform(bytes));
if ((System.currentTimeMillis() - request.getTimestamp()) > 1000 * 30) {
throw new CommonException(HttpStatus.FORBIDDEN.value(), "请检查时间是否准确");
}
return request.getData().unpack(clazz);
}
@Override
protected void writeInternal(@NonNull Message message, @NonNull HttpOutputMessage outputMessage) throws IOException, HttpMessageNotWritableException {
setProtoHeader(outputMessage, message);
OutputStream body = outputMessage.getBody();
if (message instanceof Transmission.Response) {
body.write(HttpMessage.transform(message.toByteArray()));
} else {
//如果Controller返回的是protobuf没有手动包装这里直接返回包装类并且默认成功 因为我做了异常统一处理 失败情况都是通过抛出异常处理的 最后会贴上统一异常处理类的代码Demo
body.write(HttpMessage.transform(HttpMessage.ok(message).toByteArray()));
}
body.flush();
body.close();
}
private void setProtoHeader(HttpOutputMessage response, Message message) {
response.getHeaders().set(X_PROTOBUF_SCHEMA_HEADER, message.getDescriptorForType().getFile().getName());
response.getHeaders().set(X_PROTOBUF_MESSAGE_HEADER, message.getDescriptorForType().getFullName());
}
}
需要注意的是writeInternal方法 这里针对Controller稍微做了一下处理 和拦截那边的Response输出做了一下对应 可以让Controller返回消息响应包装类或者直接返回 例如如下代码 会方便一点
@RestController
public class UserController {
@Autowired
UserServiceImpl userService;
//直接返回实体
@RequestMapping(value = "/login.bin", method = RequestMethod.POST)
public StringValue login(@RequestBody LoginDto.LoginEntity entity, HttpServletRequest request) throws Exception {
String remoteAddress = Optional.ofNullable(request.getHeader("x-forwarded-for")).orElse(request.getRemoteAddr());
String login = userService.login(entity, remoteAddress);
return StringValue.newBuilder().setValue(login).build();
}
//返回包装的类
@RequestMapping(value = "/t", method = RequestMethod.POST)
public Transmission.Response login(@RequestParam String aad, HttpServletRequest request) throws Exception {
return Transmission.Response.newBuilder().setStatus(200).setMessage("ok").build();
}
}
最后贴上异常统一处理类代码
import com.dexfun.magic.common.exception.CommonException;
import com.dexfun.magic.protobuf.Transmission;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.web.servlet.error.ErrorAttributes;
import org.springframework.boot.web.servlet.error.ErrorController;
import org.springframework.http.HttpStatus;
import org.springframework.http.converter.HttpMessageNotReadableException;
import org.springframework.util.StringUtils;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.context.request.ServletWebRequest;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.Map;
/**
* @author Smile
*/
@Slf4j
@RestController
@RestControllerAdvice
public class RestExceptionHandler implements ErrorController {
private final ErrorAttributes errorAttributes;
@Autowired
public RestExceptionHandler(ErrorAttributes errorAttributes) {
this.errorAttributes = errorAttributes;
}
//返回自定义包装的类
@ExceptionHandler(value = CommonException.class)
public Transmission.Response commonException(CommonException e) {
return Transmission.Response.newBuilder().setStatus(e.getStatus()).setMessage(e.getMessage()).build();
}
@ExceptionHandler(value = Throwable.class)
public String allException(Throwable e, HttpServletResponse response) {
log.error("Server Exception", e);
if (e instanceof HttpMessageNotReadableException) {
response.setStatus(HttpStatus.NOT_ACCEPTABLE.value());
try {
String message = e.getMessage();
if (StringUtils.isEmpty(message)) {
return "Error Not Message";
}
return e.getMessage().split(":")[0].split(";")[0];
} catch (Exception ex) {
return e.getMessage();
}
}
response.setStatus(HttpStatus.INTERNAL_SERVER_ERROR.value());
return e.getMessage();
}
//平时基本不会走这里 除了404 如果我在日志拦截器那边手动判断处理了404的情况就不会走
@RequestMapping(value = "/error")
public String error(HttpServletRequest request, HttpServletResponse response) {
ServletWebRequest servletWebRequest = new ServletWebRequest(request);
Map<String, Object> attributes = errorAttributes.getErrorAttributes(servletWebRequest, false);
Integer status = (Integer) attributes.get("status");
String path = (String) attributes.get("path");
response.setHeader("x-error-path", path);//写入错误前的原始url 日志拦截器要用
String error = (String) attributes.get("error");
String message = (String) attributes.get("message");
message = String.format("Request path %s %s", error, message);
log.error("Request Exception " + attributes.toString());
response.setStatus(status);
return message;
}
@Override
public String getErrorPath() {
return "/error";
}
}
以上错误拦截 如果是自定义业务异常就转换成protobuf消息并且http状态码永远是200 其他的异常返回对应http状态码 然后body打印错误消息 客户端直接dialog错误信息body 因为这些错误都是不可控的 系统发生严重问题了 及时解决就行