一、TCP粘包/拆包

1、什么是TCP粘包/拆包

tcp将用户从客户端发往服务端的请求数据包。进行拆分或重新组合进行发送。

例子：

  • 将数据包A,拆分成A1,A2两个数据包进行发送。（A1+A2=A,其中A1,A2就是拆包）
  • 将数据包A和B进行拆分，然后重组发送。最终发送结果为：A1 , A2+B  其中（A1+A2=A ,B=B，其中A2+B 就是粘包）

2、TCP粘包/拆包发生的原因

  • 应用程序write写入的字节大小大于套接口发送缓冲区的大小。
  • 进行MSS大小的TCP分段
  • 以太网帧的payload大于MTU进行IP分片

TCP java 拆包 粘包 netty解决tcp粘包和拆包_TCP java 拆包 粘包

 

 

3、TCP粘包/拆包发生的原因

 由于TCP无法理解上层的业务数据，所以在底层是无法保证数据包不被拆分和重组的。这个问题只能通过上层的应用协议栈设计来解决。根据业界的主流协议的解决方案，可以归纳如下。

  • 消息定长，例如每个报文的大小固定长度200字节，如果不够，空位补空格。
  • 在包尾增加回车换行符进行分割，例如FTP协议
  • 将消息分为消息头和消息体，消息头中包含消息的总长度（或者消息体长度）的字段，通常设计思路为消息头的第一个字段使用int32来表示消息的总长度
  • 更复杂的应用层协议

 

二、时间服务器TCP粘包/拆包的案例

1、按之前写的timerServer代码进行改动。客户端发送100条请求, 服务端接收请求，并响应。 

2、服务端代码

TCP java 拆包 粘包 netty解决tcp粘包和拆包_.net_02

TCP java 拆包 粘包 netty解决tcp粘包和拆包_ide_03

package com.spring.test.service.netty.nettydemo.server;

import io.netty.bootstrap.ServerBootstrap;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelOption;
import io.netty.channel.EventLoopGroup;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.nio.NioServerSocketChannel;

/**
 * @date 4:13 PM 2019/8/11
 */
public class TimerServer {


    public static void main(String[] args) {
        TimerServer timerServer=new TimerServer();
        timerServer.bind(8080);
    }

    public void bind(int port){
        //step1:配置服务端的NIO线程组(一个线程组用于服务端接收客户端连接,一个线程组用于进行socketChannel的网络读写)
        EventLoopGroup bossGroup=new NioEventLoopGroup();
        EventLoopGroup workerGroup=new NioEventLoopGroup();
        try {
            //netty用于启动NIO服务的辅助启动类，目的四降低服务端开发复杂度
            ServerBootstrap serverBootstrap=new ServerBootstrap();
            serverBootstrap.group(bossGroup,workerGroup)
                    .channel(NioServerSocketChannel.class)
                    .option(ChannelOption.SO_BACKLOG,1024)
                    .childHandler(new ChildChannelHandler());
            //step2:绑定端口，同步等待成功
            ChannelFuture f=serverBootstrap.bind(port).sync();

            //step3:等待服务端监听端口关闭
            f.channel().closeFuture().sync();

        }catch (Exception e){
            e.printStackTrace();
        }finally {
            //step4:优雅退出，释放线程池资源
            bossGroup.shutdownGracefully();
            workerGroup.shutdownGracefully();
        }



    }
}




package com.spring.test.service.netty.nettydemo.server;


import io.netty.channel.ChannelInitializer;
import io.netty.channel.socket.SocketChannel;

/**
 * @author
 * @date 4:21 PM 2019/8/11
 */
public class ChildChannelHandler extends ChannelInitializer<SocketChannel>{
    @Override
    protected void initChannel(SocketChannel socketChannel) throws Exception {
        socketChannel.pipeline().addLast(new TimerServerHandler());
    }
}




package com.spring.test.service.netty.nettydemo.server;


import org.apache.commons.lang.time.DateFormatUtils;

import io.netty.buffer.ByteBuf;
import io.netty.buffer.Unpooled;
import io.netty.channel.ChannelHandlerAdapter;
import io.netty.channel.ChannelHandlerContext;

/**
 * @author
 * @date 4:26 PM 2019/8/11
 */
public class TimerServerHandler extends ChannelHandlerAdapter{

    private int counter;

    /**
     * (接收到第一个数据包的内容如下:)
     * The time server receive order:QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUE   ;the counter is:1
     * The time server resp order:BAD ORDER
     *
     *
     * (接收到的第二个数据包内容如下:)
     * The time server receive order:Y TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER
     * QUERY TIME ORDER   ;the counter is:2
     *
     * The time server resp order:BAD ORDER
     * @param ctx
     * @param msg
     * @throws Exception
     */
    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        //读取请求
        ByteBuf buffer= (ByteBuf) msg;
        byte[] req=new byte[buffer.readableBytes()];
        buffer.readBytes(req);
        String reqbody=new String(req,"utf-8").substring(0,req.length-System.getProperty("line.separator").length());
        System.out.println("The time server receive order:"+reqbody+"   ;the counter is:"+ (++counter));
        String currentTime="QUERY TIME ORDER".equals(reqbody)? DateFormatUtils.format(System.currentTimeMillis(), "yyyy-MM-dd HH:mm:ss"):"BAD ORDER";

        //请求响应
        ByteBuf resp=Unpooled.copiedBuffer(currentTime.getBytes());
        System.out.println("The time server resp order:"+currentTime);
        //将相应结果，异步发送给客户端
        ctx.write(resp);
    }

    @Override
    public void channelReadComplete(ChannelHandlerContext ctx) throws Exception {
        //将消息发送队列中的消息写入socketChannel中发送给对方。
        ctx.flush();
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
        ctx.close();
    }
}

View Code

3、客户端代码

TCP java 拆包 粘包 netty解决tcp粘包和拆包_.net_02

TCP java 拆包 粘包 netty解决tcp粘包和拆包_ide_03

package com.spring.test.service.netty.nettydemo.client;

import io.netty.bootstrap.Bootstrap;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelInitializer;
import io.netty.channel.ChannelOption;
import io.netty.channel.EventLoopGroup;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.SocketChannel;
import io.netty.channel.socket.nio.NioSocketChannel;

/**
 * @author
 * @date 5:41 PM 2019/8/11
 */
public class TimerClient {

    public static void main(String[] args) throws InterruptedException {
        TimerClient timerClient=new TimerClient();
        timerClient.connect(8080,"127.0.0.1");
    }

    public void connect(int port,String host) throws InterruptedException {
        //step1:配置NIO客户端线程组
        EventLoopGroup group=new NioEventLoopGroup();
        try {
            Bootstrap bootstrap=new Bootstrap();
            bootstrap.group(group).channel(NioSocketChannel.class)
                    .option(ChannelOption.TCP_NODELAY,true)
                    .handler(new ChannelInitializer<SocketChannel>() {
                        @Override
                        protected void initChannel(SocketChannel ch) throws Exception {
                            ch.pipeline().addLast(new TimerClientHandler());
                        }
                    });

            //发起异步连接操作(调用同步方法，等待连接成功)
            ChannelFuture f=bootstrap.connect(host,port).sync();

            //等待客户端链路关闭
            f.channel().closeFuture().sync();

        }catch (Exception e){
            e.printStackTrace();
        }finally {
            //优雅的退出，释放NIO线程组
            group.shutdownGracefully();
        }
    }
}



package com.spring.test.service.netty.nettydemo.client;

import io.netty.buffer.ByteBuf;
import io.netty.buffer.Unpooled;
import io.netty.channel.ChannelHandlerAdapter;
import io.netty.channel.ChannelHandlerContext;

/**
 * @author
 * @date 5:49 PM 2019/8/11
 */
public class TimerClientHandler extends ChannelHandlerAdapter {

    private final byte[] firstMessage;

    private int counter;

    public TimerClientHandler(){
        firstMessage=("QUERY TIME ORDER"+System.getProperty("line.separator")).getBytes();
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
        System.out.println("释放资源");
        ctx.close();
    }

    @Override
    public void channelActive(ChannelHandlerContext ctx) throws Exception {
        ByteBuf message=null;
        //发送请求
        for(int i=0;i<100;i++){
            message=Unpooled.buffer(firstMessage.length);
            message.writeBytes(firstMessage);
            ctx.writeAndFlush(message);
        }

    }

    /**
     *（接收到服务端的响应如下:）
     *
     * Now time is:BAD ORDERBAD ORDER  ;the counter is:1
     * @param ctx
     * @param msg
     * @throws Exception
     */
    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        //读取响应
        ByteBuf buf= (ByteBuf) msg;
        byte[] resp=new byte[buf.readableBytes()];
        buf.readBytes(resp);
        String responseBody=new String(resp,"utf-8");
        System.out.println("Now time is:"+responseBody +"  ;the counter is:"+(++counter));

    }
}

View Code

4、现象描述：

  • 客户单发送100条消息，服务端只收到2条消息。且发生了对一条消息拆分成2部分的情况。说明客户端在发送时，发生了粘包和拆包的问题。
  • 服务端接收到客户端2条消息，响应了两条消息，但客户单只收到了一条响应。说明在服务端向客户端发送响应的时候，发生了粘包的问题。

 

三、解决时间服务器TCP粘包/拆包的问题

1、在服务端，客户端新增编解码器，解决tcp粘包/拆包的问题

2、服务端代码

TCP java 拆包 粘包 netty解决tcp粘包和拆包_.net_02

TCP java 拆包 粘包 netty解决tcp粘包和拆包_ide_03

package com.spring.test.service.netty.nettydemo.server;

import io.netty.bootstrap.ServerBootstrap;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelOption;
import io.netty.channel.EventLoopGroup;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.nio.NioServerSocketChannel;

/**
 * @date 4:13 PM 2019/8/11
 */
public class TimerServer {


    public static void main(String[] args) {
        TimerServer timerServer=new TimerServer();
        timerServer.bind(8080);
    }

    public void bind(int port){
        //step1:配置服务端的NIO线程组(一个线程组用于服务端接收客户端连接,一个线程组用于进行socketChannel的网络读写)
        EventLoopGroup bossGroup=new NioEventLoopGroup();
        EventLoopGroup workerGroup=new NioEventLoopGroup();
        try {
            //netty用于启动NIO服务的辅助启动类，目的四降低服务端开发复杂度
            ServerBootstrap serverBootstrap=new ServerBootstrap();
            serverBootstrap.group(bossGroup,workerGroup)
                    .channel(NioServerSocketChannel.class)
                    .option(ChannelOption.SO_BACKLOG,1024)
                    .childHandler(new ChildChannelHandler());
            //step2:绑定端口，同步等待成功
            ChannelFuture f=serverBootstrap.bind(port).sync();

            //step3:等待服务端监听端口关闭
            f.channel().closeFuture().sync();

        }catch (Exception e){
            e.printStackTrace();
        }finally {
            //step4:优雅退出，释放线程池资源
            bossGroup.shutdownGracefully();
            workerGroup.shutdownGracefully();
        }



    }
}


package com.spring.test.service.netty.nettydemo.server;


import io.netty.channel.ChannelInitializer;
import io.netty.channel.socket.SocketChannel;
import io.netty.handler.codec.LineBasedFrameDecoder;
import io.netty.handler.codec.string.StringDecoder;

/**
 * @author
 * @date 4:21 PM 2019/8/11
 */
public class ChildChannelHandler extends ChannelInitializer<SocketChannel>{
    @Override
    protected void initChannel(SocketChannel socketChannel) throws Exception {
        //新增编解码器
        socketChannel.pipeline().addLast(new LineBasedFrameDecoder(1024));
        socketChannel.pipeline().addLast(new StringDecoder());
        socketChannel.pipeline().addLast(new TimerServerHandler());
    }
}



package com.spring.test.service.netty.nettydemo.server;


import org.apache.commons.lang.time.DateFormatUtils;

import io.netty.buffer.ByteBuf;
import io.netty.buffer.Unpooled;
import io.netty.channel.ChannelHandlerAdapter;
import io.netty.channel.ChannelHandlerContext;

/**
 * @author
 * @date 4:26 PM 2019/8/11
 */
public class TimerServerHandler extends ChannelHandlerAdapter{

    private int counter;

    /**
     * @param ctx
     * @param msg
     * @throws Exception
     */
    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        //读取请求
        String body= (String) msg;
        System.out.println("The time server receive order:"+body+"   ;the counter is:"+ (++counter));
        String currentTime="QUERY TIME ORDER".equals(body)? DateFormatUtils.format(System.currentTimeMillis(), "yyyy-MM-dd HH:mm:ss"):"BAD ORDER";

        currentTime=currentTime+System.getProperty("line.separator");
        //请求响应
        ByteBuf resp=Unpooled.copiedBuffer(currentTime.getBytes());
        System.out.println("The time server resp order:"+currentTime);
        //将相应结果，异步发送给客户端
        ctx.write(resp);
    }

    @Override
    public void channelReadComplete(ChannelHandlerContext ctx) throws Exception {
        //将消息发送队列中的消息写入socketChannel中发送给对方。
        ctx.flush();
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
        ctx.close();
    }
}

View Code

3、客户端代码

TCP java 拆包 粘包 netty解决tcp粘包和拆包_.net_02

TCP java 拆包 粘包 netty解决tcp粘包和拆包_ide_03

package com.spring.test.service.netty.nettydemo.client;

import io.netty.bootstrap.Bootstrap;
import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelInitializer;
import io.netty.channel.ChannelOption;
import io.netty.channel.EventLoopGroup;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.SocketChannel;
import io.netty.channel.socket.nio.NioSocketChannel;
import io.netty.handler.codec.LineBasedFrameDecoder;
import io.netty.handler.codec.string.StringDecoder;

/**
 * @author
 * @date 5:41 PM 2019/8/11
 */
public class TimerClient {

    public static void main(String[] args) throws InterruptedException {
        TimerClient timerClient=new TimerClient();
        timerClient.connect(8080,"127.0.0.1");
    }

    public void connect(int port,String host) throws InterruptedException {
        //step1:配置NIO客户端线程组
        EventLoopGroup group=new NioEventLoopGroup();
        try {
            Bootstrap bootstrap=new Bootstrap();
            bootstrap.group(group).channel(NioSocketChannel.class)
                    .option(ChannelOption.TCP_NODELAY,true)
                    .handler(new ChannelInitializer<SocketChannel>() {
                        @Override
                        protected void initChannel(SocketChannel ch) throws Exception {
                            //新增编解码器
                            ch.pipeline().addLast(new LineBasedFrameDecoder(1024));
                            ch.pipeline().addLast(new StringDecoder());
                            ch.pipeline().addLast(new TimerClientHandler());
                        }
                    });

            //发起异步连接操作(调用同步方法，等待连接成功)
            ChannelFuture f=bootstrap.connect(host,port).sync();

            //等待客户端链路关闭
            f.channel().closeFuture().sync();

        }catch (Exception e){
            e.printStackTrace();
        }finally {
            //优雅的退出，释放NIO线程组
            group.shutdownGracefully();
        }
    }
}



package com.spring.test.service.netty.nettydemo.client;

import io.netty.buffer.ByteBuf;
import io.netty.buffer.Unpooled;
import io.netty.channel.ChannelHandlerAdapter;
import io.netty.channel.ChannelHandlerContext;

/**
 * @author
 * @date 5:49 PM 2019/8/11
 */
public class TimerClientHandler extends ChannelHandlerAdapter {

    private final byte[] firstMessage;

    private int counter;

    public TimerClientHandler() {
        firstMessage = ("QUERY TIME ORDER" + System.getProperty("line.separator")).getBytes();
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
        System.out.println("释放资源");
        ctx.close();
    }

    @Override
    public void channelActive(ChannelHandlerContext ctx) throws Exception {
        ByteBuf message = null;
        //发送请求
        for (int i = 0; i < 100; i++) {
            message = Unpooled.buffer(firstMessage.length);
            message.writeBytes(firstMessage);
            ctx.writeAndFlush(message);
        }

    }

    /**
     *
     * @param ctx
     * @param msg
     * @throws Exception
     */
    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        //读取响应
        String resp = (String) msg;
        System.out.println("Now time is:" + resp + "  ;the counter is:" + (++counter));

    }
}

View Code

4、LineBasedFrameDecoder+StringDecoder 如何解决TCP粘包/拆包问题

  • LineBasedFrameDecoder：工作原理是它依次遍历ByteBuf中的可读字节，判断看是否有“\n”或者“\r\n”，如果有，就以此位置作为结束位置，从可读索引到结束位置的区间的字节就组成了一行。作为客户端的一条请求消息。它是以换行符为结束标志的解码器。支持携带结束符或者不携带结束符两种解码方式，同时支持配置单行的最大长度。如果连续读取到最大长度后仍然没有发现换行符，就会抛出异常，同时忽略掉之前读到的异常码流。
  • StringDecoder：功能非常姜丹，就是将接受到对象转换成字符串，然后调用后面的handler。
  • LineBasedFrameDecoder+StringDecoder：组合就是按行切换文本的解码器，它被设计用来支持（上述场景）TCP的粘包和拆包。

 

四、疑问点

如何发送的消息不是以换行符结束的，该怎么办？或者没有回车换行符，靠消息头中的长度字段来分包怎么办？是不是需要自己写半包解码器？

答案是否定的。Netty提供了多种支持TCP粘包/拆包的解码器，用来满足用户的不同诉求。

 

五、应用层如何解决TCP粘包/拆包导致的读半包问题

1、消息长度固定，累计读取到长度总和为定长LEN的报文后，就认为读取到一个完整的消息；将计数器置位，重新开始读取下一个数据报。

2、将回车换行符作为消息的结束符，例如FTP协议，这种方式在文本协议中应用比较广泛；

3、将特殊的分隔符作为消息的结束标志，回车换行符就是一种特殊的结束分隔符；

4、通过在消息头中定义长度字段来标识消息的总长度