1. 背景

通过上一篇文章:深入理解NIO多路复用,了解到内核态通过事件通知+中断检测socket事件,用户态可以使用1个线程处理所有socket请求,时间复杂度为O(1)。看上去该IO方案已经很完美了,但是当连接数越来越多时,且活跃的连接越来越多时,比如10w+,单线程处理可能会很吃力。而Netty可以设置线程池处理socket事件,从而分摊单线程压力;同时,Netty框架封装了通用逻辑,大大方便了业务开发。本文将会分析Netty线程池调用关系,即reactor模型,直接了解Netty最核心的设计思想;同时还会优化下NIO Server代码,实现一个最简单的Netty服务端Demo;最后介绍下Netty常见问题。

2. Netty基本概念

Netty通用代码:

创建两个线程池,分别用于处理ServerSocket事件和Socket事件;并指定ServerSocket和Socket发生事件时执行自定义类ServerHandler中的方法:

image.png

Netty业务代码:

ServerHander定义了方法,当服务端接受到了客户端发送的数据时,调用channelRead方法处理数据;当socket/serverSocket注册到selector中时,调用channelRegistered:

image.png

上述代码中,netty架构图如下所示:

Netty工作流程图 1.png

从Netty架构图中可以看到NioEventLoopGroup和pipeline是最重要的概念,后面将会从Netty工作流程详细分析这两个概念的实现思想。

3. Netty工作流程

3.1 创建bossGroup和workerGroup对象

如下,bossGroup对应NioEventLoopGroup创建1个NioEventLoop,workerGroup创建10个NioEventLoop。每个NioEventLoop内部包含一个新的多路复用器Selector和线程,bossGroup的selector用于注册serverSocketChannel,workerGroup的selector用于注册socketChannel。在线程中则是处理selector注册的socket上发生的事件。

NioEventLoopGroup bossGroup = new NioEventLoopGroup(1);
NioEventLoopGroup workerGroup = new NioEventLoopGroup(10);

NioEventLoopGroup从子类到父类的初始化顺序为:NioEventLoopGroup -> MultithreadEventLoopGroup -> MultithreadEventExecutorGroup。

注意:bossGroup只需要指定创建1个NioEventLoop,因为服务端只有一个ServerSocketChannel对象,根本没办法注册到多个selector中。没有任何资料能够实际展示bossGroup中的多个NioEventLoop。

3.1.1 创建SelectorProvider

SelectorProvider是Selector多路复用的工厂类,用于创建Selector的实现类。NioEventLoopGroup初始化时,创建了SelectorProvider对象:

public NioEventLoopGroup(int nThreads, Executor executor) {
    this(nThreads, executor, SelectorProvider.provider());
}

SelectorProvider类通过rt.jar包中的sun.nio.ch.DefaultSelectorProvider类调用create方法,创建SelectorProvider实现:

public abstract class SelectorProvider {
    public static SelectorProvider provider() {
        synchronized (lock) {
            if (provider != null)
                return provider;
            return AccessController.doPrivileged(
                new PrivilegedAction<SelectorProvider>() {
                    public SelectorProvider run() {
                            if (loadProviderFromProperty())
                                return provider;
                            if (loadProviderAsService())
                                return provider;
                            provider = sun.nio.ch.DefaultSelectorProvider.create();
                            return provider;
                        }
                    });
        }
    }
}

不同操作系统的jdk包中rt.jar包中DefaultSelectorProvider实现不同,例如mac os的create方法返回KQueueSelectorProvider对象:

public class DefaultSelectorProvider {
    private DefaultSelectorProvider() {
    }

    public static SelectorProvider create() {
        return new KQueueSelectorProvider();
    }
}

linux操作系统rt.jar包的create方法返回EPollSelectorProvider对象:

public class DefaultSelectorProvider {
    private DefaultSelectorProvider() {
    }

    public static SelectorProvider create() {
        String var0 = (String)AccessController.doPrivileged(new GetPropertyAction("os.name"));
        if (var0.equals("SunOS")) {
            return createProvider("sun.nio.ch.DevPollSelectorProvider");
        } else {
            return (SelectorProvider)(var0.equals("Linux") ? createProvider("sun.nio.ch.EPollSelectorProvider") : new PollSelectorProvider());
        }
    }
}

EPollSelectorProvider可以通过openSelector方法创建EPollSelectorImpl对象:

public class EPollSelectorProvider extends SelectorProviderImpl {
    public EPollSelectorProvider() {
    }

    public AbstractSelector openSelector() throws IOException {
        return new EPollSelectorImpl(this);
    }
}

EPollSelectorImpl最底层封装了socket系统调用epoll_create、epoll_ctl,完成多路复用功能。

3.1.2 创建线程池

有了SelectorProvider,就可以创建线程执行器Executor了。线程池中每一个线程的创建动作由DefaultThreadFactory定义。Executor直接从线程池中使用一个线程:

public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {
    protected MultithreadEventExecutorGroup(int nThreads, Executor executor, EventExecutorChooserFactory chooserFactory, Object... args) {
        //创建线程执行器,
        if (executor == null) {
            executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());
        }
        //省略
    }

    //创建线程池
    protected ThreadFactory newDefaultThreadFactory() {
        return new DefaultThreadFactory(getClass());
    }
}

线程池的初始化操作如下:

public class DefaultThreadFactory implements ThreadFactory {
    public DefaultThreadFactory(Class<?> poolType) {
        this(poolType, false, Thread.NORM_PRIORITY);
    }

    public DefaultThreadFactory(Class<?> poolType, boolean daemon, int priority) {
        this(toPoolName(poolType), daemon, priority);
    }

    public DefaultThreadFactory(String poolName, boolean daemon, int priority, ThreadGroup threadGroup) {
        ObjectUtil.checkNotNull(poolName, "poolName");

        if (priority < Thread.MIN_PRIORITY || priority > Thread.MAX_PRIORITY) {
            throw new IllegalArgumentException(
                    "priority: " + priority + " (expected: Thread.MIN_PRIORITY <= priority <= Thread.MAX_PRIORITY)");
        }
        //使用统一的前缀作为线程名
        prefix = poolName + '-' + poolId.incrementAndGet() + '-';
        this.daemon = daemon;
        this.priority = priority;
        this.threadGroup = threadGroup;
    }

    //可以调用newThread直接创建一个线程
    public Thread newThread(Runnable r) {
        Thread t = newThread(FastThreadLocalRunnable.wrap(r), prefix + nextId.incrementAndGet());
        try {
            if (t.isDaemon() != daemon) {
                t.setDaemon(daemon);
            }

            if (t.getPriority() != priority) {
                t.setPriority(priority);
            }
        } catch (Exception ignored) {
            // Doesn't matter even if failed to set.
        }
        return t;
    }
}

定义了线程名前缀:
image.png
后续创建线程时,使用线程名做前缀:
image.png

ThreadPerTaskExecutor调用execute时,直接从线程池中创建一个新线程:

public final class ThreadPerTaskExecutor implements Executor {
    private final ThreadFactory threadFactory;

    public ThreadPerTaskExecutor(ThreadFactory threadFactory) {
        this.threadFactory = ObjectUtil.checkNotNull(threadFactory, "threadFactory");
    }

    @Override
    public void execute(Runnable command) {
        threadFactory.newThread(command).start();
    }
}

3.1.3 封装线程池和Selector

通过创建SelectorProvider和Executor两个重要依赖后,就可以构造NioEventLoop了:

public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {
    protected MultithreadEventExecutorGroup(int nThreads, Executor executor, EventExecutorChooserFactory chooserFactory, Object... args) {
        //创建线程池
        if (executor == null) {
            executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());
        }

        children = new EventExecutor[nThreads];
        //创建NioEventLoop,bossGroup指定1个NioEventLoop,workerGroup指定10个NioEventLoop
        for (int i = 0; i < nThreads; i ++) {
            boolean success = false;
            try {
                //创建NioEventLoop
                children[i] = newChild(executor, args);
                success = true;
            } catch (Exception e) {
                // TODO: Think about if this is a good exception type
                throw new IllegalStateException("failed to create a child event loop", e);
            } finally {
                //省略
            }
        }
        chooser = chooserFactory.newChooser(children);

        //省略
    }

    //创建NioEventLoop的方法由NioEventLoopGroup类实现
    protected abstract EventLoop newChild(Executor executor, Object... args) throws Exception;
}

NioEventLoopGroup实现了newChild方法,创建NioEventLoop对象:

public class NioEventLoopGroup extends MultithreadEventLoopGroup {
    protected EventLoop newChild(Executor executor, Object... args) throws Exception {
        EventLoopTaskQueueFactory queueFactory = args.length == 4 ? (EventLoopTaskQueueFactory) args[3] : null;
        return new NioEventLoop(this, executor, (SelectorProvider) args[0],
            ((SelectStrategyFactory) args[1]).newSelectStrategy(), (RejectedExecutionHandler) args[2], queueFactory);
    }
}

NioEventLoop中,通过openSelector()方法创建selector,也就是EPollSelectorImpl对象。

public final class NioEventLoop extends SingleThreadEventLoop {
    NioEventLoop(NioEventLoopGroup parent, Executor executor, SelectorProvider selectorProvider,
                 SelectStrategy strategy, RejectedExecutionHandler rejectedExecutionHandler,
                 EventLoopTaskQueueFactory queueFactory) {
        super(parent, executor, false, newTaskQueue(queueFactory), newTaskQueue(queueFactory),
                rejectedExecutionHandler);
        this.provider = ObjectUtil.checkNotNull(selectorProvider, "selectorProvider");
        this.selectStrategy = ObjectUtil.checkNotNull(strategy, "selectStrategy");
        final SelectorTuple selectorTuple = openSelector();
        this.selector = selectorTuple.selector;
        this.unwrappedSelector = selectorTuple.unwrappedSelector;
    }

    private SelectorTuple openSelector() {
        final Selector unwrappedSelector;
        try {
            //创建EPollSelectorImpl对象
            unwrappedSelector = provider.openSelector();
        } catch (IOException e) {
            throw new ChannelException("failed to open a new selector", e);
        }
        //省略
        return new SelectorTuple(unwrappedSelector);
    }
}

3.2 NioEventLoopGroup总结

NioEventLoopGroup包含多个NioEventLoop。每个NioEventLoop内部包含一个新的多路复用器Selector和线程,bossGroup的selector用于注册serverSocketChannel,workerGroup的selector用于注册socketChannel。每个NioEventLoop中,都包含一个Selector以及一个线程,线程暂时用ThreadPerTaskExecutor表示,执行ThreadPerTaskExecutor#executor就会创建NioEventLoop专属的线程。

3.3 创建启动类ServerBootstrap对象

ServerBootstrap是启动类,将NioEventLoopGroup等参数传递到ServerBootstrap中,ServerBootstrap负责启动netty服务端。

3.3.1 指定SeverSocketChannel的实现类

指定NioServerSocketChannel作为netty的SeverSocketChannel实现类:

serverBootstrap.channel(NioServerSocketChannel.class);

NioServerSocketChannel的构造函数通过EPollSelectorProvider创建ServerSocketChannel对象

public class NioServerSocketChannel extends AbstractNioMessageChannel
                             implements io.netty.channel.socket.ServerSocketChannel {
    //DEFAULT_SELECTOR_PROVIDER就是EPollSelectorProvider对象
    private static final SelectorProvider DEFAULT_SELECTOR_PROVIDER = SelectorProvider.provider();

    public NioServerSocketChannel() {
        this(newSocket(DEFAULT_SELECTOR_PROVIDER));
    }

    private static ServerSocketChannel newSocket(SelectorProvider provider) {
        try {
            //通过EPollSelectorProvider的父类SelectorProviderImpl的openServerSocketChannel()方法创建ServerSocketChannel对象。
            return provider.openServerSocketChannel();
        } catch (IOException e) {
            throw new ChannelException(
                    "Failed to open a server socket.", e);
        }
    }

    public NioServerSocketChannel(ServerSocketChannel channel) {
        super(null, channel, SelectionKey.OP_ACCEPT);
        config = new NioServerSocketChannelConfig(this, javaChannel().socket());
    }
}

NioServerSocketChannel通过父类的AbstractNioChannel构造方法设置ServerSocketChannel为非阻塞:

public abstract class AbstractNioChannel extends AbstractChannel {
    protected AbstractNioChannel(Channel parent, SelectableChannel ch, int readInterestOp) {
        super(parent);
        this.ch = ch;
        this.readInterestOp = readInterestOp;
        try {
            ch.configureBlocking(false);
        } catch (IOException e) {
            try {
                ch.close();
            } catch (IOException e2) {
                logger.warn(
                            "Failed to close a partially initialized socket.", e2);
            }

            throw new ChannelException("Failed to enter non-blocking mode.", e);
        }
    }
}

NioServerSocketChannel的父类AbstractChannel会为ServerSocketChannel创建对应的Unsafe和Pipeline,这个后面再展开:

public abstract class AbstractChannel extends DefaultAttributeMap implements Channel {
    protected AbstractChannel(Channel parent) {
        this.parent = parent;
        id = newId();
        unsafe = newUnsafe();
        pipeline = newChannelPipeline();
    }

    protected abstract AbstractUnsafe newUnsafe();

    protected DefaultChannelPipeline newChannelPipeline() {
        return new DefaultChannelPipeline(this);
    }
}

3.3.2 配置handler

handler表示socket发生事件时,应该执行的操作。

serverBootstrap.option(ChannelOption.SO_BACKLOG, 128)
                    .handler(new ChannelInitializer<ServerSocketChannel>() {
                        @Override
                        protected void initChannel(ServerSocketChannel ch) throws Exception {
                            ch.pipeline().addLast(new ServerHandler());
                        }
                    })
                    .childHandler(new ChannelInitializer<SocketChannel>() {
                        @Override
                        protected void initChannel(SocketChannel ch) throws Exception {
                            ch.pipeline().addLast(new ServerHandler());
                        }
                    });

ServerBootstrap的父类AbstractBootstrap保存ServerSocketChannel对应的handler:

public abstract class AbstractBootstrap<B extends AbstractBootstrap<B, C>, C extends Channel> implements Cloneable {
    public B handler(ChannelHandler handler) {
        this.handler = ObjectUtil.checkNotNull(handler, "handler");
        return self();
    }
}

ServerBootstrap保存SocketChannel对应的childHander:

public class ServerBootstrap extends AbstractBootstrap<ServerBootstrap, ServerChannel> {
    public ServerBootstrap childHandler(ChannelHandler childHandler) {
        this.childHandler = ObjectUtil.checkNotNull(childHandler, "childHandler");
        return this;
    }
}

3.4 netty服务端启动

通过ServerBootstrap#bind方法启动netty服务端:

ChannelFuture future = serverBootstrap.bind(8080).sync();

3.4.1 创建ServerSocketChannel

调用ServerBootstrap的父类AbstractBootstrap的doBind方法,通过AbstractBootstrap#initAndRegister开始创建ServerSocketChannel:

public abstract class AbstractBootstrap<B extends AbstractBootstrap<B, C>, C extends Channel> implements Cloneable {
    private ChannelFuture doBind(final SocketAddress localAddress) {
        final ChannelFuture regFuture = initAndRegister();
        final Channel channel = regFuture.channel();
        if (regFuture.cause() != null) {
            return regFuture;
        }
        //省略
   }

   //创建ServerSocketChannel
   final ChannelFuture initAndRegister() {
        Channel channel = null;
        try {
            channel = channelFactory.newChannel();
            init(channel);
        } catch (Throwable t) {
            //省略
        }
        //省略
}

从上面的AbstractBootstrap#initAndRegister可以看到channelFactory#newChannel方法,它就调用了NioServerSocketChannel的构造函数,而NioServerSocketChannel构造函数里面就创建了ServerSocketChannel,并设置了非阻塞。

3.4.2 初始化pipeline

在创建完NioServerSocketChannel后,通过init方法,将主程序中定义的的Handler放到NioServerSocketChannel的pipeline中:

public class ServerBootstrap extends AbstractBootstrap<ServerBootstrap, ServerChannel> {
    void init(Channel channel) {
        setChannelOptions(channel, newOptionsArray(), logger);
        setAttributes(channel, attrs0().entrySet().toArray(EMPTY_ATTRIBUTE_ARRAY));

        ChannelPipeline p = channel.pipeline();

        final EventLoopGroup currentChildGroup = childGroup;
        final ChannelHandler currentChildHandler = childHandler;
        final Entry<ChannelOption<?>, Object>[] currentChildOptions;
        synchronized (childOptions) {
            currentChildOptions = childOptions.entrySet().toArray(EMPTY_OPTION_ARRAY);
        }
        final Entry<AttributeKey<?>, Object>[] currentChildAttrs = childAttrs.entrySet().toArray(EMPTY_ATTRIBUTE_ARRAY);

        p.addLast(new ChannelInitializer<Channel>() {
            @Override
            public void initChannel(final Channel ch) {
                final ChannelPipeline pipeline = ch.pipeline();
                ChannelHandler handler = config.handler();
                if (handler != null) {
                    pipeline.addLast(handler);
                }

                ch.eventLoop().execute(new Runnable() {
                    @Override
                    public void run() {
                        pipeline.addLast(new ServerBootstrapAcceptor(
                                ch, currentChildGroup, currentChildHandler, currentChildOptions, currentChildAttrs));
                    }
                });
            }
        });
    }
}

3.4.3 DefaultChannelPipeline插入元素

pipeline基于设计模式中的责任链模式。责任链模式为请求创建了一个处理对象的链。发起请求和具体处理请求的过程进行解耦:职责链上的处理者(Handler)负责处理请求,客户只需要将请求发送到职责链上即可,无须关心请求的处理细节和请求的传递。

当用户发起请求时,服务端逐步调用Inbound Handler,响应用户请求时,服务端逐步调用Outbound Handler。如下所示:

image.png

在创建ServerSocketChannel时,创建了NioEventLoop对应的DefaultChannelPipeline对象,该pipeline专属于ServerSocketChannel。

如下可以看到,DefaultChannelPipeline就是一个链表结构,每次addLast方法插入一个handler,就将handler封装成DefaultChannelHandlerContext,加入到链表结尾:

public class DefaultChannelPipeline implements ChannelPipeline {
    final AbstractChannelHandlerContext head;
    final AbstractChannelHandlerContext tail;

    public final ChannelPipeline addLast(EventExecutorGroup group, String name, ChannelHandler handler) {
        final AbstractChannelHandlerContext newCtx;
        synchronized (this) {
            checkMultiplicity(handler);

            newCtx = newContext(group, filterName(name, handler), handler);

            addLast0(newCtx);

            // If the registered is false it means that the channel was not registered on an eventLoop yet.
            // In this case we add the context to the pipeline and add a task that will call
            // ChannelHandler.handlerAdded(...) once the channel is registered.
            if (!registered) {
                newCtx.setAddPending();
                callHandlerCallbackLater(newCtx, true);
                return this;
            }
            //执行callHandlerAdded0方法
            EventExecutor executor = newCtx.executor();
            if (!executor.inEventLoop()) {
                callHandlerAddedInEventLoop(newCtx, executor);
                return this;
            }
        }
        callHandlerAdded0(newCtx);
        return this;
    }

    private void addLast0(AbstractChannelHandlerContext newCtx) {
        AbstractChannelHandlerContext prev = tail.prev;
        newCtx.prev = prev;
        newCtx.next = tail;
        prev.next = newCtx;
        tail.prev = newCtx;
    }
}

关于pipeline中handler调用链中的调用方式,后面再展开。

3.4.4 ServerSocketChannel的pipeline添加handler

在3.4.2节中,init方法会增加一个handler,通过addLast添加到ServerSocketChannel的pipeline中:

void init(Channel channel) {
    //省略
    p.addLast(new ChannelInitializer<Channel>() {
            @Override
            public void initChannel(final Channel ch) {
                final ChannelPipeline pipeline = ch.pipeline();
                ChannelHandler handler = config.handler();
                if (handler != null) {
                    pipeline.addLast(handler);
                }

                ch.eventLoop().execute(new Runnable() {
                    @Override
                    public void run() {
                        pipeline.addLast(new ServerBootstrapAcceptor(
                                ch, currentChildGroup, currentChildHandler, currentChildOptions, currentChildAttrs));
                    }
                });
            }
        });
}

上面这个handler是netty自定义的handler,它重写了ChannelInitializer的initChannel方法,当ServerSocketChannel发生了初始化事件时,会调用ChannelInitializer的initChannel方法,它负责将用户自定义的ServerHandler加入pipeline中。随后将netty自定义的ServerBootstrapAcceptor类放到pipeline中,而ServerBootstrapAcceptor负责将ServerSocketChannel创建的SocketChannel注册到Selector中

public class ServerBootstrap extends AbstractBootstrap<ServerBootstrap, ServerChannel> {
    private static class ServerBootstrapAcceptor extends ChannelInboundHandlerAdapter {
        public void channelRead(ChannelHandlerContext ctx, Object msg) {
            final Channel child = (Channel) msg;

            child.pipeline().addLast(childHandler);

            setChannelOptions(child, childOptions, logger);
            setAttributes(child, childAttrs);

            try {
                childGroup.register(child).addListener(new ChannelFutureListener() {
                    @Override
                    public void operationComplete(ChannelFuture future) throws Exception {
                        if (!future.isSuccess()) {
                            forceClose(child, future.cause());
                        }
                    }
                });
            } catch (Throwable t) {
                forceClose(child, t);
            }
        }
    }
}

注意:initChannel方法会在register0方法注册ServerSocketChannel到selector中时,在invokeHandlerAddedIfNeeded方法里面调用。后面会展开讨论。

3.4.5 准备将ServerSocketChannel注册到selector中

public abstract class AbstractBootstrap<B extends AbstractBootstrap<B, C>, C extends Channel> implements Cloneable {
    final ChannelFuture initAndRegister() {
        Channel channel = null;
        try {
            //创建socketServerChannel
            channel = channelFactory.newChannel();
            init(channel);
        } catch (Throwable t) {
            if (channel != null) {
                // channel can be null if newChannel crashed (eg SocketException("too many open files"))
                channel.unsafe().closeForcibly();
                // as the Channel is not registered yet we need to force the usage of the GlobalEventExecutor
                return new DefaultChannelPromise(channel, GlobalEventExecutor.INSTANCE).setFailure(t);
            }
            // as the Channel is not registered yet we need to force the usage of the GlobalEventExecutor
            return new DefaultChannelPromise(new FailedChannel(), GlobalEventExecutor.INSTANCE).setFailure(t);
        }
        //注册serverSocketChannel到selector中
        ChannelFuture regFuture = config().group().register(channel);
        if (regFuture.cause() != null) {
            if (channel.isRegistered()) {
                channel.close();
            } else {
                channel.unsafe().closeForcibly();
            }
        }
        return regFuture;
    }
}

上述config().group()返回bossGroup,调用NioEventLoopGroup的父类MultithreadEventLoopGroup的register方法进行注册:

public abstract class MultithreadEventLoopGroup extends MultithreadEventExecutorGroup implements EventLoopGroup {
    public ChannelFuture register(Channel channel) {
        //注册serversocketchannel
        return next().register(channel);
    }
}

next方法表示要选择一个在NioEventLoopGroup中选择一个NioEventLoop对象,NioEventLoop包含Selctor和线程。MultithreadEventExecutorGroup通过DefaultEventExecutorChooserFactory工厂创建chooser对象,该对象用于决定如何选择NioEventLoop执行对应任务。注意,由于bossGroup只有一个NioEventLoop,因此只会挑选唯一的NioEventLoop。而对于workerGroup,chooser才会在10个EventLoopGroup轮询选择。

public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {

    private final EventExecutorChooserFactory.EventExecutorChooser chooser;

    protected MultithreadEventExecutorGroup(int nThreads, Executor executor, Object... args) {
        this(nThreads, executor, DefaultEventExecutorChooserFactory.INSTANCE, args);
    }

    public EventExecutor next() {
        return chooser.next();
    }
}

如下所示,当NioEventLoopGroup的线程数为2的次方时,使用PowerOfTwoEventExecutorChooser实现;否则用GenericEventExecutorChooser实现。PowerOfTwoEventExecutorChooser通过位运算计算下一次轮询的NioEventLoop;GenericEventExecutorChooser通过算术运算计算下一次轮询的NioEventLoop。显然PowerOfTwoEventExecutorChooser效率更高:

public final class DefaultEventExecutorChooserFactory implements EventExecutorChooserFactory {

    public static final DefaultEventExecutorChooserFactory INSTANCE = new DefaultEventExecutorChooserFactory();

    private DefaultEventExecutorChooserFactory() { }

    @Override
    public EventExecutorChooser newChooser(EventExecutor[] executors) {
        if (isPowerOfTwo(executors.length)) {
            return new PowerOfTwoEventExecutorChooser(executors);
        } else {
            return new GenericEventExecutorChooser(executors);
        }
    }

    private static boolean isPowerOfTwo(int val) {
        return (val & -val) == val;
    }

    private static final class PowerOfTwoEventExecutorChooser implements EventExecutorChooser {
        private final AtomicInteger idx = new AtomicInteger();
        private final EventExecutor[] executors;

        PowerOfTwoEventExecutorChooser(EventExecutor[] executors) {
            this.executors = executors;
        }

        @Override
        public EventExecutor next() {
            //位运算
            return executors[idx.getAndIncrement() & executors.length - 1];
        }
    }

    private static final class GenericEventExecutorChooser implements EventExecutorChooser {
               private final AtomicLong idx = new AtomicLong();
        private final EventExecutor[] executors;

        GenericEventExecutorChooser(EventExecutor[] executors) {
            this.executors = executors;
        }

        @Override
        public EventExecutor next() {
            //算术运算
            return executors[(int) Math.abs(idx.getAndIncrement() % executors.length)];
        }
    }
}

挑选好要执行的NioEventLoop对象后,调用其父类SingleThreadEventLoop的register方法:

public abstract class SingleThreadEventLoop extends SingleThreadEventExecutor implements EventLoop {
    public ChannelFuture register(final ChannelPromise promise) {
        ObjectUtil.checkNotNull(promise, "promise");
        promise.channel().unsafe().register(this, promise);
        return promise;
    }
}

其中,promise.channel()就是NioServerSocketChannel,调用它的父类AbstractChannel的unsafe方法,返回unsafe成员:

public abstract class AbstractChannel extends DefaultAttributeMap implements Channel {

   private final Unsafe unsafe;

    public Unsafe unsafe() {
        return unsafe;
    }
}

Unsafe成员是在NioServerSocketChannel初始化时创建的,调用newUnsafe方法创建:

public abstract class AbstractChannel extends DefaultAttributeMap implements Channel {
    protected AbstractChannel(Channel parent) {
        this.parent = parent;
        id = newId();
        unsafe = newUnsafe();
        pipeline = newChannelPipeline();
    }
    //抽象方法,子类实现
    protected abstract AbstractUnsafe newUnsafe();
}

newUnsafe由子类AbstractNioMessageChannel实现的unsafe()方法:

public abstract class AbstractNioMessageChannel extends AbstractNioChannel {
    protected AbstractNioUnsafe newUnsafe() {
        return new NioMessageUnsafe();
    }

    //NioMessageUnsafe定义,重写了读方法,后面的ServerSocketChannel的读操作就执行这个方法
    private final class NioMessageUnsafe extends AbstractNioUnsafe {

        private final List<Object> readBuf = new ArrayList<Object>();

        @Override
        public void read() {
            assert eventLoop().inEventLoop();
            final ChannelConfig config = config();
            final ChannelPipeline pipeline = pipeline();
            final RecvByteBufAllocator.Handle allocHandle = unsafe().recvBufAllocHandle();
            allocHandle.reset(config);

            boolean closed = false;
            Throwable exception = null;
            try {
                try {
                    do {
                        int localRead = doReadMessages(readBuf);
                        if (localRead == 0) {
                            break;
                        }
                        if (localRead < 0) {
                            closed = true;
                            break;
                        }

                        allocHandle.incMessagesRead(localRead);
                    } while (allocHandle.continueReading());
                } catch (Throwable t) {
                    exception = t;
                }

                int size = readBuf.size();
                for (int i = 0; i < size; i ++) {
                    readPending = false;
                    pipeline.fireChannelRead(readBuf.get(i));
                }
                readBuf.clear();
                allocHandle.readComplete();
                pipeline.fireChannelReadComplete();

                if (exception != null) {
                    closed = closeOnReadError(exception);

                    pipeline.fireExceptionCaught(exception);
                }

                if (closed) {
                    inputShutdown = true;
                    if (isOpen()) {
                        close(voidPromise());
                    }
                }
            } finally {
                // Check if there is a readPending which was not processed yet.
                // This could be for two reasons:
                // * The user called Channel.read() or ChannelHandlerContext.read() in channelRead(...) method
                // * The user called Channel.read() or ChannelHandlerContext.read() in channelReadComplete(...) method
                //
                // See https://github.com/netty/netty/issues/2254
                if (!readPending && !config.isAutoRead()) {
                    removeReadOp();
                }
            }
        }
    }
}

由于NioMessageUnsafe继承了AbstractNioUnsafe方法,因此执行AbstractNioUnsafe父类AbstractChannel中的register方法。register方法确保EventLoop线程启动,如果没有启动,就当场启动。在线程中执行register0方法。register0负责注册及handler添加操作等,后面逐步展开:

public abstract class AbstractChannel extends DefaultAttributeMap implements Channel {
    protected abstract class AbstractUnsafe implements Unsafe {
        public final void register(EventLoop eventLoop, final ChannelPromise promise) {
            ObjectUtil.checkNotNull(eventLoop, "eventLoop");
            if (isRegistered()) {
                promise.setFailure(new IllegalStateException("registered to an event loop already"));
                return;
            }
            if (!isCompatible(eventLoop)) {
                promise.setFailure(
                        new IllegalStateException("incompatible event loop type: " + eventLoop.getClass().getName()));
                return;
            }

            AbstractChannel.this.eventLoop = eventLoop;
            //确定eventLoop线程已经启动,如果已经启动,在线程中执行register0方法
            if (eventLoop.inEventLoop()) {
                register0(promise);
            } else {
                try {
                    //如果eventLoop线程没有启动,先启动线程,再在线程中执行register0方法
                    eventLoop.execute(new Runnable() {
                        @Override
                        public void run() {
                            register0(promise);
                        }
                    });
                } catch (Throwable t) {
                    logger.warn(
                            "Force-closing a channel whose registration task was not accepted by an event loop: {}",
                            AbstractChannel.this, t);
                    closeForcibly();
                    closeFuture.setClosed();
                    safeSetFailure(promise, t);
                }
            }
        }

        private void register0(ChannelPromise promise) {
            try {
                // check if the channel is still open as it could be closed in the mean time when the register
                // call was outside of the eventLoop
                if (!promise.setUncancellable() || !ensureOpen(promise)) {
                    return;
                }
                boolean firstRegistration = neverRegistered;
                //执行注册动作
                doRegister();
                neverRegistered = false;
                registered = true;

                // Ensure we call handlerAdded(...) before we actually notify the promise. This is needed as the
                // user may already fire events through the pipeline in the ChannelFutureListener.
                //添加Handler
                pipeline.invokeHandlerAddedIfNeeded();

                safeSetSuccess(promise);
                pipeline.fireChannelRegistered();
                // Only fire a channelActive if the channel has never been registered. This prevents firing
                // multiple channel actives if the channel is deregistered and re-registered.
                if (isActive()) {
                    if (firstRegistration) {
                        pipeline.fireChannelActive();
                    } else if (config().isAutoRead()) {
                        // This channel was registered before and autoRead() is set. This means we need to begin read
                        // again so that we process inbound data.
                        //
                        // See https://github.com/netty/netty/issues/4805
                        beginRead();
                    }
                }
            } catch (Throwable t) {
                // Close the channel directly to avoid FD leak.
                closeForcibly();
                closeFuture.setClosed();
                safeSetFailure(promise, t);
            }
        }
    }
}

3.4.6 启动NioEventLoop线程

NioEventLoop线程为慢启动,当需要执行task时,才启动线程。为了保证ServerSocketChannel注册到Selector是在子线程中执行的,在调用register时,会判断NioEventLoop是否启动,如下:

            if (eventLoop.inEventLoop()) {
                register0(promise);
            } else {
                try {
                    //如果eventLoop线程没有启动,先启动线程,再在线程中执行register0方法
                    eventLoop.execute(new Runnable() {
                        @Override
                        public void run() {
                            register0(promise);
                        }
                    });
                } catch (Throwable t) {
                    logger.warn(
                            "Force-closing a channel whose registration task was not accepted by an event loop: {}",
                            AbstractChannel.this, t);
                    closeForcibly();
                    closeFuture.setClosed();
                    safeSetFailure(promise, t);
                }
            }

NioEventLoop的父类SingleThreadEventExecutor维护了一个线程thread对象,默认情况下线程为null,传入的是main线程,因此初始情况下,inEventLoop返回false:

public abstract class SingleThreadEventExecutor extends AbstractScheduledEventExecutor implements OrderedEventExecutor {
    private volatile Thread thread;

    //输入参数为Thread.currentThread()
    public boolean inEventLoop(Thread thread) {
        return thread == this.thread;
    }
}

因此执行else分支。它封装了register0方法成为一个线程对象,传入NioEventLoop#execute方法中,

                   eventLoop.execute(new Runnable() {
                        @Override
                        public void run() {
                            register0(promise);
                        }
                    });

NioEventLoop的execute由其父类SingleThreadEventExecutor实现。先将注册任务加入任务队列中,然后启动线程:

public abstract class SingleThreadEventExecutor extends AbstractScheduledEventExecutor implements OrderedEventExecutor {
    private void execute(Runnable task, boolean immediate) {
        boolean inEventLoop = inEventLoop();
        //先将注册任务加入任务队列中
        addTask(task);
        if (!inEventLoop) {
            //启动线程
            startThread();
            if (isShutdown()) {
                boolean reject = false;
                try {
                    if (removeTask(task)) {
                        reject = true;
                    }
                } catch (UnsupportedOperationException e) {
                    // The task queue does not support removal so the best thing we can do is to just move on and
                    // hope we will be able to pick-up the task before its completely terminated.
                    // In worst case we will log on termination.
                }
                if (reject) {
                    reject();
                }
            }
        }

        if (!addTaskWakesUp && immediate) {
            wakeup(inEventLoop);
        }
    }
}

通过Executor成员启动子线程,而Executor就是在创建NioEventLoop对象时指定的ThreadPerTaskExecutor,上面分析过,每次ThreadPerTaskExecutor#execute会执行 threadFactory.newThread(command).start();命令启动新线程。线程内部如下:

public abstract class SingleThreadEventExecutor extends AbstractScheduledEventExecutor implements OrderedEventExecutor {

    private final Executor executor;

    private void doStartThread() {
        assert thread == null;
        executor.execute(new Runnable() {
            @Override
            public void run() {
                //当前线程为子线程,赋值给thread成员,供下次inEventLoop()判断是否启动子线程
                thread = Thread.currentThread();
                if (interrupted) {
                    thread.interrupt();
                }

                boolean success = false;
                updateLastExecutionTime();
                try {
                    //执行SingleThreadEventExecutor#run方法
                    SingleThreadEventExecutor.this.run();
                    success = true;
                } catch (Throwable t) {
                    logger.warn("Unexpected exception from an event executor: ", t);
                } finally {
                    //省略
                }
            }
        });
    }

run方法由NioEventLoop实现,负责处理发生的事件。它是一个死循环,有两个重要方法,processSelectedKeys负责处理事件,runAllTasks处理事件:

public final class NioEventLoop extends SingleThreadEventLoop {
protected void run() {
        int selectCnt = 0;
        for (;;) {
            try {
                int strategy;
                try {
                    strategy = selectStrategy.calculateStrategy(selectNowSupplier, hasTasks());
                    switch (strategy) {
                    case SelectStrategy.CONTINUE:
                        continue;

                    case SelectStrategy.BUSY_WAIT:
                        // fall-through to SELECT since the busy-wait is not supported with NIO

                    case SelectStrategy.SELECT:
                        long curDeadlineNanos = nextScheduledTaskDeadlineNanos();
                        if (curDeadlineNanos == -1L) {
                            curDeadlineNanos = NONE; // nothing on the calendar
                        }
                        nextWakeupNanos.set(curDeadlineNanos);
                        try {
                            //如果有task执行,就不阻塞select;否则阻塞等待socket事件
                            if (!hasTasks()) {
                                strategy = select(curDeadlineNanos);
                            }
                        } finally {
                            // This update is just to help block unnecessary selector wakeups
                            // so use of lazySet is ok (no race condition)
                            nextWakeupNanos.lazySet(AWAKE);
                        }
                        // fall through
                    default:
                    }
                } catch (IOException e) {
                    // If we receive an IOException here its because the Selector is messed up. Let's rebuild
                    // the selector and retry. https://github.com/netty/netty/issues/8566
                    rebuildSelector0();
                    selectCnt = 0;
                    handleLoopException(e);
                    continue;
                }

                selectCnt++;
                cancelledKeys = 0;
                needsToSelectAgain = false;
                final int ioRatio = this.ioRatio;
                boolean ranTasks;
                //ioRatio表示处理SelectedKeys和执行tasks的事件占比
                //如果ioRatio为100,则执行完所有task
                if (ioRatio == 100) {
                    try {
                        if (strategy > 0) {
                            processSelectedKeys();
                        }
                    } finally {
                        // Ensure we always run tasks.
                        ranTasks = runAllTasks();
                    }
                } else if (strategy > 0) {
                    //如果ioRatio小于100,则只能花费一半处理SelectedKeys时间,去处理task
                    final long ioStartTime = System.nanoTime();
                    try {
                        processSelectedKeys();
                    } finally {
                        // Ensure we always run tasks.
                        final long ioTime = System.nanoTime() - ioStartTime;
                        ranTasks = runAllTasks(ioTime * (100 - ioRatio) / ioRatio);
                    }
                } else {
                    //当ioRatio小于等于0,只执行一个task
                    ranTasks = runAllTasks(0); // This will run the minimum number of tasks
                }

                if (ranTasks || strategy > 0) {
                    if (selectCnt > MIN_PREMATURE_SELECTOR_RETURNS && logger.isDebugEnabled()) {
                        logger.debug("Selector.select() returned prematurely {} times in a row for Selector {}.",
                                selectCnt - 1, selector);
                    }
                    selectCnt = 0;
                } else if (unexpectedSelectorWakeup(selectCnt)) { // Unexpected wakeup (unusual case)
                    selectCnt = 0;
                }
            } catch (CancelledKeyException e) {
                // Harmless exception - log anyway
                if (logger.isDebugEnabled()) {
                    logger.debug(CancelledKeyException.class.getSimpleName() + " raised by a Selector {} - JDK bug?",
                            selector, e);
                }
            } catch (Throwable t) {
                handleLoopException(t);
            }
            // Always handle shutdown even if the loop processing threw an exception.
            try {
                if (isShuttingDown()) {
                    closeAll();
                    if (confirmShutdown()) {
                        return;
                    }
                }
            } catch (Throwable t) {
                handleLoopException(t);
            }
        }
    }
}

由于这时ServerSocketChannel还没有注册到Selector中,因此Selector中不可能有事件,这时就直接开始执行register0这个task。

3.4.7 将ServerSocketChannel注册到selector中

上面发现NioEventLoop线程执行的死循环,最开始执行register0方法:

private void register0(ChannelPromise promise) {
            try {
                // check if the channel is still open as it could be closed in the mean time when the register
                // call was outside of the eventLoop
                if (!promise.setUncancellable() || !ensureOpen(promise)) {
                    return;
                }
                boolean firstRegistration = neverRegistered;
                //注册
                doRegister();
                neverRegistered = false;
                registered = true;

                // Ensure we call handlerAdded(...) before we actually notify the promise. This is needed as the
                // user may already fire events through the pipeline in the ChannelFutureListener.
``  

                safeSetSuccess(promise);
                pipeline.fireChannelRegistered();
                // Only fire a channelActive if the channel has never been registered. This prevents firing
                // multiple channel actives if the channel is deregistered and re-registered.
                if (isActive()) {
                    if (firstRegistration) {
                        pipeline.fireChannelActive();
                    } else if (config().isAutoRead()) {
                        // This channel was registered before and autoRead() is set. This means we need to begin read
                        // again so that we process inbound data.
                        //
                        // See https://github.com/netty/netty/issues/4805
                        beginRead();
                    }
                }
            }
    }

上述方法中,通过doRegister方法注册,很简单就可以看到这是将ServerSocketChannel注册到了Selector中:

public abstract class AbstractNioChannel extends AbstractChannel {
    protected void doRegister() throws Exception {
        boolean selected = false;
        for (;;) {
            try {
                selectionKey = javaChannel().register(eventLoop().unwrappedSelector(), 0, this);
                return;
            } catch (CancelledKeyException e) {
                if (!selected) {
                    // Force the Selector to select now as the "canceled" SelectionKey may still be
                    // cached and not removed because no Select.select(..) operation was called yet.
                    eventLoop().selectNow();
                    selected = true;
                } else {
                    // We forced a select operation on the selector before but the SelectionKey is still cached
                    // for whatever reason. JDK bug ?
                    throw e;
                }
            }
        }
    }
}

3.4.8 执行ServerSocketChannel的pipeline中的initChannel方法

register0执行到pipeline.invokeHandlerAddedIfNeeded();最终执行到DefaultChannelPipeline.callHandlerAddedForAllHandlers(),执行pendingHandlerCallbackHead链表中的Handler:

public class DefaultChannelPipeline implements ChannelPipeline {
    private void callHandlerAddedForAllHandlers() {
        final PendingHandlerCallback pendingHandlerCallbackHead;
        synchronized (this) {
            assert !registered;

            // This Channel itself was registered.
            registered = true;

            pendingHandlerCallbackHead = this.pendingHandlerCallbackHead;
            // Null out so it can be GC'ed.
            this.pendingHandlerCallbackHead = null;
        }

        // This must happen outside of the synchronized(...) block as otherwise handlerAdded(...) may be called while
        // holding the lock and so produce a deadlock if handlerAdded(...) will try to add another handler from outside
        // the EventLoop.
        PendingHandlerCallback task = pendingHandlerCallbackHead;
        while (task != null) {
            task.execute();
            task = task.next;
        }
    }

Handler是在前面代码中放入的。在NioServerSocketChannel初始化时,调用了init方法。它加入了一个netty实现的ChannelInitializer类,内部负责在pipeline中加入自定义的Handler和ServerBootstrapAcceptor这个Handler:

p.addLast(new ChannelInitializer<Channel>() {
            @Override
            public void initChannel(final Channel ch) {
                final ChannelPipeline pipeline = ch.pipeline();
                ChannelHandler handler = config.handler();
                if (handler != null) {
                    pipeline.addLast(handler);
                }

                ch.eventLoop().execute(new Runnable() {
                    @Override
                    public void run() {
                        pipeline.addLast(new ServerBootstrapAcceptor(
                                ch, currentChildGroup, currentChildHandler, currentChildOptions, currentChildAttrs));
                    }
                });
            }
        });

而在addLast方法中,就通过callHandlerCallbackLater方法将ChannelInitializer这个Handler加入到pendingHandlerCallbackHead中:

public final ChannelPipeline addLast(EventExecutorGroup group, String name, ChannelHandler handler) {
        final AbstractChannelHandlerContext newCtx;
        synchronized (this) {
            checkMultiplicity(handler);

            newCtx = newContext(group, filterName(name, handler), handler);

            addLast0(newCtx);

            // If the registered is false it means that the channel was not registered on an eventLoop yet.
            // In this case we add the context to the pipeline and add a task that will call
            // ChannelHandler.handlerAdded(...) once the channel is registered.
            if (!registered) {
                newCtx.setAddPending();
                //将ChannelInitializer这个Handler加入到pendingHandlerCallbackHead中
                callHandlerCallbackLater(newCtx, true);
                return this;
            }

            EventExecutor executor = newCtx.executor();
            if (!executor.inEventLoop()) {
                callHandlerAddedInEventLoop(newCtx, executor);
                return this;
            }
        }
        callHandlerAdded0(newCtx);
        return this;
    }

callHandlerCallbackLater方法将ChannelInitializer加入到链表中:

    private void callHandlerCallbackLater(AbstractChannelHandlerContext ctx, boolean added) {
        assert !registered;

        PendingHandlerCallback task = added ? new PendingHandlerAddedTask(ctx) : new PendingHandlerRemovedTask(ctx);
        PendingHandlerCallback pending = pendingHandlerCallbackHead;
        if (pending == null) {
            pendingHandlerCallbackHead = task;
        } else {
            // Find the tail of the linked-list.
            while (pending.next != null) {
                pending = pending.next;
            }
            pending.next = task;
        }
    }

最终,执行Handler的initChannel方法,如下:

public abstract class ChannelInitializer<C extends Channel> extends ChannelInboundHandlerAdapter {
    public void handlerAdded(ChannelHandlerContext ctx) throws Exception {
        if (ctx.channel().isRegistered()) {
            // This should always be true with our current DefaultChannelPipeline implementation.
            // The good thing about calling initChannel(...) in handlerAdded(...) is that there will be no ordering
            // surprises if a ChannelInitializer will add another ChannelInitializer. This is as all handlers
            // will be added in the expected order.
            if (initChannel(ctx)) {

                // We are done with init the Channel, removing the initializer now.
                removeState(ctx);
            }
        }
    }
}

而initChannel方法则是向pipeline中加入Handler:

                final ChannelPipeline pipeline = ch.pipeline();
                ChannelHandler handler = config.handler();
                if (handler != null) {
                    //这里的Handler时main方法中自定义的Handler
                    pipeline.addLast(handler);
                }
                //在当前子线程中,封装一个task,task中负责将一个ServerBootstrapAcceptor放入pipeline中
                ch.eventLoop().execute(new Runnable() {
                    @Override
                    public void run() {
                        pipeline.addLast(new ServerBootstrapAcceptor(
                                ch, currentChildGroup, currentChildHandler, currentChildOptions, currentChildAttrs));
                    }
                });

在当前ChannelInitializer的方法没执行完前,又加入了一个自定义的Handler:

serverBootstrap.option(ChannelOption.SO_BACKLOG, 128)
                    .handler(new ChannelInitializer<ServerSocketChannel>() {
                        @Override
                        protected void initChannel(ServerSocketChannel ch) throws Exception {
                            ch.pipeline().addLast(new ServerHandler());
                        }
                    })

下一次handlerAdded方法会执行这个initChannel方法,又加入了ServerHandler。

通过pipeline.invokeHandlerAddedIfNeeded();调用,依次执行了netty自定义的ChannelInitializer和main线程中自定义的ChannelInitializer。在ChannelInitializer中,放入了负责处理数据的Handler。这些Handler放到了pipeline中。

3.4.9 执行Pipeline中的handler的channelRegistered方法

调用完initChannel方法后,最后执行pipeline.fireChannelRegistered(),用于执行所有Handler中channelRegistered方法,fireChannelRegistered方法最终执行AbstractChannelHandlerContext#invokeChannelRegistered,开始执行pipeline中的handler实现的channelRegistered方法:

abstract class AbstractChannelHandlerContext implements ChannelHandlerContext, ResourceLeakHint {
    private void invokeChannelRegistered() {
        if (invokeHandler()) {
            try {
                ((ChannelInboundHandler) handler()).channelRegistered(this);
            } catch (Throwable t) {
                invokeExceptionCaught(t);
            }
        } else {
            fireChannelRegistered();
        }
    }

由于pipeline中加入了自定义的NettyHandler类。该类实现了channelRegistered方法。最终必定会执行该方法:

public class ServerHandler extends ChannelInboundHandlerAdapter {
    @Override
    public void channelRegistered(ChannelHandlerContext ctx) throws Exception {
        System.out.println("channelRegistered");
        super.channelRegistered(ctx);
    }
}

执行到super.channelRegistered(ctx),会尝试找下一个Handler执行。如下,继续执行invokeChannelRegistered方法:

abstract class AbstractChannelHandlerContext implements ChannelHandlerContext, ResourceLeakHint {
    public ChannelHandlerContext fireChannelRegistered() {
        invokeChannelRegistered(findContextInbound(MASK_CHANNEL_REGISTERED));
        return this;
    }
}

其中,findContextInbound方法非常重要。他负责找下一个Handler:

abstract class AbstractChannelHandlerContext implements ChannelHandlerContext, ResourceLeakHint {
    private AbstractChannelHandlerContext findContextInbound(int mask) {
        AbstractChannelHandlerContext ctx = this;
        EventExecutor currentExecutor = executor();
        do {
            ctx = ctx.next;
        } while (skipContext(ctx, currentExecutor, mask, MASK_ONLY_INBOUND));
        return ctx;
    }
}

最终,通过在每个channelRegistered方法最后,加入super.channelRegistered(ctx)语句,就可以执行完所有责任链中所有Handler的channelRegistered方法。对于其他方法,比如channelActivechannelRead,通过加入类似语句,同样可以执行完责任链中的所有相关方法。

3.4.10 绑定端口

在上面ServerSocketChannel绑定到Selector后,且自定义的Handler和最重要的ServerBootstrapAcceptor这个Handler加入pipeline后,就可以将ServerSocketChannel绑定端口,提供服务了:

public abstract class AbstractBootstrap<B extends AbstractBootstrap<B, C>, C extends Channel> implements Cloneable {
    private ChannelFuture doBind(final SocketAddress localAddress) {
        final ChannelFuture regFuture = initAndRegister();
        final Channel channel = regFuture.channel();
        if (regFuture.cause() != null) {
            return regFuture;
        }

        if (regFuture.isDone()) {
            // At this point we know that the registration was complete and successful.
            ChannelPromise promise = channel.newPromise();
            //绑定端口
            doBind0(regFuture, channel, localAddress, promise);
            return promise;
        } 
    }
}

3.4.11 ServerSocketChannel接受客户端请求

上述register注册任务完成,并且bind完成后,NioEventLoop子线程也在进行死循环。只要task队列没有任务,就阻塞进行select,如果阻塞超时或者有socket事件发生,后面就调用processSelectedKeys方法处理socket事件:

protected void run() {
        int selectCnt = 0;
        for (;;) {
            try {
                int strategy;
                try {
                    strategy = selectStrategy.calculateStrategy(selectNowSupplier, hasTasks());
                    switch (strategy) {
                    case SelectStrategy.CONTINUE:
                        continue;

                    case SelectStrategy.BUSY_WAIT:
                        // fall-through to SELECT since the busy-wait is not supported with NIO

                    case SelectStrategy.SELECT:
                        long curDeadlineNanos = nextScheduledTaskDeadlineNanos();
                        if (curDeadlineNanos == -1L) {
                            curDeadlineNanos = NONE; // nothing on the calendar
                        }
                        nextWakeupNanos.set(curDeadlineNanos);
                        try {
                            if (!hasTasks()) {
                                strategy = select(curDeadlineNanos);
                            }
                        } finally {
                            // This update is just to help block unnecessary selector wakeups
                            // so use of lazySet is ok (no race condition)
                            nextWakeupNanos.lazySet(AWAKE);
                        }
                        // fall through
                    default:
                    }
                }//省略

                selectCnt++;
                cancelledKeys = 0;
                needsToSelectAgain = false;
                final int ioRatio = this.ioRatio;
                boolean ranTasks;
                if (ioRatio == 100) {
                    try {
                        if (strategy > 0) {
                            //处理socket事件
                            processSelectedKeys();
                        }
                    } finally {
                        // Ensure we always run tasks.
                        ranTasks = runAllTasks();
                    }
                } else if (strategy > 0) {
                    final long ioStartTime = System.nanoTime();
                    try {
                        processSelectedKeys();
                    } finally {
                        // Ensure we always run tasks.
                        final long ioTime = System.nanoTime() - ioStartTime;
                        ranTasks = runAllTasks(ioTime * (100 - ioRatio) / ioRatio);
                    }
                } else {
                    ranTasks = runAllTasks(0); // This will run the minimum number of tasks
                }
                //省略
            }
        }
    }

NioEventLoop类中的select()方法就是通过EPollSelectorImpl的select()方法阻塞等待socket事件发生,当然会设置阻塞的超时时间:

public final class NioEventLoop extends SingleThreadEventLoop {

    private Selector selector;

    private int select(long deadlineNanos) throws IOException {
        if (deadlineNanos == NONE) {
            return selector.select();
        }
        // Timeout will only be 0 if deadline is within 5 microsecs
        long timeoutMillis = deadlineToDelayNanos(deadlineNanos + 995000L) / 1000000L;
        return timeoutMillis <= 0 ? selector.selectNow() : selector.select(timeoutMillis);
    }
}

每次EPollSelectorImpl有socket事件发生时,EPollSelectorImpl多路复用器中的selectedKeySet成员就会更新。在之前创建EPollSelectorImpl时,就指定NioEventLoop的成员selectedKeys指向EPollSelectorImpl对象的selectedKeySet成员:

public final class NioEventLoop extends SingleThreadEventLoop {
    private SelectorTuple openSelector() {
        final Selector unwrappedSelector;
        try {
            unwrappedSelector = provider.openSelector();
        } catch (IOException e) {
            throw new ChannelException("failed to open a new selector", e);
        }

        if (DISABLE_KEY_SET_OPTIMIZATION) {
            return new SelectorTuple(unwrappedSelector);
        }

        Object maybeSelectorImplClass = AccessController.doPrivileged(new PrivilegedAction<Object>() {
            @Override
            public Object run() {
                try {
                    return Class.forName(
                            "sun.nio.ch.SelectorImpl",
                            false,
                            PlatformDependent.getSystemClassLoader());
                } catch (Throwable cause) {
                    return cause;
                }
            }
        });

        if (!(maybeSelectorImplClass instanceof Class) ||
            // ensure the current selector implementation is what we can instrument.
            !((Class<?>) maybeSelectorImplClass).isAssignableFrom(unwrappedSelector.getClass())) {
            if (maybeSelectorImplClass instanceof Throwable) {
                Throwable t = (Throwable) maybeSelectorImplClass;
                logger.trace("failed to instrument a special java.util.Set into: {}", unwrappedSelector, t);
            }
            return new SelectorTuple(unwrappedSelector);
        }

        final Class<?> selectorImplClass = (Class<?>) maybeSelectorImplClass;
        final SelectedSelectionKeySet selectedKeySet = new SelectedSelectionKeySet();

        Object maybeException = AccessController.doPrivileged(new PrivilegedAction<Object>() {
            @Override
            public Object run() {
                try {
                    //获取EPollSelectorImpl的selectedKeys字段
                    Field selectedKeysField = selectorImplClass.getDeclaredField("selectedKeys");
                    //获取EPollSelectorImpl的publicSelectedKeys字段
                    Field publicSelectedKeysField = selectorImplClass.getDeclaredField("publicSelectedKeys");

                    if (PlatformDependent.javaVersion() >= 9 && PlatformDependent.hasUnsafe()) {
                        // Let us try to use sun.misc.Unsafe to replace the SelectionKeySet.
                        // This allows us to also do this in Java9+ without any extra flags.
                        long selectedKeysFieldOffset = PlatformDependent.objectFieldOffset(selectedKeysField);
                        long publicSelectedKeysFieldOffset =
                                PlatformDependent.objectFieldOffset(publicSelectedKeysField);

                        if (selectedKeysFieldOffset != -1 && publicSelectedKeysFieldOffset != -1) {
                            PlatformDependent.putObject(
                                    unwrappedSelector, selectedKeysFieldOffset, selectedKeySet);
                            PlatformDependent.putObject(
                                    unwrappedSelector, publicSelectedKeysFieldOffset, selectedKeySet);
                            return null;
                        }
                        // We could not retrieve the offset, lets try reflection as last-resort.
                    }
                    //设置selectedKeysField可见
                    Throwable cause = ReflectionUtil.trySetAccessible(selectedKeysField, true);
                    if (cause != null) {
                        return cause;
                    }
                    //设置publicSelectedKeysField可见
                    cause = ReflectionUtil.trySetAccessible(publicSelectedKeysField, true);
                    if (cause != null) {
                        return cause;
                    }
                    //设置unwrappedSelector的selectedKeysField对象为上述可见对象
                    selectedKeysField.set(unwrappedSelector, selectedKeySet);
                    //设置unwrappedSelector的unwrappedSelector对象为上述可见对象
                    publicSelectedKeysField.set(unwrappedSelector, selectedKeySet);
                    return null;
                } catch (NoSuchFieldException e) {
                    return e;
                } catch (IllegalAccessException e) {
                    return e;
                }
            }
        });

        if (maybeException instanceof Exception) {
            selectedKeys = null;
            Exception e = (Exception) maybeException;
            logger.trace("failed to instrument a special java.util.Set into: {}", unwrappedSelector, e);
            return new SelectorTuple(unwrappedSelector);
        }
        //selectedKeys成员变量就是上述EPollSelectorImpl的selectedKeySet成员变量
        selectedKeys = selectedKeySet;
        logger.trace("instrumented a special java.util.Set into: {}", unwrappedSelector);
        //返回EPollSelectorImpl对象即其包装类对象
        return new SelectorTuple(unwrappedSelector,
                                 new SelectedSelectionKeySetSelector(unwrappedSelector, selectedKeySet));
    }
}

有了上述知识后,就可以正式开始开始看selectedKeys的处理。如下,正常情况下,socket事件对象selectedKeys不为空,直接走processSelectedKeysOptimized方法处理事件否;则直接访问EPollSelectorImpl的selectedKeys字段:

public final class NioEventLoop extends SingleThreadEventLoop {
    private void processSelectedKeys() {
        if (selectedKeys != null) {
            //处理事件
            processSelectedKeysOptimized();
        } else {
            //处理EPollSelectorImpl事件,这部分不展开
            processSelectedKeysPlain(selector.selectedKeys());
        }
    }
}

processSelectedKeysOptimized方法中,遍历所有发生事件的key,此时key对应的k.attachment()就是NioServerSocketChannel,直接处理该NioServerSocketChannel上的事件:

public final class NioEventLoop extends SingleThreadEventLoop {
    private void processSelectedKeysOptimized() {
        for (int i = 0; i < selectedKeys.size; ++i) {
            final SelectionKey k = selectedKeys.keys[i];
            // null out entry in the array to allow to have it GC'ed once the Channel close
            // See https://github.com/netty/netty/issues/2363
            selectedKeys.keys[i] = null;

            final Object a = k.attachment();

            if (a instanceof AbstractNioChannel) {
                //处理NioServerSocketChannel上的事件
                processSelectedKey(k, (AbstractNioChannel) a);
            } else {
                //不展开
                @SuppressWarnings("unchecked")
                NioTask<SelectableChannel> task = (NioTask<SelectableChannel>) a;
                processSelectedKey(k, task);
            }

            if (needsToSelectAgain) {
                // null out entries in the array to allow to have it GC'ed once the Channel close
                // See https://github.com/netty/netty/issues/2363
                selectedKeys.reset(i + 1);

                selectAgain();
                i = -1;
            }
        }
    }
}

NioEventLoop#processSelectedKey负责判断socket事件类型,并调用不同方法处理对应事件:

public final class NioEventLoop extends SingleThreadEventLoop {
    private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
        final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();
        if (!k.isValid()) {
            final EventLoop eventLoop;
            try {
                eventLoop = ch.eventLoop();
            } catch (Throwable ignored) {
                // If the channel implementation throws an exception because there is no event loop, we ignore this
                // because we are only trying to determine if ch is registered to this event loop and thus has authority
                // to close ch.
                return;
            }
            // Only close ch if ch is still registered to this EventLoop. ch could have deregistered from the event loop
            // and thus the SelectionKey could be cancelled as part of the deregistration process, but the channel is
            // still healthy and should not be closed.
            // See https://github.com/netty/netty/issues/5125
            if (eventLoop == this) {
                // close the channel if the key is not valid anymore
                unsafe.close(unsafe.voidPromise());
            }
            return;
        }

        try {
            //事件类型
            int readyOps = k.readyOps();
            // We first need to call finishConnect() before try to trigger a read(...) or write(...) as otherwise
            // the NIO JDK channel implementation may throw a NotYetConnectedException.
            //客户端SocketChannel才会触发OP_CONNECT事件,服务端不会触发该事件
            if ((readyOps & SelectionKey.OP_CONNECT) != 0) {
                // remove OP_CONNECT as otherwise Selector.select(..) will always return without blocking
                // See https://github.com/netty/netty/issues/924
                int ops = k.interestOps();
                ops &= ~SelectionKey.OP_CONNECT;
                k.interestOps(ops);

                unsafe.finishConnect();
            }
            //处理写事件
            // Process OP_WRITE first as we may be able to write some queued buffers and so free memory.
            if ((readyOps & SelectionKey.OP_WRITE) != 0) {
                // Call forceFlush which will also take care of clear the OP_WRITE once there is nothing left to write
                ch.unsafe().forceFlush();
            }
            //处理读数据事件和客户端连接事件
            // Also check for readOps of 0 to workaround possible JDK bug which may otherwise lead
            // to a spin loop
            if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
                unsafe.read();
            }
        } catch (CancelledKeyException ignored) {
            unsafe.close(unsafe.voidPromise());
        }
    }
}

当用户连接时,执行unsafe.read(),unsafe是NioServerSocketChannel创建时,其父类AbstractNioMessageChannel的内部类NioMessageUnsafe。其中有做重要的方法doReadMessages:

public abstract class AbstractNioMessageChannel extends AbstractNioChannel {
    private final class NioMessageUnsafe extends AbstractNioUnsafe {

        private final List<Object> readBuf = new ArrayList<Object>();

        @Override
        public void read() {
            assert eventLoop().inEventLoop();
            final ChannelConfig config = config();
            final ChannelPipeline pipeline = pipeline();
            final RecvByteBufAllocator.Handle allocHandle = unsafe().recvBufAllocHandle();
            allocHandle.reset(config);

            boolean closed = false;
            Throwable exception = null;
            try {
                try {
                    do {
                        //readBuf放入的新连接SocketChannel
                        int localRead = doReadMessages(readBuf);
                        if (localRead == 0) {
                            break;
                        }
                        if (localRead < 0) {
                            closed = true;
                            break;
                        }

                        allocHandle.incMessagesRead(localRead);
                    } while (allocHandle.continueReading());
                } catch (Throwable t) {
                    exception = t;
                }

                int size = readBuf.size();
                for (int i = 0; i < size; i ++) {
                    readPending = false;
                    //执行pipeline责任链中的read方法
                    pipeline.fireChannelRead(readBuf.get(i));
                }
                readBuf.clear();
                allocHandle.readComplete();
                pipeline.fireChannelReadComplete();

                if (exception != null) {
                    closed = closeOnReadError(exception);

                    pipeline.fireExceptionCaught(exception);
                }

                if (closed) {
                    inputShutdown = true;
                    if (isOpen()) {
                        close(voidPromise());
                    }
                }
            } finally {
                // Check if there is a readPending which was not processed yet.
                // This could be for two reasons:
                // * The user called Channel.read() or ChannelHandlerContext.read() in channelRead(...) method
                // * The user called Channel.read() or ChannelHandlerContext.read() in channelReadComplete(...) method
                //
                // See https://github.com/netty/netty/issues/2254
                if (!readPending && !config.isAutoRead()) {
                    removeReadOp();
                }
            }
        }
    }
}

doReadMessages由NioServerSocketChannel实现。可以看到SocketUtils.accept方法:

public class NioServerSocketChannel extends AbstractNioMessageChannel
                             implements io.netty.channel.socket.ServerSocketChannel {
    protected int doReadMessages(List<Object> buf) throws Exception {
        SocketChannel ch = SocketUtils.accept(javaChannel());

        try {
            if (ch != null) {
                buf.add(new NioSocketChannel(this, ch));
                return 1;
            }
        } catch (Throwable t) {
            logger.warn("Failed to create a new channel from an accepted socket.", t);

            try {
                ch.close();
            } catch (Throwable t2) {
                logger.warn("Failed to close a socket.", t2);
            }
        }

        return 0;
    }
}

此时可以看到ServerSocketChannel调用accept方法并返回SocketChannel:

public final class SocketUtils {
    public static SocketChannel accept(final ServerSocketChannel serverSocketChannel) throws IOException {
        try {
            return AccessController.doPrivileged(new PrivilegedExceptionAction<SocketChannel>() {
                @Override
                public SocketChannel run() throws IOException {
                    return serverSocketChannel.accept();
                }
            });
        } catch (PrivilegedActionException e) {
            throw (IOException) e.getCause();
        }
    }
}

返回后,将SocketChannel封装成NioSocketChannel,并创建NioSocketChannel实例,放入buf中。然后调用NioServerSocketChannel的pipeline的fireChannelRead方法。

3.4.12 ServerSocketChannel将SocketChannel放入workerGroup中

上一节中产生了SocketChannel,并即将执行NioServerSocketChannel的pipeline的fireChannelRead。它会执行所有Handler的channelRead方法,执行到最后一个channelRead时,会在childGroup中注册该SocketChannel:

public class ServerBootstrap extends AbstractBootstrap<ServerBootstrap, ServerChannel> {
    private static class ServerBootstrapAcceptor extends ChannelInboundHandlerAdapter {
       public void channelRead(ChannelHandlerContext ctx, Object msg) {
            final Channel child = (Channel) msg;

            child.pipeline().addLast(childHandler);

            setChannelOptions(child, childOptions, logger);
            setAttributes(child, childAttrs);

            try {
                //workerGroup注册SocketChannel
                childGroup.register(child).addListener(new ChannelFutureListener() {
                    @Override
                    public void operationComplete(ChannelFuture future) throws Exception {
                        if (!future.isSuccess()) {
                            forceClose(child, future.cause());
                        }
                    }
                });
            } catch (Throwable t) {
                forceClose(child, t);
            }
        }
    }
}

3.4.13 创建NioSocketChannel对象

Nio创建如下:

public class NioSocketChannel extends AbstractNioByteChannel implements io.netty.channel.socket.SocketChannel {
    public NioSocketChannel(SocketChannel socket) {
        this(null, socket);
    }

    public NioSocketChannel(Channel parent, SocketChannel socket) {
        super(parent, socket);
        config = new NioSocketChannelConfig(this, socket.socket());
    }
}

监控SocketChannel的OP_READ事件:

public abstract class AbstractNioByteChannel extends AbstractNioChannel {
    protected AbstractNioByteChannel(Channel parent, SelectableChannel ch) {
        super(parent, ch, SelectionKey.OP_READ);
    }
}

配置socket为非阻塞,这样read操作就是非阻塞的了:

public abstract class AbstractNioChannel extends AbstractChannel {
    protected AbstractNioChannel(Channel parent, SelectableChannel ch, int readInterestOp) {
        super(parent);
        this.ch = ch;
        this.readInterestOp = readInterestOp;
        try {
            ch.configureBlocking(false);
        } catch (IOException e) {
            try {
                ch.close();
            } catch (IOException e2) {
                logger.warn(
                            "Failed to close a partially initialized socket.", e2);
            }

            throw new ChannelException("Failed to enter non-blocking mode.", e);
        }
    }
}

最后初始化unsafe成员和pipeline:

public abstract class AbstractChannel extends DefaultAttributeMap implements Channel {
    protected AbstractChannel(Channel parent) {
        this.parent = parent;
        id = newId();
        unsafe = newUnsafe();
        pipeline = newChannelPipeline();
    }
}

NioSocketChannel的父类AbstractNioByteChannel有内部类NioByteUnsafe,该类就是NioSocketChannel就是NioSocketChannel的unsafe类型:

public abstract class AbstractNioByteChannel extends AbstractNioChannel {
    protected class NioByteUnsafe extends AbstractNioUnsafe {

         public final void read() {
            final ChannelConfig config = config();
            if (shouldBreakReadReady(config)) {
                clearReadPending();
                return;
            }
            final ChannelPipeline pipeline = pipeline();
            final ByteBufAllocator allocator = config.getAllocator();
            final RecvByteBufAllocator.Handle allocHandle = recvBufAllocHandle();
            allocHandle.reset(config);

            ByteBuf byteBuf = null;
            boolean close = false;
            try {
                do {
                    byteBuf = allocHandle.allocate(allocator);
                    allocHandle.lastBytesRead(doReadBytes(byteBuf));
                    if (allocHandle.lastBytesRead() <= 0) {
                        // nothing was read. release the buffer.
                        byteBuf.release();
                        byteBuf = null;
                        close = allocHandle.lastBytesRead() < 0;
                        if (close) {
                            // There is nothing left to read as we received an EOF.
                            readPending = false;
                        }
                        break;
                    }

                    allocHandle.incMessagesRead(1);
                    readPending = false;
                    pipeline.fireChannelRead(byteBuf);
                    byteBuf = null;
                } while (allocHandle.continueReading());

                allocHandle.readComplete();
                pipeline.fireChannelReadComplete();

                if (close) {
                    closeOnRead(pipeline);
                }
            } catch (Throwable t) {
                handleReadException(pipeline, byteBuf, t, close, allocHandle);
            } finally {
                // Check if there is a readPending which was not processed yet.
                // This could be for two reasons:
                // * The user called Channel.read() or ChannelHandlerContext.read() in channelRead(...) method
                // * The user called Channel.read() or ChannelHandlerContext.read() in channelReadComplete(...) method
                //
                // See https://github.com/netty/netty/issues/2254
                if (!readPending && !config.isAutoRead()) {
                    removeReadOp();
                }
            }
        }
    }
}

3.4.13 在NioSocketChannel对象的pipeline中添加handler

NioServerSocketChannel中,将childHandler传入NioSocketChannel的pipeline中。而childHandler就是在ServerBootstrap中的init方法中获取到的自定义NettyHandler:

private static class ServerBootstrapAcceptor extends ChannelInboundHandlerAdapter {
        public void channelRead(ChannelHandlerContext ctx, Object msg) {
            final Channel child = (Channel) msg;

            child.pipeline().addLast(childHandler);

            setChannelOptions(child, childOptions, logger);
            setAttributes(child, childAttrs);

            try {
                childGroup.register(child).addListener(new ChannelFutureListener() {
                    @Override
                    public void operationComplete(ChannelFuture future) throws Exception {
                        if (!future.isSuccess()) {
                            forceClose(child, future.cause());
                        }
                    }
                });
            } catch (Throwable t) {
                forceClose(child, t);
            }
        }
}

3.4.14 NioSocketChannel注册到selector中并读取数据

注册同样复用NioServerSocketChannel的register方法,即在workerGroup的子线程中执行register方法。读取数据则用NioByteUnsafe的read方法,最后就是执行自定义childHandler的责任链上的read方法,不重复赘述。

3.4.15 启动流程总结

从上面流程可以看到,Netty的启动流程底层就是多路复用的NIO代码,之所以这么复杂,是因为Netty基于pipeline责任链进行开发,为了方便用户直接在自定义Handler中写业务逻辑,Netty框架本身为责任链模式开发了复杂的逻辑。

4. Netty客户端简述

客户端代码示例如下,发送数据的代码在main方法中,而接受服务端响应的代码在自定义Handler中:

public class LoginClient {
    public static void main(String[] args) {
        //
        NioEventLoopGroup workerGroup = new NioEventLoopGroup();

        Bootstrap bootstrap = new Bootstrap();
        bootstrap.group(workerGroup)
                .channel(NioSocketChannel.class)
                .handler(new ChannelInitializer<SocketChannel>() {
                    @Override
                    protected void initChannel(SocketChannel sc) throws Exception {
                        sc.pipeline().addLast(new ClientHandler());
                    }
                });
        //连接服务端
        connect(bootstrap,"127.0.0.1",8000,null);
    }

    private static void connect(Bootstrap bootstrap,String host,Integer port,Integer retry) {
        bootstrap.connect(host,port).addListener(future -> {
            System.out.println("future状态 :" + future.isSuccess());
            //如果连接成功,就向服务端发送数据
            if (future.isSuccess()) {
                Channel channel = ((ChannelFuture) future).channel();
                startConsoleThread(channel);
            }
        });
    }

    private static void startConsoleThread(Channel channel) {
        //新建一个线程,向服务端发送数据
        new Thread(() -> {
            while (!Thread.interrupted()) {
                System.out.println("channel状态 :" + channel);
                if (LoginUtil.hasLogin(channel)) {
                    System.out.println("输入消息发送至服务端 : ");
                    Scanner sc = new Scanner(System.in);
                    String line = sc.nextLine();

                    MessageRequestPacket packet = new MessageRequestPacket();
                    packet.setMessage(line);

                    ByteBuf byteBuf = PacketCodeC.INSTANCE.encode(channel.alloc().ioBuffer(), packet);
                    channel.writeAndFlush(byteBuf);
                }
            }
        }).start();
    }
}

客户端将BootStrap作为启动类,通过connect方法建立与服务端的连接,同时将SocketChannel放入到NioEventLoop中,监听SocketChannel是否有事件发生。对于客户端NioEventLoopGroup线程大小,推荐设置如下:

  • 对于客户端来说,如果只是连接的一个server节点,那么只要设置1条线程即可。即使出现了断线重连,在连接断开之后,之前的Channel会从NioEventLoop移除。重连之后,仍然只会在仅有的一个NioEventLoop注册一个新的Channel。
  • 如果客户端同时如下方式多次调用io.netty.bootstrap.Bootstrap#connect(String inetHost, int inetPort)连接多个Server节点,那么线程可以设置大一点,但不要超过2*c,而且只要出现断线重连,同样不能保证每个NioEventLoop都会绑定一个客户端Channel。

备注:客户端并没有采用Ractor架构,也没办法采用Ractor架构,因此客户端用netty其实没有性能上的收益,甚至linux的nc命令都可以取代netty客户端。不过客户端使用pipeline能够方便业务开发。

5. 基于NIO的netty Demo

从上面的逻辑可以看到,Netty = Reactor架构的NIO + 责任链。责任链模式不能优化NIO服务端性能,因此我写了省略责任链部分代码的Netty Demo:

import java.io.IOException;
import java.net.InetSocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.ServerSocketChannel;
import java.nio.channels.SocketChannel;
import java.util.Iterator;
import java.util.Set;

public class NettyDemo {
    public static void main(String[] args) throws IOException {
        //创建ServerSocket
        ServerSocketChannel ss = ServerSocketChannel.open();
        System.out.println("Step1: new ServerSocketChannel(8890) ");
        ss.configureBlocking(false);
        //创建多路复用器,多路复用表示使用一个selector对象监控多个socket状态,监控过程发生在内核态。
        Selector selector = Selector.open();
        //将ServerSocket注册到多路复用器中,多路复用器负责监听OP_ACCEPT事件
        ss.register(selector, SelectionKey.OP_ACCEPT);
        System.out.println("step2:register serversocketchannel in selector");
        //workerGroup大小
        int thread_cnt = 10;
        //10个thread,10个selector,通过下标起到对应关系,即第一个thread和selector搭配在一起执行,...,第十个thread和第十个selector搭配在一起执行
        Thread[] threads = new Thread[thread_cnt];
        Selector[] selectors = new Selector[thread_cnt];
        boolean[] threadStatus = new boolean[thread_cnt];
        for (int i = 0; i < thread_cnt; ++i) {
            selectors[i] = Selector.open();
            threadStatus[i] = false;
            int finalI = i;
            //每个workerGroup的每个NioEventLoop遍历读事件
            threads[i] = new Thread(() -> {
                try {
                    while (selectors[finalI].select() > 0) {
                        //获取状态发生变化的socket
                        Set<SelectionKey> childSelectionKeys = selectors[finalI].selectedKeys();
                        Iterator<SelectionKey> childSelectionKeysIterator = childSelectionKeys.iterator();
                        //遍历上述socket
                        while (childSelectionKeysIterator.hasNext()) {
                            SelectionKey childKey = childSelectionKeysIterator.next();
                            //如果是socket,就打印客户端传递过来的数据
                            if (childKey.isReadable()) {
                                SocketChannel clientChannel = (SocketChannel) childKey.channel();
                                ByteBuffer byteBuffer = ByteBuffer.allocate(4096);
                                int num = clientChannel.read(byteBuffer);
                                if (num > 0) {
                                    byteBuffer.flip();
                                    byte[] bytes = new byte[byteBuffer.limit()];
                                    byteBuffer.get(bytes);
                                    String data = new String(bytes);
                                    System.out.println(data);
                                    byteBuffer.clear();
                                }
                                System.out.println("step5:process client data in thread: " + Thread.currentThread().getName());
                            }
                            //处理完上述socket后,一定要从iterator列表中删除该socket,因为selector不会主动删除处理结束的socket
                            childSelectionKeysIterator.remove();
                        }
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }, "workerGroup-NioEventLooop-" + finalI);
        }

        //只有一个bossGroup,在boosGroup线程中注册socketChannel到selector中
        new Thread(() -> {
            try {
                int idx = 0;
                ss.bind(new InetSocketAddress(8890));
                //阻塞式地等待多路复用器监听的socket状态,只有socket状态发生变化,才会结束阻塞。selectNow为非阻塞,不过非阻塞会死循环,浪费CPU资源
                //将状态发生变化的socket放入内核的selectedKeys区域
                while (selector.select() > 0) {
                    //获取状态发生变化的socket
                    Set<SelectionKey> selectionKeys = selector.selectedKeys();
                    Iterator<SelectionKey> selectionKeysIterator = selectionKeys.iterator();
                    //遍历上述socket
                    while (selectionKeysIterator.hasNext()) {
                        SelectionKey key = selectionKeysIterator.next();
                        //如果是serversocket,接受accept请求,创建并注册新的socket到selector中
                        if (key.isAcceptable()) {
                            SocketChannel client = ss.accept();
                            InetSocketAddress addr = (InetSocketAddress) client.getRemoteAddress();
                            System.out.println("step3:client\t" + addr.getPort());
                            client.configureBlocking(false);
                            //轮训遍历NioEventLoop
                            int curr_idx = idx % thread_cnt;
                            client.register(selectors[curr_idx], SelectionKey.OP_READ);
                            System.out.println("step4:register socketchannel in selector num: " + curr_idx + " current thread name: " + Thread.currentThread().getName());
                            if (threadStatus[curr_idx] == false) {
                                threadStatus[curr_idx] = true;
                                threads[curr_idx].start();
                            }
                        }
                        idx++;
                    }
                    //处理完上述socket后,一定要从iterator列表中删除该socket,因为selector不会主动删除处理结束的socket
                    selectionKeysIterator.remove();
                }
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
        }, "bossGroup-NioEventLoop-1").start();
    }
}

分别启动3个客户端访问服务端。可以发现bossGroup将三个socket连接均匀分配到3个线程的selector中,在大量连接场景中,就能实现高性能服务器:

image.png

代码总结:从上面看到代码可以看到,要实现高性能Reactor架构的服务器,实现100行最核心的代码即可。

6. Netty常见问题

6.1 Netty长连接与短连接

TCP层面的连接保活机制

Netty基于socket Java API建立连接。默认情况下Socket基于TCP通信,TCP默认通过SO_KEEPALIVE配置心跳监听TCP连接状态。

当在TCP网络套接字上设置SO_KEEPALIVE选项时,计时器将在套接字空闲时开始运行。当超过keepalive空闲时间,socket没有进一步的活动时,内核将向client发送一个“keepalive packet”。如果client应答,则认为连接良好,计时器再次开始运行。

如果没有应答,内核在发送另一个keepalive包之前会等待keepalive间隔时间。重复这个过程,直到发送keepalive报文的数量达到keepalive计数。在那之后,连接被认为已失效,尝试使用网络套接字将导致错误。

注意,发送keepalive消息的是操作系统内核,而不是应用程序(数据库server或client)。应用程序不知道这个过程。

keepalive有两个目的:

  • 保持网络不空闲,如上,间隔性的发送keepalive packet
  • 检测另一个通信端是否已经断开,比如断电导致的异常连接失效,通过该设置,可以关闭网络连接。

TPC keepalive参数如下:

tcp_keepalive_intvl (integer; default: 75; since Linux 2.4)
       The number of seconds between TCP keep-alive probes.

tcp_keepalive_probes (integer; default: 9; since Linux 2.2)
       The  maximum  number  of  TCP  keep-alive  probes  to send before giving up and killing the connection if no
       response is obtained from the other end.

tcp_keepalive_time (integer; default: 7200; since Linux 2.2)
       The number of seconds a connection needs to be idle before TCP begins sending out keep-alive probes.   Keep-
       alives  are  sent only when the SO_KEEPALIVE socket option is enabled.  The default value is 7200 seconds (2
       hours).  An idle connection is terminated after approximately an additional 11 minutes (9 probes an interval
       of 75 seconds apart) when keep-alive is enabled.

Netty应用层面的长连接/短连接逻辑

长连接和短连接的区别不是根据netty区分的,而是根据你的服务区分的。比如以tcp通信为例,不管是长连接还是短连接,都是socket通信,区别在于,长连接在建立tcp握手成功之后,就不会主动断开连接,而是保持连接,而短连接是一次请求结束之后,服务器就会主动断开连接,以释放资源。

Netty其实并不能感知到TCP层面的keepAlive流程,TCP层面的keepAlive也不能决定Netty应用层何时关闭连接。

  • 可以理解当Netty客户端发送完请求数据,就调用socket.close断开连接,就是短连接。
  • 当Netty客户端发送完请求数据后,还维持连接,就是长连接。

如下,实现了服务端长连接逻辑:

  • 输入1表示发心跳包。服务端应用层通过心跳包决定保持连接。
  • 输入2表示发content。
  • 5秒内不输入1则服务端会自动断开连接。
public class LongConnTest {

    private Logger logger = LoggerFactory.getLogger(LongConnTest.class);

    String host = "localhost";
    int port = 8080;

    public void testLongConn() throws Exception {
        logger.debug("start");
        final Socket socket = new Socket();
        socket.connect(new InetSocketAddress(host, port));
        Scanner scanner = new Scanner(System.in);
        new Thread(() -> {
            while (true) {
                try {
                    byte[] input = new byte[64];
                    int readByte = socket.getInputStream().read(input);
                    logger.debug("readByte " + readByte);
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }).start();
        int code;
        while (true) {
            code = scanner.nextInt();
            logger.debug("input code:" + code);
            if (code == 0) {
                break;
            } else if (code == 1) {
                ByteBuffer byteBuffer = ByteBuffer.allocate(5);
                byteBuffer.put((byte) 1);
                byteBuffer.putInt(0);
                socket.getOutputStream().write(byteBuffer.array());
                logger.debug("write heart finish!");
            } else if (code == 2) {
                byte[] content = ("hello, I'm" + hashCode()).getBytes();
                ByteBuffer byteBuffer = ByteBuffer.allocate(content.length + 5);
                byteBuffer.put((byte) 2);
                byteBuffer.putInt(content.length);
                byteBuffer.put(content);
                socket.getOutputStream().write(byteBuffer.array());
                logger.debug("write content finish!");
            }
        }
        socket.close();
    }

    // 因为Junit不支持用户输入,所以用main的方式来执行用例
    public static void main(String[] args) throws Exception {
        new LongConnTest().testLongConn();
    }
}

6.2 粘包与拆包

TCP流与UDP包

在了解粘包与拆包之前,需要了解TCP流和UDP包的区别:

UDP建立在网络不稳定的基础上。UDP报头格式如下所示,它携带了UDP数据包总大小的字段。当数据包传输到接受方的传输层时,传输层根据包大小,将相应大小的数据传输给应用层。
image.png

TCP建立在网络稳定的基础上。TCP报文格式如下所示,它没有携带数据包总大小。这是因为TCP以流的方式传输数据,应用层将数据先写到socket发送缓冲区中,缓冲区满了,再通过TCP发送数据包到接受方。接受方将数据发送到socket接受缓冲区中,应用层读取socket缓冲区数据。这样,单个TCP数据包并不能确定一次消息的大小,因此TCP就没有消息大小的字段了。
image.png

linux中,TCP的流对应是SOCK_STREAM类型;UDP的数据包对应是SOCK_DGRAM类型。

粘包与拆包

同样的一个socket连接,如果应用层分别发送了两次消息,那么socket缓冲区可能会存在两条消息,因为TCP流的原因,TCP不知道每个消息的大小,一次性把两条消息放到一个包中发送出去了。接受方将解析了这个TCP包,可能直接将数据包上报给应用层。这样应用层就把发送的两条消息当成一条消息处理了,这就是粘包。

如果一条消息在接受方socket缓冲区中放不下,socket缓冲区满了,那么应用层只处理了一半的消息,另一半的消息下次处理。这样发送的一条消息在接受方当成两个消息处理了,这就是拆包。

Netty解决粘包与拆包

  • 固定长度的拆包器FixedLengthFrameDecoder:每个应用层数据包的都拆分成都是固定长度的大小,比如 1024字节。编码时会将不足指定长度的部分进行补全;拆包器会每次读取固定长度的消息,如果当前读取到的消息不足指定长度,那么就会等待下一个消息到达后进行补足。
  • 行拆包器LineBasedFrameDecoder。每个应用层数据包,都以换行符作为分隔符,进行分割拆分。数据在编码发送的时候,会以换行符作为一条完整的消息。
  • 分隔符拆包器 DelimiterBasedFrameDecoder。每个应用层数据包,都通过自定义的分隔符,进行分割拆分。这个版本,是LineBasedFrameDecoder 的通用版本,本质上是一样的。数据在编码发送的时候,会以一个自定义的分隔符作为一条完整的消息。
  • 基于数据包长度的拆包器LengthFieldBasedFrameDecoder: LengthFieldPrepender是编码器,在数据前面添加指定的字节数据,这个字节数据中保存了当前消息体的整体字节数据长度。LengthFieldBasedFrameDecoder将应用层数据包的长度,作为接收端应用层数据包的拆分依据。按照应用层数据包的大小,拆包。这个拆包器,有一个要求,就是应用层协议中包含数据包的长度。

HTTP为什么不会粘包与拆包

HTTP请求头或响应头中有Content-Length字段,表示消息长度,单位是字节,类似于Netty的LengthFieldBasedFrameDecoder。