zookeeper客户端源码调试debug zookeeper源码分析

前言：

无论是使用框架还是使用原生的Zookeeper.java来创建客户端连接，本质上都是使用Zookeeper来连接。在Zookeeper-3.4.13的源码中不但包含服务端的代码还提供了客户端的代码，我们先来分析下客户端的代码，了解下客户端是如何创建对服务端的连接，后续我们再来看下其是如何发送请求。

1.Zookeeper解析

public class ZooKeeper {
 
    // 只有两个属性，主要用的就是cnxn，后续会重点介绍
    protected final ClientCnxn cnxn;
    private static final Logger LOG;
    static {
        // 打印Env相关信息
        LOG = LoggerFactory.getLogger(ZooKeeper.class);
        Environment.logEnv("Client environment:", LOG);
    }
}

1.1 构造方法解析

public ZooKeeper(String connectString, int sessionTimeout, Watcher watcher,
                 boolean canBeReadOnly)
    throws IOException
    {
        LOG.info("Initiating client connection, connectString=" + connectString
                + " sessionTimeout=" + sessionTimeout + " watcher=" + watcher);

		// 如果在构造方法中传入watcher，则会将这个watcher作为默认watcher
        watchManager.defaultWatcher = watcher;

        ConnectStringParser connectStringParser = new ConnectStringParser(
                connectString);
        // 构造一个连接地址的解析器，最终会获取到一个具体的地址进行连接。在2中分析        
        HostProvider hostProvider = new StaticHostProvider(
                connectStringParser.getServerAddresses());
        // 创建ClientCnxn 对象，这个比较关键，在3中分析
        cnxn = new ClientCnxn(connectStringParser.getChrootPath(),
                hostProvider, sessionTimeout, this, watchManager,
                getClientCnxnSocket(), canBeReadOnly);
        cnxn.start();
    }

2.地址解析器StaticHostProvider

在生产环境，我们以集群的方式搭建Zookeeper服务端，故我们创建客户端时会将所有的服务端地址都写入，比如如下这样

ZooKeeper zkCli = new ZooKeeper("192.168.0.1:2181,192.168.0.2:2181,192.168.0.3:2181/zktest", 3000, new Watcher() {
    // 回调监听
    @Override
    public void process(WatchedEvent event) {
        try {
            List<String> children = zkCli.getChildren("/", true);
            for (String c : children) {
                // do...
            }
        } catch (KeeperException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
});

客户端在创建连接时会获取到某一个具体的服务端ip:port，创建对其长连接，这个选择的过程就是HostProvider来做的。我们来看下其具体选择过程

2.1 ConnectStringParser解析服务端ip信息

public final class ConnectStringParser {
    // 默认端口为2181
    private static final int DEFAULT_PORT = 2181;

    // 根目录，一般应用在使用时都会创建一个独特的根目录信息，比如dubbo的注册信息都以/dubbo为根目录
    private final String chrootPath;
    // 解析出来的具体server信息集合
    private final ArrayList<InetSocketAddress> serverAddresses = new ArrayList<InetSocketAddress>();
    
    public ConnectStringParser(String connectString) {
        // 我们以connectString="192.168.0.1:2181,192.168.0.2:2181,192.168.0.3:2181/zktest"为例
        int off = connectString.indexOf('/');
        if (off >= 0) {
            String chrootPath = connectString.substring(off);
            // ignore "/" chroot spec, same as null
            if (chrootPath.length() == 1) {
                this.chrootPath = null;
            } else {
                PathUtils.validatePath(chrootPath);
                // 这里会使用zktest作为根目录
                this.chrootPath = chrootPath;
            }
            // 192.168.0.1:2181,192.168.0.2:2181,192.168.0.3:2181被解析成connectString
            connectString = connectString.substring(0, off);
        } else {
            this.chrootPath = null;
        }

        String hostsList[] = connectString.split(",");
        for (String host : hostsList) {
            int port = DEFAULT_PORT;
            int pidx = host.lastIndexOf(':');
            if (pidx >= 0) {
                // otherwise : is at the end of the string, ignore
                if (pidx < host.length() - 1) {
                    port = Integer.parseInt(host.substring(pidx + 1));
                }
                host = host.substring(0, pidx);
            }
            // 将connectString解析成InetSocketAddress
            serverAddresses.add(InetSocketAddress.createUnresolved(host, port));
        }
    }

代码比较简单，主要就是解析chrootPath和服务端地址，封装成InetSocketAddress即可。

2.2 StaticHostProvider选择一个具体的服务端连接

public final class StaticHostProvider implements HostProvider {
 
    // 服务端地址集合
    private final List<InetSocketAddress> serverAddresses = new ArrayList<InetSocketAddress>(5);

    // 使用两个index来循环引用
    private int lastIndex = -1;
    private int currentIndex = -1;

    // 解析器
    private Resolver resolver;
    
    public StaticHostProvider(Collection<InetSocketAddress> serverAddresses) {
        this.resolver = new Resolver() {
            @Override
            public InetAddress[] getAllByName(String name) throws UnknownHostException {
                return InetAddress.getAllByName(name);
            }
        };
        // 初始化
        init(serverAddresses);
    }
	private void init(Collection<InetSocketAddress> serverAddresses) {
        if (serverAddresses.isEmpty()) {
            throw new IllegalArgumentException(
                    "A HostProvider may not be empty!");
        }

        // 初始化很简单，就是将在ConnectStringParser解析出来的服务端列表传入serverAddresses，并进行一次shuffle
        this.serverAddresses.addAll(serverAddresses);
        Collections.shuffle(this.serverAddresses);
    }
}

StaticHostProvider初始化的方法很简单，就是接收ConnectStringParser解析出来的服务端列表传入serverAddresses。重点方法在next()中，我们来分析下

2.3 StaticHostProvider.next() 获取特定的服务端地址

public InetSocketAddress next(long spinDelay) {
    // currentIndex递增下
    currentIndex = ++currentIndex % serverAddresses.size();
    // 如果达到了lastIndex，且需要sleep下，则先sleep下
    if (currentIndex == lastIndex && spinDelay > 0) {
        try {
            Thread.sleep(spinDelay);
        } catch (InterruptedException e) {
            LOG.warn("Unexpected exception", e);
        }
        // 初始值为-1，重置为0
    } else if (lastIndex == -1) {
        // We don't want to sleep on the first ever connect attempt.
        lastIndex = 0;
    }

    // 根据获取到的currentIndex，获取指定的InetSocketAddress，代码比较简单
    InetSocketAddress curAddr = serverAddresses.get(currentIndex);
    try {
        String curHostString = getHostString(curAddr);
        // 如果选定的服务端地址是一个域名提供的话，就获取下域名对应的ip列表
        List<InetAddress> resolvedAddresses = new ArrayList<InetAddress>(Arrays.asList(this.resolver.getAllByName(curHostString)));
        if (resolvedAddresses.isEmpty()) {
            return curAddr;
        }
        Collections.shuffle(resolvedAddresses);
        return new InetSocketAddress(resolvedAddresses.get(0), curAddr.getPort());
    } catch (UnknownHostException e) {
        return curAddr;
    }
}

代码很简单，笔者不再赘述。指定currentIndex和lastIndex两个指针，通过递增currentIndex，来获取serverAddresses具体index的InetAddress信息。

3. ClientCnxn的创建

我们直接来分析下其构造方法

public class ClientCnxn {
	public ClientCnxn(String chrootPath, HostProvider hostProvider, int sessionTimeout, ZooKeeper zooKeeper,
            ClientWatchManager watcher, ClientCnxnSocket clientCnxnSocket,
            long sessionId, byte[] sessionPasswd, boolean canBeReadOnly) {
        this.zooKeeper = zooKeeper;
        this.watcher = watcher;
        // 默认sessionId=0
        this.sessionId = sessionId;
        this.sessionPasswd = sessionPasswd;
        // 我们在构造Zookeeper时设置的session过期时间
        this.sessionTimeout = sessionTimeout;
        // 通过connectString解析出来的服务端地址和跟路径chrootpath
        this.hostProvider = hostProvider;
        this.chrootPath = chrootPath;

        // 这里的连接超时时间设置比较有意思，如果我们设置sessionTimeout为3000（也就是3秒钟），提供的服务端有三个ip，那么真正的connectTimeout=1s
        connectTimeout = sessionTimeout / hostProvider.size();
        readTimeout = sessionTimeout * 2 / 3;
        readOnly = canBeReadOnly;

        // 两个比较关键的Thread，后续继续分析
        sendThread = new SendThread(clientCnxnSocket);
        eventThread = new EventThread();

    }
}

3.1 ClientCnxn.start()

public class ClientCnxn {
	public void start() {
        sendThread.start();
        eventThread.start();
    }
}

ClientCnxn的start方法比较简单，就是启动两个线程。那么这两个线程具体是做什么的呢？

3.2 SendThread

/**
     * This class services the outgoing request queue and generates the heart
     * beats. It also spawns the ReadThread.
     */
// 从注释中可以看出，SendThread的主要作用就是发送请求到服务端；生成心跳
class SendThread extends ZooKeeperThread {
    private long lastPingSentNs;
    // 表示对服务端的一个长连接
    private final ClientCnxnSocket clientCnxnSocket;
    private Random r = new Random(System.nanoTime());        
    private boolean isFirstConnect = true;

    public void run() {
        ...
        while (state.isAlive()) {
            try {
                if (!clientCnxnSocket.isConnected()) {
                    ...
                    // 未连接的情况下，则调用hostProvider.next()方法选择一个ip:port进行连接    
                    if (rwServerAddress != null) {
                        serverAddress = rwServerAddress;
                        rwServerAddress = null;
                    } else {
                        serverAddress = hostProvider.next(1000);
                    }
                    // 在这里创建连接，创建连接过程并不复杂，就是NioSocketChannel的连接过程，读者可自行阅读
                    // 如果当前连接失败的话，在这个while循环中，会自动寻找下一个服务端ip:port进行连接
                    startConnect(serverAddress);
                    clientCnxnSocket.updateLastSendAndHeard();
                }

                ...

                if (state.isConnected()) {
                    //1000(1 second) is to prevent race condition missing to send the second ping
                    //also make sure not to send too many pings when readTimeout is small 
                    int timeToNextPing = readTimeout / 2 - clientCnxnSocket.getIdleSend() - 
                        ((clientCnxnSocket.getIdleSend() > 1000) ? 1000 : 0);
                    //send a ping request either time is due or no packet sent out within MAX_SEND_PING_INTERVAL
                    // 在适当的时候进行心跳发送
                    if (timeToNextPing <= 0 || clientCnxnSocket.getIdleSend() > MAX_SEND_PING_INTERVAL) {
                        sendPing();
                        clientCnxnSocket.updateLastSend();
                    } else {
                        if (timeToNextPing < to) {
                            to = timeToNextPing;
                        }
                    }
                }

                ...

                // 发送请求包到服务端    
                clientCnxnSocket.doTransport(to, pendingQueue, outgoingQueue, ClientCnxn.this);
            } catch (Throwable e) {
                ...
        }
        cleanup();
        clientCnxnSocket.close();
        if (state.isAlive()) {
            eventThread.queueEvent(new WatchedEvent(Event.EventType.None,
                                                    Event.KeeperState.Disconnected, null));
        }
        ZooTrace.logTraceMessage(LOG, ZooTrace.getTextTraceLevel(),
                                 "SendThread exited loop for session: 0x"
                                 + Long.toHexString(getSessionId()));
    }
}

SendThread的工作就是创建对服务端的连接，然后将请求包发送出去，至于具体如何发送请求包，我们后续在分析各种不同请求类型时在进行分析，目前只需要记住SendThread就是用来发送请求的即可。

还有就是：SendThread也可以读取请求响应结果，具体在readResponse()方法中

所以，总结下来，SendThread负责发送请求和接收响应结果。

3.3 EventThread

最后还有一个EventThread没有介绍，这个线程是做什么用的呢？主要是用来处理Event的，这个Event主要是Watcher Event，就是我们在Zookeeper上注册的事件监听器，如果被触发，则会接收到一个WatcherEvent事件，接收到之后，就是由EventThread来处理的。

关于其处理细节，本文不再详述，后面我们会专门写一篇文章来讲述事件监听处理。

总结：

本文主要介绍了Zookeeper客户端的创建过程，创建连接、发送请求、接收响应的工作主要交由SendThread来操作；而事件监听工作主要交由EventThread来处理。

后续我们来分析下各种请求类型下Zookeeper的处理过程。