目录
RESTful方式见 使用Nginx实现TDengine的负载均衡-基于RESTful
实验环境服务器：
172.16.216.2 test1 ##TDengine Server 1 & Arbitrator
172.16.216.3 test2 ##TDengine Server 2
172.16.216.4 test3 ##Nginx 服务器
**操作系统：**CentOS Linux release 7.9
软件版本：
nginx 1.20.0
TDengine 2.0.20.0
1.1.安装依赖包
yum install gcc gcc-c++ autoconf automake
yum install pcre-devel
yum install zlib-devel
1.2.安装Nginx
Naginx在1.9.0以后支持TCP负载均衡，如果要Naginx支持TCP/UDP，需要安装stream模块。
./configure --with-stream
make
make install
1.3.配置负载均衡
TDengine客户端与服务端通信端口为6030-6034，使用TCP和UDP协议，详见官方文档。
vi /usr/local/nginx/conf/nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
stream {
}
server {
listen 6030;
proxy_pass app6030;
}
upstream app6030 {
server 172.16.216.2:6030;
server 172.16.216.3:6030;
}
server {
listen 6031;
proxy_pass app6031;
}
upstream app6031 {
server 172.16.216.2:6031;
server 172.16.216.3:6031;
}
server {
listen 6032;
proxy_pass app6032;
}
upstream app6032 {
server 172.16.216.2:6032;
server 172.16.216.3:6032;
}
server {
listen 6033;
proxy_pass app6033;
}
upstream app6033 {
server 172.16.216.2:6033;
server 172.16.216.3:6033;
}
server {
listen 6034;
proxy_pass app6034;
}
upstream app6034 {
server 172.16.216.2:6034;
server 172.16.216.3:6034;
}
server {
listen 6030 udp;
proxy_pass uapp6030;
}
upstream uapp6030 {
server 172.16.216.2:6030;
server 172.16.216.3:6030;
}
server {
listen 6031 udp;
proxy_pass uapp6031;
}
upstream uapp6031 {
server 172.16.216.2:6031;
server 172.16.216.3:6031;
}
server {
listen 6032 udp;
proxy_pass uapp6032;
}
upstream uapp6032 {
server 172.16.216.2:6032;
server 172.16.216.3:6032;
}
server {
listen 6033 udp;
proxy_pass uapp6033;
}
upstream uapp6033 {
server 172.16.216.2:6033;
server 172.16.216.3:6033;
}
server {
listen 6034 udp;
proxy_pass uapp6034;
}
upstream uapp6034 {
server 172.16.216.2:6034;
server 172.16.216.3:6034;
}
}
1.4.启动Nginx
/usr/local/nginx/sbin/nginx
[root@test3 logs]# netstat -ant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:6031 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:6032 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:6033 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:6034 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:6041 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:6030 0.0.0.0:* LISTEN
tcp 0 0 172.16.216.4:22 172.16.216.1:62439 ESTABLISHED
tcp6 0 0 :::22 :::* LISTEN
tcp6 0 0 ::1:25 :::* LISTEN
具体安装步骤可见 TDengine学习笔记-集群安装
2.1. 安装TDengine
tar xvzf TDengine-server-2.0.20.0-Linux-x64.tar.gz
cd TDengine-server-2.0.20.0
./install
2.2. 安装Arbitrator
TDengine中的Arbitrator 类似于PostgreSQL中的witness服务，防止集群出现脑裂。因为实验只有2个节点，为防止停止一个节点后，无法通过无法，需要安装Arbitrator服务。
官方文档-Arbitrator的使用
tar xvzf TDengine-arbitrator-2.0.20.0-Linux-x64.tar.gz
cd TDengine-arbitrator-2.0.20.0
./install_arbi.sh
systemctl start tarbitratord
2.3. 配置文件taos.cfg
##test1
firstEp test1:6030
secondEp test2:6030
arbitrator test1:6042
fqdn test1
##test2
firstEp test1:6030
secondEp test2:6030
arbitrator test1:6042
fqdn test2
2.4.启动集群
systemctl start taosd
添加节点
taos> create dnode test2:6030;
taos> show dnodes;
id | end_point | vnodes | cores | status | role | create_time | offline reason |
======================================================================================================================================
1 | test1:6030 | 1 | 8 | ready | any | 2021-05-05 17:20:15.802 | |
2 | test2:6030 | 1 | 8 | ready | any | 2021-05-05 17:20:30.431 | |
0 | test1:6042 | 0 | 0 | ready | arb | 1970-01-01 08:00:00.000 | - |
Query OK, 3 row(s) in set (0.004701s)
2.5.安装客户端
在nginx服务器上安装TDengine客户端，模拟客户连接。
tar xzvf TDengine-enterprise-client-2.0.20.0-Linux-x64.tar.gz
cd TDengine-enterprise-client-2.0.20.0
./install_client.sh
配置客户端文件taos.cfg。注意：这里的firstEP要配置成本地。
/etc/taos/taos.cfg
firstEp test3:6030
fqdn test3
在test3上启动客户端连接TDengine集群。
[root@test3 ~]# taos
Welcome to the TDengine shell from Linux, Client Version:2.0.20.0
Copyright (c) 2020 by TAOS Data, Inc. All rights reserved.
taos> exit
[root@test3 ~]# taos -h 172.16.216.4
Welcome to the TDengine shell from Linux, Client Version:2.0.20.0
Copyright (c) 2020 by TAOS Data, Inc. All rights reserved.
taos> show dnodes;
id | end_point | vnodes | cores | status | role | create_time | offline reason |
======================================================================================================================================
1 | test1:6030 | 1 | 8 | ready | any | 2021-05-05 17:20:15.802 | |
2 | test2:6030 | 1 | 8 | ready | any | 2021-05-05 17:20:30.431 | |
0 | test1:6042 | 0 | 0 | ready | arb | 1970-01-01 08:00:00.000 | - |
Query OK, 3 row(s) in set (0.007414s)
停止节点test1上TDengine，在test3上重新连接集群。
[root@test1 ~]# systemctl stop taosd
[root@test1 ~]# ps -ef | grep taos
root 35454 12164 0 20:59 pts/0 00:00:00 grep --color=auto taos
[root@test3 ~]# taos
Welcome to the TDengine shell from Linux, Client Version:2.0.20.0
Copyright (c) 2020 by TAOS Data, Inc. All rights reserved.
taos> show dnodes;
id | end_point | vnodes | cores | status | role | create_time | offline reason |
======================================================================================================================================
1 | test1:6030 | 1 | 8 | offline | any | 2021-05-05 17:20:15.802 | status msg timeout |
2 | test2:6030 | 1 | 8 | ready | any | 2021-05-05 17:20:30.431 | |
0 | test1:6042 | 0 | 0 | ready | arb | 1970-01-01 08:00:00.000 | - |
Query OK, 3 row(s) in set (0.008756s)
Nginx日志显示172.16.216.2无法联通。
2021/05/05 21:00:00 [error] 9706#0: *9 recv() failed (111: Connection refused) while proxying and reading from upstream, udp client: 172.16.216.4, server: 0.0.0.0:6030, upstream: "172.16.216.2:6030", bytes from/to client:377/0, bytes from/to upstream:0/377
2021/05/05 21:00:02 [error] 9706#0: *9 sendmsg() failed (111: Connection refused) while proxying and sending to upstream, udp client: 172.16.216.4, server: 0.0.0.0:6030, upstream: "172.16.216.2:6030", bytes from/to client:1131/0, bytes from/to upstream:0/754
采用客户端taosc进行连接时，均能连接正常。但使用TDengine提供的测试工具taosdemo进行测试时，偶尔会报错，如下：
create database db01 success!
ERROR: startMultiThreadCreateChildTable() LN2743, Failed to connect to TDengine, reason:Unable to establish connection
Spent 4.0241 seconds to create 10 tables with 10 thread(s)
WARNING: offset and limit will not be used since the child tables are not exists!
ERROR: connect to server fail from insert sub thread, reason: Unable to establish connection
报错信息显示数据库创建成功，但无法创建表。
