rocketmq nameserver 账号密码认证 rocketmq connect

转载

mob6454cc7225b4 2024-06-20 10:06:48

文章标签 java 中间件分布式 bc json 文章分类 架构后端开发

首先声明，这里是RocketMQ的单机环境搭建，集群搭建请参考集群搭建

首先，从github clone代码

git clone https://github.com/apache/rocketmq.git

构建项目

cd rocketmq
# meavn打包
mvn -Prelease-all -DskipTests clean install -U
# 这里的4.5.x是版本号，可能不同，请注意自己的版本
cd distribution/target/rocketmq-4.7.0/rocketmq-4.7.0/bin

先启动NameServer，再启动Broker

# 启动nameserver
nohup sh bin/mqnamesrv &
# 启动broker
nohup sh bin/mqbroker -n localhost:9876 &
# 导入nameserver环境变量
export NAMESRV_ADDR=localhost:9876

注意：

这里我建议开启自动创建topic的选项 nohup sh mqbroker -n localhost:9876 autoCreateTopicEnable=true &否则，每次都需要在代码或者mqadmin中手动添加topic，比较麻烦
有时使用mqadmin时会报错， connect to failed，这是因为没有引入nameserver环境变量，据我使用过程发现，有时你引入了，一段时间后还是会报异常，所以最好还是将其写入系统文件，当人nameserver地址有变化的话就另说了

启动broker后，查看broker日志，看是否启动成功

tail -f ~/logs/rocketmqlogs/broker.log

如果看到如下内容，说明启动成功

The broker[%s, 172.30.30.233:10911] boot success
register broker[0]to name server localhost:9876 OK

RocketMQ Runtime环境搭建

接下来是RocketMQ Connect Runtime，首先同样是项目下载和构建

git clone https://github.com/apache/rocketmq-externals

cd rocketmq-externals/rocketmq-connect

mvn clean install -Dmaven.test.skip=true

项目配置：修改配置文件connect.conf

cd rocketmq-connect/rocketmq-connect-runtime/target/distribution/conf
vim  connect.conf

# REST API的端口地址
httpPort=8081

# 本地配置存储路径
storePathRootDir=~/storeRoot

# Rocketmq namesrvAddr
namesrvAddr=127.0.0.1:9876  

# connector-plugins文件夹所在的位置
# Source or sink connector jar file dir
pluginPaths=/usr/local/connector-plugins/

然后，由于RocketMQ Connect Runtime需要4个topic和RocketMQ传递消息，所以需要先在RocketMQ中创建topic，此时在RocketMQ目录下，并保证RocketMQ的broker和nameserver是开启状态

# 创建集群发现的Topic
$ sh mqadmin updateTopic -b 4xx.1xx.2xx.1xx:10911 -t connector-cluster-topic
....
create topic to 4xx.1xx.2xx.1xx:10911 success.
TopicConfig [topicName=connector-cluster-topic, readQueueNums=8, writeQueueNums=8, perm=RW-, topicFilterType=SINGLE_TAG, topicSysFlag=0, order=false] 

# 创建connector设置信息Topic
$ sh mqadmin updateTopic -b 47.106.2xx.1xx:10911 -t connector-config-topic
....
create topic to 4xx.1xx.2xx.1xx:10911 success.
TopicConfig [topicName=connector-config-topic, readQueueNums=8, writeQueueNums=8, perm=RW-, topicFilterType=SINGLE_TAG, topicSysFlag=0, order=false]

# 创建Source Connector消费Topic
$ sh mqadmin updateTopic -b 4xx.1xx.2xx.1xx:10911 -t connector-position-topic
....
create topic to 4xx.1xx.2xx.1xx:10911 success.
TopicConfig [topicName=connector-position-topic, readQueueNums=8, writeQueueNums=8, perm=RW-, topicFilterType=SINGLE_TAG, topicSysFlag=0, order=false]

# 创建Sink Connector消费Topic
$ sh mqadmin updateTopic -b 4xx.1xx.2xx.1xx:10911 -t connector-offset-topic
....
create topic to 4xx.1xx.2xx.1xx:10911 success.
TopicConfig [topicName=connector-offset-topic, readQueueNums=8, writeQueueNums=8, perm=RW-, topicFilterType=SINGLE_TAG, topicSysFlag=0, order=false]

**注意：**如果之前开启自动创建topic时，这里要注意，由于connector-cluster-topic充当consumer中集群发现的功能，所以我的理解是不会往topic中发送消息，所以不会自动该topic，而是要手动添加connector-cluster-topic，否则会无法发现集群中的worker。

项目运行：运行脚本run_worker

# 在rocketmq-connect-runtime目录下运行脚本
$ sh ./run_worker.sh

run rumtime worker
2019-10-11 10:46:57 INFO main - Logging initialized @1207ms to org.eclipse.jetty.util.log.Slf4jLog
2019-10-11 10:46:57 INFO main - 
 _________________________________________
|        _                  _ _           |
|       | | __ ___   ____ _| (_)_ __      |
|    _  | |/ _` \ \ / / _` | | | '_ \     |
|   | |_| | (_| |\ V / (_| | | | | | |    |
|    \___/ \__,_| \_/ \__,_|_|_|_| |_|    |
|_________________________________________|
|                                         |
|    https://javalin.io/documentation     |
|_________________________________________|
...

The worker [10.134.142.157@43326] boot success.

看到上述结果表示worker启动成功

测试 File-Connector Demo

首先还是项目构建，这里由于和rocketmq-connect同属于一个目录，所以就不用构建了

将jar包拷贝到connector插件目录下

# 进入到rocketmq-connect-sample所在的目录
cd {rocketmq-external目录}/rocketmq-connect/rocketmq-connect-sample/
# 进入jar包所在的目录
cd target
# 将jar包拷贝到connector插件目录下
cp rocketmq-connect-sample-0.0.1-SNAPSHOT.jar /usr/local/connector-plugins/

此时保证启动RocketMQ、RocketMQ Connect Runtime是运行状态

创建Topic

# 创建Topic
# 注意这里的ip地址要换成自己服务器ip
$ sh mqadmin updateTopic -b 4xx.1xx.2xx.2xx:10911 -t fileTopic
....
create topic to 4xx.1xx.2xx.2xx:10911 success.

根据RESTful接口和官方文档启动SourceConnector和SinkConnector 官方文档

这里也可以使用我的一键quick-start脚本

#!/bin/bash
echo "run file-connector"
source_location=$1
target_location=$2
basic_url="http://localhost:8081/connectors/fileConnectorSink?config="
source_json=\{\"connector-class\":\"org.apache.rocketmq.connect.file.FileSourceConnector\",\"topic\":\"fileTopic\",\"filename\":\"$source_location\",\"source-record-converter\":\"org.apache.rocketmq.connect.runtime.converter.JsonConverter\"\}
target_json=\{\"connector-class\":\"org.apache.rocketmq.connect.file.FileSinkConnector\",\"topicNames\":\"fileTopic\",\"filename\":\"$target_location\",\"source-record-converter\":\"org.apache.rocketmq.connect.runtime.converter.JsonConverter\"}
echo $source_json
echo $target_json
urlencode() {
      python -c 'import urllib, sys; print urllib.quote(sys.argv[1], sys.argv[2])' \
              "$1" "$urlencode_safe"
}
target_json=$(urlencode "$target_json")
source_json=$(urlencode "$source_json")
source_connector=$basic_url$source_json
sink_connector=$basic_url$target_json
source_result=$(curl "$source_connector")
if [ "$source_result" == "success" ]
then
    target_result=$(curl "$sink_connector")
    if [$target_result -eq "success"]
    then
        echo success
    else
        echo target-file path is not exist!
    fi
else
    echo source-file path is not exist!
fi

由于curl命令无法解析json字符串，所以这里我将config的json用urlencode编码

运行时，只需要在运行时提供source_location和target_location，file-connector就会自动把文件发送到RocketMQ，然后从RocketMQ中消费到目的地址，实现了文件传输。

Connector-jdbc 使用说明

首先还是clone项目，打包

mvn clean install -Dmaven.test.skip=true
# 进入jar包所在的目录
cd target
# 拷贝构建好的jar包
cp rocketmq-connect-jdbc-0.0.1-SNAPSHOT-jar-with-dependencies.jar /usr/local/connector-plugins/

在RocketMQ中创建topic

# 创建Topic
# 注意这里的ip地址要换成自己服务器ip
$ sh mqadmin updateTopic -b 4xx.1xx.2xx.2xx:10911 -t jdbcTopic

然后就是通过RESTful接口启动connector

# source jdbc-connector
http://xxx.xxx.xxx.xxx:8081/connectors/jdbcConnectorSource?config=
{
    "connector-class":"org.apache.rocketmq.connect.jdbc.connector.JdbcSourceConnector",
    "rocketmqTopic":"jdbcTopic",
    "topicNames": "jdbcTopic",
    "dbUrl":"127.0.0.1",
    "dbPort":"3306",
    "dbUsername":"root",
    "dbPassword":"123456",
    "whiteDataBase": {
        "test":{
           "jdbcTopic": {"NO-FILTER": "10"}
         }
      },
    "mode": "bulk",
    "task-parallelism": 1,
    "source-cluster": "172.18.0.1:10911",
    "source-rocketmq": "127.0.0.1:9876",
    "source-record-converter":"org.apache.rocketmq.connect.runtime.converter.JsonConverter"
}

# sink jdbc-connector
http://xxx.xxx.xxx.xxx:8081/connectors/jdbcConnectorSink?config=
{
    "connector-class":"org.apache.rocketmq.connect.jdbc.connector.JdbcSinkConnector",
    "topicNames":"jdbcTopic",
    "rocketmqTopic":"jdbcTopic",
    "dbUrl":"127.0.0.1",
    "dbPort":"3306",
    "dbUsername":"root",
    "dbPassword":"123456",
    "source-rocketmq": "127.0.0.1:9876",
    "source-cluster": "172.18.0.1:10911",
    "mode": "bulk",
    "source-record-converter":"org.apache.rocketmq.connect.runtime.converter.JsonConverter"
}

注意：

保证rocketmqTopic，topicNames和WhiteDataBase中的表名一样（表名为什么要和topic一样，这里我也不太清楚，应该是一个坑），这里我都是jdbcTopic
"whiteDataBase"是一个json，里面表示需要存入RocketMQ的数据，可以自己定义，嵌套配置，为{DB名：{表名：{字段名：字段值}}}，若无指定字段数据同步，字段名可设为NO-FILTER，值为任意，例如{“DATABASE_TEST”:{“TEST_DATA”:{“name”:“test”}}}

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。