Ollama docker安装 docker安装odoo

转载

技术领航探索者 2024-07-04 15:16:54

文章标签 Ollama docker安装 docker TCP 文章分类 Docker 云计算

一、Apache Druid 部署

1.1 单机版

1.1.1 Jar 包下载

从 https://imply.io/get-started 下载最新版本安装包

1.1.2 Druid 的安装部署

说明：imply 集成了Druid，提供了Druid 从部署到配置到各种可视化工具的完整的解决方案， imply 有点类似Cloudera Manager。

「1.解压」

tar -zxvf imply-2.7.10.tar.gz -C /opt/module

目录说明如下：

• bin/ - run scripts for included software.
• conf/ - template configurations for a clustered setup.
• conf-quickstart/* - configurations for the single-machine quickstart.
• dist/ - all included software.
• quickstart/ - files related to the single-machine quickstart.

「2.修改配置文件」

1)修改Druid 的ZK 配置

[chris@hadoop102 _common]$ pwd/opt/module/imply/conf/druid/_common[chris@hadoop102 _common]$ vi common.runtime.propertiesdruid.zk.service.host=hadoop102:2181,hadoop103:2181,hadoop104:2181

2)修改启动命令参数，使其不校验不启动内置ZK

[chris@hadoop102 supervise]$ pwd/opt/module/imply/conf/supervise:verify bin/verify-java#:verify bin/verify-default-ports#:verify bin/verify-version-check:kill-timeout 10#!p10 zk bin/run-zk conf-quickstart

「3.启动」

1)启动zookeeper

2)启动imply

[chris@hadoop102 imply]$ bin/supervise -cconf/supervise/quickstart.conf

说明：每启动一个服务均会打印出一条日志。可以通过/opt/module/imply-2.7.10/var/sv/查看服务启动时的日志信息

3)查看端口号9095 的启动情况

[chris@hadoop102 ~]$ netstat -anp | grep 9095tcp 0 0 :::9095 :::*LISTEN 3930/imply-ui-linuxtcp 00 ::ffff:192.168.1.102:9095 ::ffff:192.168.1.1:52567ESTABLISHED 3930/imply-ui-linuxtcp 00 ::ffff:192.168.1.102:9095 ::ffff:192.168.1.1:52568ESTABLISHED 3930/imply-ui-linux

「4.登录hadoop102:9095 查看」

Ollama docker安装 docker安装odoo_Ollama docker安装

「5.停止服务」

按Ctrl + c 中断监督进程，如果想中断服务后进行干净的启动，请删除/opt/module/imply-2.7.10/var/目录。

1.2 Docker 容器版

1.2.1 下载

# 搜索 Docker Hub$ docker search druid# 下载最新版本的镜像$ docker pull apache/druid:0.19.0# 检查镜像是否下载成功$ docker image list

1.2.2 配置 Docker 文件共享

打开配置面板，进入 File Sharing 配置页面，增加 ${the path of your source code}/distribution/docker/storage 路径，随后点击 Apply & Restart 按钮，应用并重启

1.2.3 启动

$ git clone https://github.com/apache/druid.git$ cd druid$ docker-compose -f distribution/docker/docker-compose.yml up# 同理，也可以使用 start/stop 命令启停容器$ docker-compose -f distribution/docker/docker-compose.yml stop$ docker-compose -f distribution/docker/docker-compose.yml start# 或者使用 down 命令移除容器$ docker-compose -f distribution/docker/docker-compose.yml down

1.2.4 校验

Historical 容器

$ docker exec -it historical sh$ ls /opt/data/  indexing-logs  segments$ ls /opt/data/segments/  intermediate_pushes  wikipedia$ ls /opt/data/segments/wikipedia/  2016-06-27T00:00:00.000Z_2016-06-28T00:00:00.000Z$ ls /opt/data/segments/wikipedia/2016-06-27T00\:00\:00.000Z_2016-06-28T00\:00\:00.000Z/  2020-06-04T07:11:42.714Z$ ls /opt/data/segments/wikipedia/2016-06-27T00\:00\:00.000Z_2016-06-28T00\:00\:00.000Z/2020-06-04T07\:11\:42.714Z/0/  index.zip$ ls -lh

total 8M-rw-r--r--    1 druid    druid       5.9M Jun  4 07:49 00000.smoosh-rw-r--r--    1 druid    druid         29 Jun  4 07:49 factory.json-rw-r--r--    1 druid    druid       1.7M Jun  4 07:14 index.zip-rw-r--r--    1 druid    druid        707 Jun  4 07:49 meta.smoosh-rw-r--r--    1 druid    druid          4 Jun  4 07:49 version.bin

$ cat factory.json

{"type":"mMapSegmentFactory"}

$ xxd version.bin

00000000: 0000 0009                                ....

$ cat meta.smoosh

v1,2147483647,1__time,0,0,1106channel,0,145739,153122cityName,0,153122,195592comment,0,195592,1598156count,0,1106,2063countryIsoCode,0,1598156,1614170countryName,0,1614170,1630859diffUrl,0,1630859,4224103flags,0,4224103,4252873index.drd,0,6162513,6163275isAnonymous,0,4252873,4262876isMinor,0,4262876,4282592isNew,0,4282592,4290896isRobot,0,4290896,4298796isUnpatrolled,0,4298796,4307345metadata.drd,0,6163275,6163925namespace,0,4307345,4342089page,0,4342089,5710071regionIsoCode,0,5710071,5730339regionName,0,5730339,5759351sum_added,0,2063,37356sum_commentLength,0,37356,66244sum_deleted,0,66244,81170sum_delta,0,81170,126275sum_deltaBucket,0,126275,145739user,0,5759351,6162513

其中，index.drd 包含该 Segment 覆盖的时间范围、指定的 Bitmap 种类(concise / roaring)，以及包含的列和维度；而 metadata.drd 包含是否 Rollup、哪些聚合函数、查询的粒度，时间戳字段信息，以及可用于存储任意 Key-Value 数据的 Map 结构(例如 Kafka Firehose 用来存储 offset 信息)。更多细节详见 org.apache.druid.segment.IndexIO.V9IndexLoader#load

PostgreSQL 容器

$ docker exec -it postgres bash$ psql -U druid -d druid$ \l

List of databases   Name    | Owner | Encoding |  Collate   |   Ctype    | Access privileges-----------+-------+----------+------------+------------+------------------- druid     | druid | UTF8     | en_US.utf8 | en_US.utf8 | postgres  | druid | UTF8     | en_US.utf8 | en_US.utf8 | template0 | druid | UTF8     | en_US.utf8 | en_US.utf8 | =c/druid         +           |       |          |            |            | druid=CTc/druid template1 | druid | UTF8     | en_US.utf8 | en_US.utf8 | =c/druid         +           |       |          |            |            | druid=CTc/druid(4 rows)

$ \c druid

You are now connected to database "druid" as user "druid".

$ \dt

List of relations Schema |         Name          | Type  | Owner--------+-----------------------+-------+------- public | druid_audit           | table | druid public | druid_config          | table | druid public | druid_datasource      | table | druid public | druid_pendingsegments | table | druid public | druid_rules           | table | druid public | druid_segments        | table | druid public | druid_supervisors     | table | druid public | druid_tasklocks       | table | druid public | druid_tasklogs        | table | druid public | druid_tasks           | table | druid(10 rows)

> select id, datasource, created_date, start, "end", partitioned, version, used from public.druid_segments;

wikipedia_2016-06-27T00:00:00.000Z_2016-06-28T00:00:00.000Z_2020-06-04T07:11:42.714Z | wikipedia  | 2020-06-04T07:14:50.619Z | 2016-06-27T00:00:00.000Z | 2016-06-28T00:00:00.000Z | t           | 2020-06-04T07:11:42.714Z | t

1.3 Kubernetes 集群版

1.3.1 安装

$ helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com$ helm install incubator/druid --version 0.2.6 --generate-name

NAME: druid-1592218780LAST DEPLOYED: Mon Jun 15 23:59:42 2020NAMESPACE: defaultSTATUS: deployedREVISION: 1TEST SUITE: NoneNOTES:1. Get the router URL by running these commands:  export POD_NAME=$(kubectl get pods --namespace default -l "app=druid,release=druid-1592218780" -o jsonpath="{.items[0].metadata.name}")  echo "Visit http://127.0.0.1:8080 to use your application"  kubectl port-forward $POD_NAME 8080:8888

$ export POD_NAME=$(kubectl get pods --namespace default -l "app=druid,release=`helm list | grep druid- | awk '{print $1}'`" | grep router | awk '{print $1}')$ nohup kubectl port-forward $POD_NAME 8888:8888 --address 0.0.0.0 2>&1 &

1.3.2 校验

$ kubectl get all

NAME                                                READY   STATUS    RESTARTS   AGEpod/druid-1592364086-broker-76bf68c8bc-96d56        1/1     Running   0          2m36spod/druid-1592364086-coordinator-5f645bd5c8-rhhpz   1/1     Running   0          2m36spod/druid-1592364086-historical-0                   1/1     Running   0          2m36spod/druid-1592364086-middle-manager-0               1/1     Running   0          2m36spod/druid-1592364086-postgresql-0                   1/1     Running   0          2m36spod/druid-1592364086-router-67f678b6c5-mw6b4        1/1     Running   0          2m36spod/druid-1592364086-zookeeper-0                    1/1     Running   0          2m36spod/druid-1592364086-zookeeper-1                    1/1     Running   0          2m8spod/druid-1592364086-zookeeper-2                    1/1     Running   0          85spod/local-volume-provisioner-8sjtx                  1/1     Running   0          8m59spod/local-volume-provisioner-9z7mh                  1/1     Running   0          8m59spod/local-volume-provisioner-m2xrt                  1/1     Running   0          8m59spod/local-volume-provisioner-ptbqs                  1/1     Running   0          8m59spod/local-volume-provisioner-tw2fn                  1/1     Running   0          8m59sNAME                                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGEservice/druid-1592364086-broker                ClusterIP   10.10.10.128             8082/TCP                     2m36sservice/druid-1592364086-coordinator           ClusterIP   10.10.10.195             8081/TCP                     2m36sservice/druid-1592364086-historical            ClusterIP   10.10.10.226             8083/TCP                     2m36sservice/druid-1592364086-middle-manager        ClusterIP   10.10.10.108             8091/TCP                     2m36sservice/druid-1592364086-postgresql            ClusterIP   10.10.10.155             5432/TCP                     2m36sservice/druid-1592364086-postgresql-headless   ClusterIP   None                     5432/TCP                     2m36sservice/druid-1592364086-router                ClusterIP   10.10.10.29              8888/TCP                     2m36sservice/druid-1592364086-zookeeper             ClusterIP   10.10.10.122             2181/TCP                     2m36sservice/druid-1592364086-zookeeper-headless    ClusterIP   None                     2181/TCP,3888/TCP,2888/TCP   2m36sservice/kubernetes                             ClusterIP   10.10.10.1               443/TCP                      30mNAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGEdaemonset.apps/local-volume-provisioner   5         5         5       5            5                     9mNAME                                           READY   UP-TO-DATE   AVAILABLE   AGEdeployment.apps/druid-1592364086-broker        1/1     1            1           2m36sdeployment.apps/druid-1592364086-coordinator   1/1     1            1           2m36sdeployment.apps/druid-1592364086-router        1/1     1            1           2m36sNAME                                                      DESIRED   CURRENT   READY   AGEreplicaset.apps/druid-1592364086-broker-76bf68c8bc        1         1         1       2m36sreplicaset.apps/druid-1592364086-coordinator-5f645bd5c8   1         1         1       2m36sreplicaset.apps/druid-1592364086-router-67f678b6c5        1         1         1       2m36sNAME                                               READY   AGEstatefulset.apps/druid-1592364086-historical       1/1     2m36sstatefulset.apps/druid-1592364086-middle-manager   1/1     2m36sstatefulset.apps/druid-1592364086-postgresql       1/1     2m36sstatefulset.apps/druid-1592364086-zookeeper        3/3     2m36s

上述 CLUSTER-IP 信息已脱敏

1.3.3 ZooKeeper 元数据

$ zkCli.sh[zk: localhost:2181(CONNECTED) 0] ls -R /druid

/druid/druid/announcements/druid/coordinator/druid/discovery/druid/indexer/druid/internal-discovery/druid/loadQueue/druid/overlord/druid/segments/druid/servedSegments/druid/announcements/10.10.10.63:8083/druid/coordinator/_COORDINATOR/druid/coordinator/_COORDINATOR/_c_281fb87b-c40c-4d71-a657-8254cbcf3730-latch-0000000000/druid/discovery/druid:broker/druid/discovery/druid:coordinator/druid/discovery/druid:overlord/druid/discovery/druid:router/druid/discovery/druid:broker/0e76bfc1-87f8-4799-9c36-0fb0e5617aef/druid/discovery/druid:coordinator/035b1ada-531e-4a71-865b-7a1a6d6f1734/druid/discovery/druid:overlord/a74523d6-1708-45b3-9c0b-87f438cda4e3/druid/discovery/druid:router/c0bb18d3-51b1-4089-932b-a5d6e05ab91c/druid/indexer/announcements/druid/indexer/status/druid/indexer/tasks/druid/indexer/announcements/10.10.10.65:8091/druid/indexer/status/10.10.10.65:8091/druid/indexer/tasks/10.10.10.65:8091/druid/internal-discovery/BROKER/druid/internal-discovery/COORDINATOR/druid/internal-discovery/HISTORICAL/druid/internal-discovery/INDEXER/druid/internal-discovery/MIDDLE_MANAGER/druid/internal-discovery/OVERLORD/druid/internal-discovery/PEON/druid/internal-discovery/ROUTER/druid/internal-discovery/BROKER/10.10.10.73:8082/druid/internal-discovery/COORDINATOR/10.10.10.72:8081/druid/internal-discovery/HISTORICAL/10.10.10.63:8083/druid/internal-discovery/MIDDLE_MANAGER/10.10.10.65:8091/druid/internal-discovery/OVERLORD/10.10.10.72:8081/druid/internal-discovery/ROUTER/10.10.10.55:8888/druid/loadQueue/10.10.10.63:8083/druid/overlord/_OVERLORD/druid/overlord/_OVERLORD/_c_ecacbc56-4d36-4ca0-ac1d-0df919c40bff-latch-0000000000/druid/segments/10.10.10.63:8083/druid/segments/10.10.10.63:8083/10.10.10.63:8083_historical__default_tier_2020-06-20T04:08:23.309Z_1b957acb6850491ca6ea885fca1b3c210/druid/segments/10.10.10.63:8083/10.10.10.63:8083_historical__default_tier_2020-06-20T04:10:16.643Z_57c1f60104a94c459bf0331eb3c1f0a01/druid/servedSegments/10.10.10.63:8083

上述 IP 地址相关信息已脱敏

1.3.4 Broker 健康检查

$ kill `ps -ef | grep 8082 | grep -v grep | awk '{print $2}'`; export POD_NAME=$(kubectl get pods --namespace default -l "app=druid,release=`helm list | grep druid- | awk '{print $1}'`" | grep broker | awk '{print $1}') ; nohup kubectl port-forward $POD_NAME 8082:8082 --address 0.0.0.0 2>&1 &$ curl localhost:8082/status/health

true

1.3.5 Historical 缓存

$ cd /opt/druid/var/druid/segment-cache/info_dir$ cat wikipedia_2016-06-27T00:00:00.000Z_2016-06-27T01:00:00.000Z_2020-06-20T04:10:01.833Z

{  "dataSource": "wikipedia",  "interval": "2016-06-27T00:00:00.000Z/2016-06-27T01:00:00.000Z",  "version": "2020-06-20T04:10:01.833Z",  "loadSpec": {    "type": "hdfs",    "path": "hdfs://10.10.10.44:8020/druid/segments/wikipedia/20160627T000000.000Z_20160627T010000.000Z/2020-06-20T04_10_01.833Z/0_index.zip"  },  "dimensions": "channel,cityName,comment,countryIsoCode,countryName,diffUrl,flags,isAnonymous,isMinor,isNew,isRobot,isUnpatrolled,namespace,page,regionIsoCode,regionName,user",  "metrics": "count,sum_added,sum_commentLength,sum_deleted,sum_delta,sum_deltaBucket",  "shardSpec": {    "type": "numbered",    "partitionNum": 0,    "partitions": 0  },  "binaryVersion": 9,  "size": 241189,  "identifier": "wikipedia_2016-06-27T00:00:00.000Z_2016-06-27T01:00:00.000Z_2020-06-20T04:10:01.833Z"}

$ cd /opt/druid/var/druid/segment-cache/wikipedia/2016-06-27T00:00:00.000Z_2016-06-27T01:00:00.000Z/2020-06-20T04:10:01.833Z/0$ ls

00000.smoosh  factory.json  meta.smoosh   version.bin

1.3.6 Segment 文件

$ kubectl exec -it hdfs-1593317115-namenode-0 bash$ hdfs dfs -get /druid/segments/wikipedia/20160627T000000.000Z_20160627T010000.000Z/2020-06-28T04_10_01.833Z/0_index.zip /tmp/index/$ exit$ kubectl cp hdfs-1593317115-namenode-0:/tmp/index/0_index.zip /tmp/0_index.zip$ unzip 0_index.zip$ ls

00000.smoosh  0_index.zip  factory.json  meta.smoosh  version.bin

1.3.7 Coordinator 动态配置

$ kill `ps -ef | grep 8081 | grep -v grep | awk '{print $2}'`; export POD_NAME=$(kubectl get pods --namespace default -l "app=druid,release=`helm list | grep druid- | awk '{print $1}'`" | grep coordinator | awk '{print $1}') ; nohup kubectl port-forward $POD_NAME 8081:8081 --address 0.0.0.0 2>&1 &$ curl localhost:8081/druid/coordinator/v1/config | python -m json.tool

{  "balancerComputeThreads": 1,  "decommissioningMaxPercentOfMaxSegmentsToMove": 70,  "decommissioningNodes": [],  "emitBalancingStats": false,  "killAllDataSources": false,  "killDataSourceWhitelist": [],  "killPendingSegmentsSkipList": [],  "maxSegmentsInNodeLoadingQueue": 0,  "maxSegmentsToMove": 5,  "mergeBytesLimit": 524288000,  "mergeSegmentsLimit": 100,  "millisToWaitBeforeDeleting": 900000,  "pauseCoordination": false,  "replicantLifetime": 15,  "replicationThrottleLimit": 10}

# 将 maxSegmentsToMove 调整为 50$ curl -XPOST -H 'Content-Type:application/json' localhost:8081/druid/coordinator/v1/config -d '{"maxSegmentsToMove":50}'$ curl localhost:8081/druid/coordinator/v1/config | python -m json.tool

{  "balancerComputeThreads": 1,  "decommissioningMaxPercentOfMaxSegmentsToMove": 70,  "decommissioningNodes": [],  "emitBalancingStats": false,  "killAllDataSources": false,  "killDataSourceWhitelist": [],  "killPendingSegmentsSkipList": [],  "maxSegmentsInNodeLoadingQueue": 0,  "maxSegmentsToMove": 50,  "mergeBytesLimit": 524288000,  "mergeSegmentsLimit": 100,  "millisToWaitBeforeDeleting": 900000,  "pauseCoordination": false,  "replicantLifetime": 15,  "replicationThrottleLimit": 10}

动态配置项 maxSegmentsToMove 可以用于控制同时被 rebalance 的 Segment 数量

1.3.8 Druid SQL 查询

# 映射 Broker 容器的 8082 端口$ kill `ps -ef | grep 8082 | grep -v grep | awk '{print $2}'`; export POD_NAME=$(kubectl get pods --namespace default -l "app=druid,release=`helm list | grep druid- | awk '{print $1}'`" | grep broker | awk '{print $1}') ; nohup kubectl port-forward $POD_NAME 8082:8082 --address 0.0.0.0 2>&1 &$ echo '{"query":"SELECT COUNT(*) as res FROM wikipedia"}' > druid_query.sql$ curl -XPOST -H'Content-Type: application/json' http://localhost:8082/druid/v2/sql/ -d@druid_query.sql

[{"res":24433}]

二、配置

2.1 常用端口

Ollama docker安装 docker安装odoo_docker_02

生产中，建议将 ZooKeeper 和 Metadata Stroage 部署在独立的物理机上，而不是混合部署在 Coordinator 节点上

2.2 rollup

在 Apache Druid 0.9.2 版本之后，我们可以通过在 granularitySpec 中配置 "rollup": false，来完全关闭 RollUp 特性，即在数据摄入的过程中，不做任何的预聚合，只保留最原始的数据点。即便是同一时刻的、具有相同维度的、完全相同的多个数据点，都会全部存储下来，不会发生覆盖写

2.3 selectStrategy

该参数默认为 fillCapacity，意味着分配 Task 的时候，会将某个 MiddleManager 分配满，才会分配新的 Task 到其他 MiddleManager 上。这里可以考虑使用 equalDistribution 策略，将 Task 均匀分配到 MiddleManager 上

$ cd $DRUID_HOME$ vim conf/druid/overlord/runtime.properties

druid.indexer.selectStrategy=equalDistribution

在 0.11.0 版本中，默认策略已经改成了 equalDistribution。详见 WorkerBehaviorConfig#DEFAULT_STRATEGY

2.4 maxRowsPerSegment

该参数用于控制每个 Segment 中最大能够存储的记录行数(默认值为 500,0000)，只有二级分区采用的是 dynamic 才会生效。如果 spec.tuningConfig.type 设置的是 hashed，则需要指定 shard 的数量，以及哪些 Dimension 被用于 hash 计算(默认为所有 Dimension)。另外，在 dynamic 类型的二级分区中，还有一个 maxTotalRows 参数(默认值为 2000,0000)，用来控制所有尚未被存储到 Deep Storage 中的 segment 的记录行数，一旦达到 maxTotalRows 则会立即触发 push 操作

如果该参数设置得很低，会产生很多小的 Segment 文件。一方面，如果 DeepStorage 为 HDFS 的话，会触发小文件问题，影响到集群性能(访问大量小文件不同于访问少数大文件，需要不断地在 DataNode 之间跳转，大部分时间都会耗费在 task 的启动和释放上，并且 NameNode 要监控的数据块变多后，网络带宽和内存占用也会比高，还会拖慢 NameNode 节点的故障恢复)；另一方面，如果操作系统中 vm.max_map_count 参数为默认的 65530 的话，可能会达到这个阈值，使得 MMap 操作失败，进而导致 Historical 进程 crash 退出，如果 Segment 一直无法完成 handoff 的过程，则会促使 Coordinator 进程 kill 实时任务

2.5 druid.server.tier

该参数相当于给 Historical 节点加了一个标签，可以将相同 Tier 名称的 Historical 进行分组，便于实现冷热分层。在 historical/runtime.properties 配置文件中设置，默认值为 _default_tier。例如，可以创建 hot、cold 两个 Tier，hot Tier 用于存储最近三个月的数据，cold Tier 用于存储最近一年的数据。如此一来，因为 cold 分组下的 Historical 节点存储的数据只需要应对一些低频查询，便可以使用非 SSD 的磁盘，以节约硬件成本