consul简介

Consul是一个服务网格(微服务间的 TCP/IP,负责服务之间的网络调用、限流、熔断和监控)解决方案,它是一个一个分布式的,高度可用的系统,而且开发使用都很简便。它提供了一个功能齐全的控制平面,主要特点是:服务发现、健康检查、键值存储、安全服务通信、多数据中心。

安装准备

  • 树莓派k8s集群
root@pi4-master01:~# kubectl get nodes -o wide
NAME           STATUS   ROLES    AGE     VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
pi4-master01   Ready    master   4d18h   v1.15.10   192.168.5.18   <none>        Ubuntu 20.04 LTS   5.4.0-1011-raspi   docker://18.9.9
pi4-node01     Ready    node     4d17h   v1.15.10   192.168.5.19   <none>        Ubuntu 20.04 LTS   5.4.0-1011-raspi   docker://18.9.9
pi4-node02     Ready    node     4d17h   v1.15.10   192.168.5.20   <none>        Ubuntu 20.04 LTS   5.4.0-1011-raspi   docker://18.9.9
  • 树莓派k8s集群已安装helm
root@pi4-master01:~/k8s/cluster-monitoring-0.37.0# helm version
Client: &version.Version{SemVer:"v2.15.0", GitCommit:"c2440264ca6c078a06e088a838b0476d2fc14750", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.15.0+unreleased", GitCommit:"9668ad4d90c5e95bd520e58e7387607be6b63bb6", GitTreeState:"dirty"}
  • 树莓派k8s集群已安装存储类,并设置为默认存储
root@pi4-master01:~/k8s/cluster-monitoring-0.37.0# kubectl get storageclass
NAME                   PROVISIONER             AGE
local-path (default)   rancher.io/local-path   4d15h

对了,如果你还没有树莓派k8s集群,这里有一篇树莓派 k8s 集群入坑指南可以参考,欢迎入坑。

helm安装consul集群

这里我们选择用helm安装,helm有一个默认仓库stable,指向https://kubernetes-charts.storage.googleapis.com, 国内访问比较困难,所以网上很多安装helm的文档都以https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts 来代替。但最近发现,这个仓库也不再更新了,很多chart的最新版本都没有,这里提供一个新地址http://mirror.azure.cn/kubernetes/charts 。经简单验证,发现这个仓库至少比阿里的那个仓库新,建议切换。

root@pi4-master01:~# helm repo remove stable
"stable" has been removed from your repositories
root@pi4-master01:~# helm repo add stable http://mirror.azure.cn/kubernetes/charts/
"stable" has been added to your repositories
root@pi4-master01:~# helm repo list
NAME   	URL
local  	http://127.0.0.1:8879/charts
stable 	http://mirror.azure.cn/kubernetes/charts/
root@k8s-master:~# helm search consul
NAME                     	CHART VERSION	APP VERSION	DESCRIPTION                                                 
NAME                             	CHART VERSION	APP VERSION	DESCRIPTION                                                 
stable/consul                    	3.9.6        	1.5.3      	Highly available and distributed service discovery and ke...
stable/prometheus-consul-exporter	0.1.5        	0.4.0      	A Helm chart for the Prometheus Consul Exporter             

stable/consul就是我们需要安装的chart。

  • 使用helm安装consul集群命令如下
helm install -n consul \
stable/consul \
--set image=arm64v8/consul

通过上述命令,将以默认的配置在Kubernetes中部署consul集群。默认情况下,chart会安装部署3个consul节点。

该chart中涉及的consul镜像是x86的,只要把镜像换成arm64v8架构的镜像即可。另外,该chart默认开启了持久化存储,不指定storageclass的话,将使用k8s的默认存储。因为我们的树莓派k8s集群已经设置了local-path为默认storageclass,所以不需要关闭持久化存储既可正常安装。

  • 查看创建的pod
root@pi4-master01:~# kubectl get po|grep consul
consul-0                                         1/1     Running   0          6m30s
consul-1                                         1/1     Running   0          5m28s
consul-2                                         1/1     Running   0          4m36s
  • 查看创建的svc
root@pi4-master01:~# kubectl get svc|grep consul
consul                          ClusterIP   None            <none>         8500/TCP,8400/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP   6m58s
consul-ui                       NodePort    10.96.189.212   <none>         8500:5514/TCP                                                                      6m58s

  • 查看创建的pv,pvc
root@pi4-master01:~# kubectl get svc|grep consul
consul                          ClusterIP   None            <none>         8500/TCP,8400/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP   6m58s
consul-ui                       NodePort    10.96.189.212   <none>         8500:5514/TCP                                                                      6m58s
root@pi4-master01:/home/charts/stable# kubectl get pvc,pv|grep consul
persistentvolumeclaim/datadir-consul-0                          Bound    pvc-5152f066-7500-40ed-b008-a2c037b499bf   1Gi        RWO            local-path     7m37s
persistentvolumeclaim/datadir-consul-1                          Bound    pvc-176f02f6-3e5d-4cf2-a506-0348b397e33c   1Gi        RWO            local-path     6m35s
persistentvolumeclaim/datadir-consul-2                          Bound    pvc-b7dd4760-caf7-43f7-82ae-1dcf36e0618c   1Gi        RWO            local-path     5m43s

persistentvolume/pvc-176f02f6-3e5d-4cf2-a506-0348b397e33c   1Gi        RWO            Delete           Bound    default/datadir-consul-1                          local-path              6m31s
persistentvolume/pvc-5152f066-7500-40ed-b008-a2c037b499bf   1Gi        RWO            Delete           Bound    default/datadir-consul-0                          local-path              7m34s
persistentvolume/pvc-b7dd4760-caf7-43f7-82ae-1dcf36e0618c   1Gi        RWO            Delete           Bound    default/datadir-consul-2                          local-path              5m40s

验证

访问http://192.168.5.18:5514/ui/dc1/services ,页面显示如下
树莓派k8s集群安装consul集群及监控_树莓派
该chart默认使用的consul镜像是consul:v1.5.3。

  • 查看当前consul版本
root@pi4-master01:~#  kubectl exec consul-0  -- sh -c 'consul version'
Consul v1.5.3
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

可以看到对应consul的版本是v1.5.3。

  • 查看consul集群信息
root@pi4-master01:~# kubectl exec consul-0  -- sh -c 'consul members'
Node      Address            Status  Type    Build  Protocol  DC   Segment
consul-0  10.244.0.184:8301  alive   server  1.5.3  2         dc1  <all>
consul-1  10.244.2.110:8301  alive   server  1.5.3  2         dc1  <all>
consul-2  10.244.1.107:8301  alive   server  1.5.3  2         dc1  <all>
  • 动态伸缩能力验证
    默认安装是3个节点

root@pi4-master01:~# kubectl get po|grep consul
consul-0                                         1/1     Running      0          36m
consul-1                                         1/1     Running      0          35m
consul-2                                         1/1     Running      0          35m

将consul水平扩容为5个节点

root@pi4-master01:~# kubectl patch statefulset/consul -p '{"spec":{"replicas": 5}}'
statefulset.apps/consul patched
root@pi4-master01:~# kubectl get po|grep consul
consul-0                                         1/1     Running      0          38m
consul-1                                         1/1     Running      0          37m
consul-2                                         1/1     Running      0          36m
consul-3                                         1/1     Running      0          6m27s
consul-4                                         1/1     Running      0          6m4s

将consul水平缩容为3个节点


root@pi4-master01:~# kubectl patch statefulset/consul -p '{"spec":{"replicas": 3}}'
statefulset.apps/consul patched
root@pi4-master01:~# kubectl get po|grep consul
consul-0                                         1/1     Running      0          38m
consul-1                                         1/1     Running      0          37m
consul-2                                         1/1     Running      0          36m

其它配置

基于helm在树莓派k8s集群安装consul集群还是很方便的,只需要使用arm64v8架构的镜像即可。但是如果想要部署的consul集群更具有个性,还是很有必要研究一下该chart的其它配置项,这些配置项我们可以通过helm inspect命令来查看

root@pi4-master01:~# helm inspect values stable/consul
# Default values for consul.
# This is a YAML-formatted file.
# Declare name/value pairs to be passed into your templates.
# name: value

## Consul service ports
HttpPort: 8500
RpcPort: 8400
SerflanPort: 8301
SerflanUdpPort: 8301
SerfwanPort: 8302
SerfwanUdpPort: 8302
ServerPort: 8300
ConsulDnsPort: 8600

## Specify the domain with which consul should run with
## This will be passed as a -domain parameter
Domain: consul

## Used as selector
Component: "consul"
Replicas: 3
Image: "consul"
ImageTag: "1.5.3"
ImagePullPolicy: "Always"
Resources: {}
 # requests:
 #   cpu: "100m"
 #   memory: "256Mi"
 # limits:
 #   cpu: "500m"
 #   memory: "512Mi"
## Persistent volume size

priorityClassName: ""

Storage: "1Gi"

## Needed for 0.8.0 and later IF all consul containers are spun up
## on the same machine. Without this they all generate the same
## host id.
DisableHostNodeId: false

## Datacenter name for consul. If not supplied, will use the consul
## default 'dc1'
# DatacenterName: dc1

## Explicitly set LAN hosts to join.
# Added as -retry-join argument
# If you set joinPeers then we will not auto-build the list of peers for you
# These hostnames will be verified to be resolvable and ping-able before the consul service will start
joinPeers: []

## Set list of WAN hosts to join
# Added as -retry-join-wan argument
# These hostnames will be verified to be resolvable before the consul service will start
joinWan: []

## Encrypt Gossip Traffic
Gossip:
  Encrypt: true
  Create: true

## predefined value for gossip key.
## Will use a generated random alpha numeric if not provided
# GossipKey: key

## consul data Persistent Volume Storage Class
## If defined, StorageClassName: <storageClass>
## If set to "-", StorageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
##   set, choosing the default provisioner.  (gp2 on AWS, standard on
##   GKE, AWS & OpenStack)
##
# StorageClass: "-"

## Setting maxUnavailable will create a pod disruption budget that will prevent
## voluntarty cluster administration from taking down too many consul pods. If
## you set maxUnavailable, you should set it to ceil((n/2) - 1), where
## n = Replicas. For example, if you have 5 or 6 Replicas, you'll want to set
## maxUnavailable = 2. If you are using the default of 3 Replicas, you'll want
## to set maxUnavailable to 1.
maxUnavailable: 1

## nodeAffinity settings
# nodeAffinity:
#   requiredDuringSchedulingIgnoredDuringExecution:
#     nodeSelectorTerms:
#     - matchExpressions:
#       - key: cloud.google.com/gke-preemptible
#         operator: NotIn
#         values:
#         - true

## Affinity settings
affinity: |
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 1
      podAffinityTerm:
        topologyKey: kubernetes.io/hostname
        labelSelector:
          matchExpressions:
          - key: component
            operator: In
            values:
            - "{{ .Release.Name }}-{{ .Values.Component }}"

## Enable Consul Web UI
##
ui:
  enabled: true
## Create dedicated UI service
##
uiService:
  enabled: true
  type: "NodePort"
  annotations: {}

ConsulConfig: []
#  - type: secret
#    name: consul-defaults
#  - type: configMap
#    name: consul-defaults

## Create an Ingress for the Web UI
uiIngress:
  enabled: false
  annotations: {}
  labels: {}
  hosts: []
  path: /
  tls: []

## Useful when ACLs are enabled
acl:
  enabled: false
  masterToken: ""
  agentToken: ""

## test container details
test:
  image: lachlanevenson/k8s-kubectl
  imageTag: v1.4.8-bash
  rbac:
    create: false
    serviceAccountName: ""

nodeSelector: {}
tolerations: []
additionalLabels: {}
podAnnotations: {}

# Lifecycle for StatefulSet
# lifecycle:
#   preStop:
#     exec:
#       command:
#       - sh
#       - -c
#       - "sleep 60"
forceIpv6: false

显然,配置项很丰富,从端口配置、是否开启ingress都能支持。

安装监控

安装监控前,需要在集群部署prometheus,否则不会有效果。可以参考树莓派k8s集群安装监控prometheus进行部署。 consul的监控需要单独安装,这里我们选择https://github.com/pi4k8s/pi4charts/tree/master/monitor/charts/consul-monitor 进行安装,该仓库内置的镜像是prom/consul-exporter:latest,该镜像支持多架构,可以同时支持arm64v8和x86。

使用helm安装开consul监控命令如下

root@pi4-master01:~# git clone https://github.com/pi4k8s/pi4charts.git
root@pi4-master01:~# cd pi4charts
root@pi4-master01:~/pi4charts# helm install -n consul-monitor \
monitor/charts/consul-monitor/ \
--set consulServer=consul:8500

确认安装状态

root@pi4-master01:~# kubectl get pod | grep consul
consul-0                                         1/1     Running      0          3h48m
consul-1                                         1/1     Running      0          3h47m
consul-2                                         1/1     Running      0          3h46m
consul-monitor-f68cdb4d-w72zq                    1/1     Running      0          68s
root@pi4-master01:~# kubectl get svc | grep consul
consul                          ClusterIP   None            <none>         8500/TCP,8400/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP   3h48m
consul-monitor                  ClusterIP   10.111.89.48    <none>         9107/TCP                                                                           112s
consul-ui                       NodePort    10.96.189.212   <none>         8500:5514/TCP                                                                      3h48m
root@pi4-master01:~#  kubectl get servicemonitor --all-namespaces |grep consul
default      consul-monitor            2m42s

在prometheus查看exporter是否生效
树莓派k8s集群安装consul集群及监控_helm_02

生效后在grafana系统配置consul监控面板

部署步骤如下:
树莓派k8s集群安装consul集群及监控_kubernetes_03

树莓派k8s集群安装consul集群及监控_kubernetes_04

点"Upload .json file"按钮,选择已经下载好的consul监控面板文件 ,进入面板设置界面如下

树莓派k8s集群安装consul集群及监控_prometheus_05 设置好相应参数后,点"Import"按钮,继续导入,引导入consul监控页面如下

树莓派k8s集群安装consul集群及监控_bash_06

总结

本实践简单介绍了如何使用chart在树莓派k8s集群用helm部署consul和consul-monitor,读者可以基于自己的需要自行调整consul参数以便适用更多场景。

参考: