operator架构 operator管理框架

转载

mob64ca13f7ecc9 2023-12-25 22:10:08

文章标签 operator架构自定义 Deployment Web 文章分类 架构后端开发

kubernetes、kubebuilder、operator 入门

一、Operator简介

在Kubernetes中我们经常使用Deployment、DaemonSet、Service、ConfigMap等资源，这些资源都是Kubernetes的内置资源，他们的创建、更新、删除等均有Controller Manager负责管理。

二、Operator组成

Operator(Controller+CRD），Operator是由Kubernetes自定义资源（CRD）和控制器（Controller）构成的元原生扩展服务，其中CRD定义了每个Operator需要创建和管理的自定义资源对象，底层实际就是通过APIServer接口在ETCD中注册一种新的资源类型，注册完成后就可以创建该资源类型的对象了。但是仅注册资源和创建资源对象是没有任何实际意义的，CRD最重要的是需要配合对应的Controller来实现自定义资源的功能达到自定义资源期望的状态，比如内置的Deployment Controller用来控制Department资源对象的功能，根据配置生成特定数量的Pod监控其状态，并根据事件做出相应的动作。

三、Operator使用

用户想为自己的自定义资源构建一个Kubernetes Operator，有很多工具可供选择比如Operator SDK、Kubebuilder，甚至可以使用Operator SDK（HELM、Ansible、Go）。这些工具创建Kubernetes Operator用来监控自定义资源，并且根据资源的变化调整资源状态。

Operator作为自定义扩展资源以Deployment的方式部署到Kubernetes中，通过List-Watch方式监听对应资源的变化，当用户修改自定义资源中的任何内容，Operator会监控资源的更改，并根据更改内容执行特定的操作，这些操作通常会对Kubernetes API中某些资源进行调用。

四、Operator应用案例

Kubebuilder介绍

Kubebuilder是一个用Go语言构建Kubenetes API控制器和CRD的脚手架工具，通过使用Kubebuilder,用户可以遵循一套简单的编程框架，编写Operator使用实例。

kubenetes: v1.23.8
go

tar xf go1.18.3.linux-amd64.tar.gz
mv go /usr/local/
vim /etc/profile.d/go183.sh
export GO111MODULE=on
export GOROOT=/usr/local/go 
export GOPATH=/root/gopath
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

source /etc/profile.d/go183.sh

kubebuilder: 3.5.0

yum -y install gcc
curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

kubebuilder version

案例

　　Welcome案例主要实现使用Operator和CRD部署一套完整的应用环境，可以实现根据自定义类型创建资源，通过创建一个Welcome类型的资源，后台自动创建Deployment和Service，通过Web页面访问Service呈现应用部署，通过自定义控制器方式进行控制管理

　　我们西药创建Welcome自定义资源及对应的Controllers，最终我们可以通过类似如下代码的YAML文件部署简单的Web应用

apiVersion: webapp.demo.welcome.domain/v1
kind: Welcome
metadata:
  name: welcome-sample
spec:
  name: myfriends

Web应用介绍

本案例中，我们使用Go语言HTTP模块创建一个Web服务，用户访问页面后会自动加载NAME及PORT环境变量并渲染index.html静态文件中，代码如下：

package main

import (
	"fmt"
	"net/http"
	"os"
)

func main() {
	name := os.Getenv("NAME")
	hello := fmt.Sprintf("Hello %s", name)
	http.Handle("/hello/", http.StripPrefix("/hello/", http.FileServer(http.Dir("static"))))
	f, err := os.OpenFile("./static/index.html", os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
	if err != nil {
		panic(err)
	}
	defer f.Close()
	if _, err = f.WriteString(hello); err != nil {
		panic(err)
	}
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
}

其中，NAME环境变量通过我们在Welcome中定义的name字段获取，我们在下面的控制器编写中会详细介绍获取字段的详细方法。我们将index.html放在Static文件夹下，并将工程文件打包为Docker镜像，Dockerfile如下：

FROM golang:1.17 as builder

WORKDIR /
COPY . .
COPY static

RUN CGO_ENABLED=0 GOOS=linux go build -v -o main
FROM alpine
RUN apk add --no-cache ca-certificates
COPY --from=builder /main /usr/local/main
COPY --from=builder static /static
CMD ["/usr/local/main"]

项目初始化

使用Kubebuilder命令镜像项目初始化

mkdir demo
cd demo
go mod init welcome_demo.domain
kubebuilder init --domain demo.welcome.domain

初始化项目后，kubebuilder生成项目接口如下

[root@k8s-01 demo]# tree .
.
├── config
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   └── rbac
│       ├── auth_proxy_client_clusterrole.yaml
│       ├── auth_proxy_role_binding.yaml
│       ├── auth_proxy_role.yaml
│       ├── auth_proxy_service.yaml
│       ├── kustomization.yaml
│       ├── leader_election_role_binding.yaml
│       ├── leader_election_role.yaml
│       ├── role_binding.yaml
│       └── service_account.yaml
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
├── PROJECT
└── README.md

6 directories, 25 files

创建`Welcome` Kind和其对应的控制器

[root@k8s-01 demo]# kubebuilder create api --group webapp --kind Welcome --version v1
Create Resource [y/n]
y
Create Controller [y/n]
y

输入两次y, Kubebuilder 分别创建了资源和控制器的模板，此处的group、version、kind这3个属性组合起来标识一个k8s的CRD，创建完成后，Kubebuilder添加文件如下：

[root@k8s-01 demo]# tree .
.
├── api
│   └── v1
│       ├── groupversion_info.go
│       ├── welcome_types.go
│       └── zz_generated.deepcopy.go
├── bin
│   └── controller-gen
├── config
│   ├── crd
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │       ├── cainjection_in_welcomes.yaml
│   │       └── webhook_in_welcomes.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── role_binding.yaml
│   │   ├── service_account.yaml
│   │   ├── welcome_editor_role.yaml
│   │   └── welcome_viewer_role.yaml
│   └── samples
│       └── webapp_v1_welcome.yaml
├── controllers
│   ├── suite_test.go
│   └── welcome_controller.go
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
├── PROJECT
└── README.md

13 directories, 38 files

后续需要执行两步操作：

修改Resource Type
修改Controller 逻辑

修改Resource Type

此处Resource Type为需要定义的资源字段，用于在Yaml文件中进行声明，本案例中需要新增name字段用于“Welcome”Kind中的Web应用，代码如下：

vim api/v1/welcome_types.go

// WelcomeSpec defines the desired state of Welcome
type WelcomeSpec struct {
        // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
        // Important: Run "make" to regenerate code after modifying this file

        // Foo is an example field of Welcome. Edit welcome_types.go to remove/update
        // Foo string `json:"foo,omitempty"`
        Name string `json:"name,omitempty"`
}

修改Controller逻辑

在Controller中需要通过Reconcile方法完成Deployment和Service部署，并最终达到期望的状态。

vim controllers/welcome_controller.go

//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Welcome object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.12.1/pkg/reconcile
func (r *WelcomeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        // _ = log.FromContext(ctx)

        // TODO(user): your logic here
        log := r.Log.WithValues("welcome", req.NamespacedName)
        log.Info("reconcilling welcome")
        return ctrl.Result{}, nil
}

最终代码如下

/*
Copyright 2022.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controllers

import (
	"context"
	"fmt"
	appsv1 "k8s.io/api/apps/v1"
	"k8s.io/apimachinery/pkg/api/resource"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/util/intstr"
	"sigs.k8s.io/controller-runtime/pkg/log"

	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	webappv1 "welcome_demo.domain/api/v1"
)

// WelcomeReconciler reconciles a Welcome object
type WelcomeReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Welcome object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.12.1/pkg/reconcile
func (r *WelcomeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)

	// TODO(user): your logic here
	welcome := &webappv1.Welcome{}
	if err := r.Client.Get(ctx, req.NamespacedName, welcome); err != nil {
		return ctrl.Result{}, err
	}
	deployment, err := r.createWelcomeDeployment(welcome)
	if err != nil {
		return ctrl.Result{}, err
	}
	fmt.Println("create deployment success!")
	svc, err := r.createService(welcome)
	if err != nil {
		return ctrl.Result{}, err
	}
	fmt.Println("create service success!")
	applyOpts := []client.PatchOption{client.ForceOwnership, client.FieldOwner("welcome_controller")}
	err = r.Patch(ctx, &deployment, client.Apply, applyOpts...)
	if err != nil {
		return ctrl.Result{}, err
	}
	err = r.Patch(ctx, &svc, client.Apply, applyOpts...)
	if err != nil {
		return ctrl.Result{}, err
	}
	return ctrl.Result{}, nil
}

func (r *WelcomeReconciler) createWelcomeDeployment(welcome *webappv1.Welcome) (appsv1.Deployment, error) {
	defOne := int32(1)
	name := welcome.Spec.Name
	if name == "" {
		name = "world"
	}
	depl := appsv1.Deployment{
		TypeMeta: metav1.TypeMeta{
			APIVersion: appsv1.SchemeGroupVersion.String(),
			Kind:       "Deployment",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      welcome.Name,
			Namespace: welcome.Namespace,
		},
		Spec: appsv1.DeploymentSpec{
			Replicas: &defOne,
			Selector: &metav1.LabelSelector{
				MatchLabels: map[string]string{"welcome": welcome.Name},
			},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{
					Labels: map[string]string{"welcome": welcome.Name},
				},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{
						{
							Name: "welcome",
							Env: []corev1.EnvVar{
								{Name: "NAME", Value: name},
							},
							Ports: []corev1.ContainerPort{
								{ContainerPort: 8080,
									Name:     "http",
									Protocol: "TCP",
								},
							},
							Image: "sdfcdwefe/operatordemo:v1",
							Resources: corev1.ResourceRequirements{
								Requests: corev1.ResourceList{
									corev1.ResourceCPU:    *resource.NewMilliQuantity(100, resource.DecimalSI),
									corev1.ResourceMemory: *resource.NewMilliQuantity(100000, resource.BinarySI),
								},
							},
						},
					},
				},
			},
		},
	}

	return depl, nil
}

func (r *WelcomeReconciler) createService(welcome *webappv1.Welcome) (corev1.Service, error) {
	svc := corev1.Service{
		TypeMeta: metav1.TypeMeta{
			APIVersion: corev1.SchemeGroupVersion.String(),
			Kind:       "Service"},
		ObjectMeta: metav1.ObjectMeta{
			Name:      welcome.Name,
			Namespace: welcome.Namespace,
		},
		Spec: corev1.ServiceSpec{
			Ports: []corev1.ServicePort{
				{Name: "http",
					Port:       8080,
					Protocol:   "TCP",
					TargetPort: intstr.FromString("http")},
			},
			Selector: map[string]string{"welcome": welcome.Name},
			Type:     corev1.ServiceTypeLoadBalancer,
		},
	}
	return svc, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *WelcomeReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&webappv1.Welcome{}).
		Complete(r)
}

Welcome应部署

生成CRD资源

[root@k8s-01 demo]# make manifests
/root/demo/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

[root@k8s-01 demo]# tree .
.
├── api
│   └── v1
│       ├── groupversion_info.go
│       ├── welcome_types.go
│       └── zz_generated.deepcopy.go
├── bin
│   └── controller-gen
├── config
│   ├── crd
│   │   ├── bases
│   │   │   └── webapp.demo.welcome.domain_welcomes.yaml
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │       ├── cainjection_in_welcomes.yaml
│   │       └── webhook_in_welcomes.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── role_binding.yaml
│   │   ├── role.yaml
│   │   ├── service_account.yaml
│   │   ├── welcome_editor_role.yaml
│   │   └── welcome_viewer_role.yaml
│   └── samples
│       └── webapp_v1_welcome.yaml
├── controllers
│   ├── suite_test.go
│   └── welcome_controller.go
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
├── PROJECT
└── README.md

14 directories, 40 files


[root@k8s-01 demo]# cat config/crd/bases/webapp.demo.welcome.domain_welcomes.yaml 
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.9.0
  creationTimestamp: null
  name: welcomes.webapp.demo.welcome.domain
spec:
  group: webapp.demo.welcome.domain
  names:
    kind: Welcome
    listKind: WelcomeList
    plural: welcomes
    singular: welcome
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: Welcome is the Schema for the welcomes API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: WelcomeSpec defines the desired state of Welcome
            properties:
              name:
                description: Foo is an example field of Welcome. Edit welcome_types.go
                  to remove/update Foo string `json:"foo,omitempty"`
                type: string
            type: object
          status:
            description: WelcomeStatus defines the observed state of Welcome
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}

创建Welcome类型资源

[root@k8s-01 demo]# kubectl create -f config/crd/bases/
customresourcedefinition.apiextensions.k8s.io/welcomes.webapp.demo.welcome.domain created
[root@k8s-01 demo]# kubectl create -f config/samples/webapp_v1_welcome.yaml 
welcome.webapp.demo.welcome.domain/welcome-sample created

使用`kubectl get crd`命令查看自定义对象

[root@k8s-01 demo]# kubectl get crd | grep welcome
welcomes.webapp.demo.welcome.domain          2022-06-25T09:10:37Z

通过kubectl get welcome命令可以看到创建的welcome对象

[root@k8s-01 demo]# kubectl get welcome
NAME             AGE
welcome-sample   2m20s

此时CRD并不会完成任何工作，只是在ETCD中创建了一条记录，我们需要运行Controller才能帮助我们完成工作并最终达到welcome定义的状态。

[root@k8s-01 demo]# make run
/root/demo/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/demo/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
api/v1/welcome_types.go
go vet ./...
go run ./main.go
1.6561485987622015e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
1.6561485987624757e+09	INFO	setup	starting manager
1.6561485987638762e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.656148598763948e+09	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
1.656148598764167e+09	INFO	Starting EventSource	{"controller": "welcome", "controllerGroup": "webapp.demo.welcome.domain", "controllerKind": "Welcome", "source": "kind source: *v1.Welcome"}
1.6561485987641926e+09	INFO	Starting Controller	{"controller": "welcome", "controllerGroup": "webapp.demo.welcome.domain", "controllerKind": "Welcome"}
1.6561485988653958e+09	INFO	Starting workers	{"controller": "welcome", "controllerGroup": "webapp.demo.welcome.domain", "controllerKind": "Welcome", "worker count": 1}
create deployment success!
create service success!

以上方式在本地启动控制器，方便调试和验证

查看创建的deployment、service

[root@k8s-01 demo]# kubectl get deploy
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
welcome-sample   1/1     1            1           34s
[root@k8s-01 demo]# kubectl get svc
NAME             TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes       ClusterIP      10.96.0.1       <none>        443/TCP          4d2h
welcome-sample   LoadBalancer   10.106.36.129   <pending>     8080:30037/TCP   39s

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。