k8s部署Keepalived k8s部署项目

目录

1、架构图

2、部署一个java网站项目实践

2.1 编译java

2.2、构建与上传镜像

2.3、k8s中部署项目

访问

3、资源配置

3.2、加入配置文件挂载

4、部署数据库并连接

恢复数据

项目连接数据库

5、更新

执行

6、ingress暴露（前提需要运行ingress服务）

7、配置外网负载均衡

---

1、架构图

在没有k8s前，一个项目的开发部署架构大概是这样的

在有k8s后，这个架构变成了这样

在k8s中部署项目的流程

2、部署一个java网站项目实践

2.1 编译java

第一步：先将代码拉取到本地（这里有一个实例代码拉取），下载好的代码就是一个包，没有的可以通过此命令拉取

git clone https://github.com/lizhenliang/tomcat-java-demo

java项目需要下载maven依赖和java-1.8环境编译，yum安装

yum install java-1.8.0-openjdk maven git -y

安装成功

进入java包内

cd tomcat-java-demo-master

修改mvn为阿里云镜像地址

vi /etc/maven/settings.xml

在mirror位置加入阿里云地址

阿里云仓库地址为

<mirror>
   
  <id>aliyunmaven</id>
   
  <mirrorOf>*</mirrorOf>
   
  <name>阿里云公共仓库</name>
   
  <url>https://maven.aliyun.com/repository/public</url></mirror>

清理过去的依赖并跳过测试并编译代码

mvn clean package -Dmaven.test.skip=ture

成功，成功后会生成一个target目录

target目录里面会多一个war包

ls target/

这个war包就是要运行的程序

解压这个war包内容到ROOT目录

unzip target/ly-simple-tomcat-0.0.1-SNAPSHOT.war -d ROOT

root内就是可运行的项目

2.2、构建与上传镜像

制作一个dockerfile

vi Dockerfile

FROM lizhenliang/tomcat

LABEL maintainer af

COPY target/ROOT /usr/local/tomcat/webapps/ROOT

这个dockerfile意思是引入一个tomcat镜像，在将运行项目放入这个tomcat的运行目录，注意，这个docker要在tomcat-java-demo-master目录执行

执行打包命令，将此镜像命名为java-demo:v1 

docker build -t java-demo:v1 .

在harbor创建一个demo私有仓库

注意：在harbor中，公开仓库是免凭据拉取，推送需要凭据

                               私有仓库拉取与推送都需要凭据

给刚刚新建的镜像打上tag(镜像名前面对应仓库服务器ip与要推送的仓库名)

docker tag java-demo:v1 192.168.209.100/demo/java-demo:v1

vi /etc/docker/daemon.json

{
   
192.168.209.100"]
   
}

在服务器上加入域名解析

vi /etc/hosts

192.168.209.100 reg.ctnrs.com

登录harbor

docker login 192.168.209.100

账号admin，密码harbor

上传镜像

docker push 192.168.209.100/demo/java-demo:v1

2.3、k8s中部署项目

创建一个测试yaml

kubectl create deployment java-demo --image=192.168.209.100/demo/java-demo:v1 --dry-run=client -o yaml > deployment.yaml

此时如果直接执行yaml会报仓库没有授权

所以，此时需要k8s中的Secret来进行操作，创建一个k8s的Secret来保存凭据，kubectl调用这个凭据

kubectl create secret docker-registry harbor-auth --docker-username=admin --docker-password=Harbor12345 --docker-server=192.168.209.100

查看是否创建成功

kubectl get secret

在deployment中引用这个secret（他与containers同级）

vi deployment

apiVersion: apps/v1
   
kind: Deployment
   
metadata:
   
  labels:
   
    app: java-demo
   
  name: java-demo
   
spec:
   
  replicas: 1
   
  selector:
   
    matchLabels:
   
      app: java-demo
   
  strategy: {}
   
  template:
   
    metadata:
   
      creationTimestamp: null
   
      labels:
   
        app: java-demo
   
    spec:
   
imagePullSecrets:
   
      - name: harbor-auth
   
      containers:
   
      - image: 192.168.209.100/demo/java-demo:v1
   
        name: java-demo
   
        resources: {}

执行

kubectl apply -f deployment.yaml

成功

访问

为这个deployment创建service访问

kubectl expose deployment java-demo --port=80 --target-port=8080 --type=NodePort

成功，对外端口为32575

访问

访问成功

3、资源配置

使用控制器部署镜像

给项目加上资源限制与健康检查

vi deployment

一般项目部署都需要配置这些

apiVersion: apps/v1
   
kind: Deployment
   
metadata:
   
  labels:
   
    app: java-demo
   
  name: java-demo
   
spec:
   
  replicas: 1
   
  selector:
   
    matchLabels:
   
      app: java-demo
   
  strategy: {}
   
  template:
   
    metadata:
   
      creationTimestamp: null
   
      labels:
   
        app: java-demo
   
    spec:
   
      imagePullSecrets:
   
      - name: harbor-auth
   
      containers:
   
      - image: 192.168.209.100/demo/java-demo:v1
   
        name: java-demo
   
 resources:
   
          limits:
   
            cpu: 1
   
            memory: 1Gi
   
          requests:
   
            cpu: 0.7
   
            memory: 700Mi
   
        livenessProbe:
   
          httpGet:
   
            path: /
   
            port: 8080
   
          initialDelaySeconds: 50
   
          periodSeconds: 30
   
        readinessProbe:
   
          httpGet:
   
            path: /
   
            port: 8080
   
          initialDelaySeconds: 50
   
          periodSeconds: 30

3.2、加入配置文件挂载

先将配置文件存入configMap

vi configmap.yaml

apiVersion: v1
   
kind: ConfigMap
   
metadata:
   
  name: java-demo-config
   
data:
   
  application.yaml: |
   
      server:
   
        port: 8080
   
      spring:
   
        datasource:
   
          url: jdbc:mysql://localhost:3306/test?characterEncoding=utf-8
   
          username: root
   
          password: 12345
   
          driver-class-name: com.mysql.jdbc.Driver
   
        freemarker:
   
          allow-request-override: false
   
          cache: true
   
          check-template-location: true
   
          charset: UTF-8
   
          content-type: text/html; charset=utf-8
   
          expose-request-attributes: false
   
          expose-session-attributes: false
   
          expose-spring-macro-helpers: false
   
          suffix: .ftl
   
          template-loader-path:
   
            - classpath:/templates/

kubectl apply -f configmap.yaml

kubectl get configmap

成功

在deployment中通过configmap挂载配置文件到项目

vi deployment

apiVersion: apps/v1
   
kind: Deployment
   
metadata:
   
  labels:
   
    app: java-demo
   
  name: java-demo
   
spec:
   
  replicas: 1
   
  selector:
   
    matchLabels:
   
      app: java-demo
   
  strategy: {}
   
  template:
   
    metadata:
   
      creationTimestamp: null
   
      labels:
   
        app: java-demo
   
    spec:
   
      imagePullSecrets:
   
      - name: harbor-auth
   
      containers:
   
      - image: 192.168.209.100/demo/java-demo:v1
   
        name: java-demo
   
        resources:
   
          limits:
   
            cpu: 1
   
            memory: 1Gi
   
          requests:
   
            cpu: 0.7
   
            memory: 700Mi
   
        livenessProbe:
   
          httpGet:
   
            path: /
   
            port: 8080
   
          initialDelaySeconds: 50
   
          periodSeconds: 30
   
        readinessProbe:
   
          httpGet:
   
            path: /
   
            port: 8080
   
          initialDelaySeconds: 50
   
          periodSeconds: 30
   
volumeMounts:
   
        - name: config
   
          mountPath: "/usr/local/tomcat/webapps/ROOT/WEB-INF/classes/application.yml"
   
          subPath: "application.yml"
   
      volumes:
   
      - name: config
   
        configMap:
   
          name: java-demo-config
   
          items:
   
          - key: "application.yaml"
   
            path: "application.yml"

kubectl applyment -f deployment

4、部署数据库并连接

一般情况下不建议k8s中部署mysql

将mysql.yaml传入服务器

此mysql的部署用到了数据卷nfs动态供给（pv、pvc）没有做的参考前面的nfs部署与pv、pvc设置文档

如果没有nfs动态供给环境，可以删除标红内容改为emptyDir: {}，与name同缩进，做临时数据库存储

vi mysql.yaml

apiVersion: v1
   
kind: Secret
   
metadata:
   
  name: java-demo-db
   
type: Opaque
   
data:
   
  mysql-root-password: "MTIzNDU2"
   
  mysql-password: "MTIzNDU2"
   
---
   
apiVersion: apps/v1
   
kind: Deployment
   
metadata:
   
  name: java-demo-db
   
spec:
   
  selector:
   
    matchLabels:
   
      project: www
   
      app: mysql
   
  template:
   
    metadata:
   
      labels:
   
        project: www
   
        app: mysql
   
    spec:
   
      containers:
   
      - name: db
   
        image: mysql:5.7.30
   
        resources:
   
          requests:
   
            cpu: 500m
   
            memory: 512Mi
   
          limits:
   
            cpu: 500m
   
            memory: 512Mi
   
        env:
   
        - name: MYSQL_ROOT_PASSWORD
   
          valueFrom:
   
            secretKeyRef:
   
              name: java-demo-db
   
              key: mysql-root-password
   
        - name: MYSQL_PASSWORD
   
          valueFrom:
   
            secretKeyRef:
   
              name: java-demo-db
   
              key: mysql-password
   
        - name: MYSQL_USER
   
          value: "aliang"
   
        - name: MYSQL_DATABASE
   
          value: "k8s"
   
        ports:
   
        - name: mysql
   
          containerPort: 3306
   
        livenessProbe:
   
          exec:
   
            command:
   
            - sh
   
            - -c
   
            - "mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}"
   
          initialDelaySeconds: 30
   
          periodSeconds: 10
   
        readinessProbe:
   
          exec:
   
            command:
   
            - sh
   
            - -c
   
            - "mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}"
   
          initialDelaySeconds: 5
   
          periodSeconds: 10
   
        volumeMounts:
   
        - name: data
   
          mountPath: /var/lib/mysql
   

   

   
      volumes:
   
      - name: data
   
 persistentVolumeClaim:
   
          claimName: java-demo-db
   
---
   
apiVersion: v1
   
kind: PersistentVolumeClaim
   
metadata:
   
  name: java-demo-db
   
spec:
   
  storageClassName: "managed-nfs-storage"
   
  accessModes:
   
    - "ReadWriteOnce"
   
  resources:
   
    requests:
   
      storage: "8Gi"
   
---
   
apiVersion: v1
   
kind: Service
   
metadata:
   
  name: java-demo-db
   
spec:
   
  type: ClusterIP
   
  ports:
   
  - name: mysql
   
    port: 3306
   
    targetPort: mysql
   
  selector:
   
    project: www
   
    app: mysql

kubectl apply -f mysql.yaml

恢复数据

将sql文件导入到k8s中的mysql容器中

sql文件所在位置

cd /root/tomcat-java-demo-master/db

将sql文件拷贝进容器

kubectl cp tables_ly_tomcat.sql java-demo-db-76d66777b6-bcwx4:/

进入mysql容器，sql文件已传入

kubectl exec -it java-demo-db-76d66777b6-bcwx4 -- bash

连接数据库，密码为123456，用show databases可以看到数据库内有k8s库，变量生效

mysql -uroot -p123456

在mysql中创建一个数据库test并切换到这个数据库

create database test;

use test；

在test库中执行拷贝进来的sql

source /tables_ly_tomcat.sql;

可以看到表已经导入

show tables；

项目连接数据库

修改之前创建的与deployment绑定的configmap，修改他的mysql连接地址为mysql的地址，使用mysql的service中CLUSTER-IP的ip与端口，账号密码也进行对应修改

vi configmap.yaml

apiVersion: v1
   
kind: ConfigMap
   
metadata:
   
  name: java-demo-config
   
data:
   
  application.yaml: |
   
      server:
   
        port: 8080
   
      spring:
   
        datasource:
   
10.108.140.66:3306/test?characterEncoding=utf-8
   
 username: root
   
          password: 123456
   
          driver-class-name: com.mysql.jdbc.Driver
   
        freemarker:
   
          allow-request-override: false
   
          cache: true
   
          check-template-location: true
   
          charset: UTF-8
   
          content-type: text/html; charset=utf-8
   
          expose-request-attributes: false
   
          expose-session-attributes: false
   
          expose-spring-macro-helpers: false
   
          suffix: .ftl
   
          template-loader-path:
   
            - classpath:/templates/

执行

kubectl apply -f configmap.yaml

5、更新

修改了configmap但程序也不会动态更新，需要人工来更新，此时我们进行一次更新的全套模拟操作

cd /root/tomcat-java-demo-master

随便修改项目主页一点内容，模拟一次更新

vi src/main/resources/templates/index.ftl

用maven重新编译

mvn clean package -Dmaven.test.skip=ture

重新解压放入ROOT

cd target/

unzip ly-simple-tomcat-0.0.1-SNAPSHOT.war -d ROOT

回到docker位置重新打包

cd ..

docker build -t java-demo:v2 .

修改tag

docker tag java-demo:v2 192.168.209.100/demo/java-demo:v2

上传镜像到仓库

docker push 192.168.209.100/demo/java-demo:v2

修改deployment的yaml，改用新的镜像v2版本

vi deployment.yaml

apiVersion: apps/v1
   
kind: Deployment
   
metadata:
   
  labels:
   
    app: java-demo
   
  name: java-demo
   
spec:
   
  replicas: 1
   
  selector:
   
    matchLabels:
   
      app: java-demo
   
  strategy: {}
   
  template:
   
    metadata:
   
      creationTimestamp: null
   
      labels:
   
        app: java-demo
   
    spec:
   
      imagePullSecrets:
   
      - name: harbor-auth
   
      containers:
   
v2
   
        name: java-demo
   
        resources:
   
          limits:
   
            cpu: 1
   
            memory: 1Gi
   
          requests:
   
            cpu: 0.7
   
            memory: 700Mi
   
        livenessProbe:
   
          httpGet:
   
            path: /
   
            port: 8080
   
          initialDelaySeconds: 50
   
          periodSeconds: 30
   
        readinessProbe:
   
          httpGet:
   
            path: /
   
            port: 8080
   
          initialDelaySeconds: 50
   
          periodSeconds: 30
   
        volumeMounts:
   
        - name: config
   
          mountPath: "/usr/local/tomcat/webapps/ROOT/WEB-INF/classes/application.yml"
   
          subPath: "application.yml"
   
      volumes:
   
      - name: config
   
        configMap:
   
          name: java-demo-config
   
          items:
   
          - key: "application.yaml"
   
            path: "application.yml"

kubectl apply -f deployment.yaml

执行

访问项目，可以看见页面已经修改，数据库也连接成功

6、ingress暴露（前提需要运行ingress服务）

vi ingress.yaml

apiVersion: networking.k8s.io/v1
   
kind: Ingress
   
metadata:
   
  name: java-demo
   
spec:
   
  ingressClassName: nginx
   
  rules:
   
  - host: java.aliangedu.cn
   
    http:
   
      paths:
   
      - path: /
   
        pathType: Prefix
   
        backend:
   
          service:
   
            name: java-demo
   
            port:
   
              number: 80

kubectl apply -f ingress.yaml

ingress代理成功

7、配置外网负载均衡

使用nginx模拟外网负载均衡

先安装epel源

yum install epel-release

安装nginx

yum install nginx

修改nginx配置

vi /etc/nginx/nginx.conf

加入这两段对ingress进行外网暴露与负载均衡

upstream ingress-controller {
   
        server 192.168.209.111:30761;
   
        server 192.168.209.112:30761;
   
    }
   
    server {
   
        listen       80;
   
        listen       [::]:80;
   
        server_name  _;
   
        root         /usr/share/nginx/html;
   
 location / {
   
            proxy_pass http://ingress-controller;
   
            proxy_set_header Host $Host;
   
        }

启动nginx

systemctl start nginx

这时访问nginx的端口会被代理到ingress服务上

监听nginx访问日志可以看到我们访问信息

代理成功