k8s集群中搭建gitlab
https://www.qikqiak.com/k8s-book/docs/64.Gitlab.html
gitlab有没有必要部署在k8s集群中? gitlab占用的资源较多,依赖组建复杂。部署在k8s集群中也是一个不错的选择。
依赖组件:ruby 1.9.3+,MySQL,git,redis, Sidekiq。
最低配置CPU 1G,RAM 1G+swap可以支持100用户。
gitlab(一):gitlab简介和安装
Gitlab
官方提供了 Helm 的方式在 Kubernetes 集群中来快速安装,但是在使用的过程中发现 Helm 提供的 Chart 包中有很多其他额外的配置,所以我们这里使用自定义的方式来安装,也就是自己来定义一些资源清单文件。
Gitlab
主要涉及到3个应用:Redis、Postgresql、Gitlab 核心程序,实际上我们只要将这3个应用分别启动起来,然后加上对应的配置就可以很方便的安装 Gitlab 了,我们这里选择使用的镜像不是官方的,而是 Gitlab 容器化中使用非常多的一个第三方镜像:sameersbn/gitlab
,基本上和官方保持同步更新,地址:http://www.damagehead.com/docker-gitlab/
如果我们已经有可使用的 Redis 或 Postgresql 服务的话,那么直接配置在 Gitlab 环境变量中即可,如果没有的话就单独部署。
Ingress采用的Nginx
# kubectl get pods -n ingress-nginx -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ingress-controller-8n6fz 1/1 Running 0 8d 10.6.76.23 k8s-node-1 <none> <none>
nginx-ingress-controller-jt82z 1/1 Running 0 8d 10.6.76.24 k8s-node-2 <none> <none>
1 创建PVC和storageclass做持久化
redis、postgresql、gitlab都需要持久化
1.1 创建持久化StorageClass
#cat gitlab-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gitlab-storageclass
provisioner: fuseim.pri/ifs
1.2 vim gitlab-redis-pvc.yaml
#vim gitlab-redis-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gitlab-redis-pvc
namespace: kube-ops
annotations:
volume.beta.kubernetes.io/storage-class: "gitlab-storageclass"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
1.3 vim gitlab-postgresql-pvc.yaml:
#vim gitlab-postgresql-pvc.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gitlab-postgresql-pvc
namespace: kube-ops
annotations:
volume.beta.kubernetes.io/storage-class: "gitlab-storageclass"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
1.4 vim gitlab-pvc.yaml
#vim gitlab-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gitlab-pvc
namespace: kube-ops
annotations:
volume.beta.kubernetes.io/storage-class: "gitlab-storageclass"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
2 gitlab-redis.yaml
首先部署需要的 Redis 服务,对应的资源清单文件如下:(gitlab-redis.yaml)
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: redis
namespace: kube-ops
labels:
name: redis
spec:
template:
metadata:
name: redis
labels:
name: redis
spec:
containers:
- name: redis
image: sameersbn/redis
imagePullPolicy: IfNotPresent
ports:
- name: redis
containerPort: 6379
volumeMounts:
- mountPath: /var/lib/redis
name: data
livenessProbe:
exec:
command:
- redis-cli
- ping
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
exec:
command:
- redis-cli
- ping
initialDelaySeconds: 5
timeoutSeconds: 1
volumes:
- name: data
persistentVolumeClaim:
claimName: gitlab-redis-pvc
---
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: kube-ops
labels:
name: redis
spec:
ports:
- name: redis
port: 6379
targetPort: redis
selector:
name: redis
3 gitlab-postgresql.yaml
然后是数据库 Postgresql,对应的资源清单文件如下:(gitlab-postgresql.yaml)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gitlab-storageclass
provisioner: fuseim.pri/ifs
[root@k8s-master gitlab]# cat gitlab-postgresql.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: postgresql
namespace: kube-ops
labels:
name: postgresql
spec:
template:
metadata:
name: postgresql
labels:
name: postgresql
spec:
containers:
- name: postgresql
image: sameersbn/postgresql
imagePullPolicy: IfNotPresent
env:
- name: DB_USER
value: gitlab
- name: DB_PASS
value: passw0rd
- name: DB_NAME
value: gitlab_production
- name: DB_EXTENSION
value: pg_trgm
ports:
- name: postgres
containerPort: 5432
volumeMounts:
- mountPath: /var/lib/postgresql
name: data
livenessProbe:
exec:
command:
- pg_isready
- -h
- localhost
- -U
- postgres
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
exec:
command:
- pg_isready
- -h
- localhost
- -U
- postgres
initialDelaySeconds: 5
timeoutSeconds: 1
volumes:
- name: data
persistentVolumeClaim:
claimName: gitlab-postgresql-pvc
---
apiVersion: v1
kind: Service
metadata:
name: postgresql
namespace: kube-ops
labels:
name: postgresql
spec:
ports:
- name: postgres
port: 5432
targetPort: postgres
selector:
name: postgresql
4 gitlab.yaml
然后就是我们最核心的 Gitlab 的应用,对应的资源清单文件如下:(gitlab.yaml)
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: gitlab
namespace: kube-ops
labels:
name: gitlab
spec:
template:
metadata:
name: gitlab
labels:
name: gitlab
spec:
containers:
- name: gitlab
image: sameersbn/gitlab:12.1.6
imagePullPolicy: IfNotPresent
env:
- name: TZ
value: Asia/Shanghai
- name: GITLAB_TIMEZONE
value: Beijing
- name: GITLAB_SECRETS_DB_KEY_BASE
value: long-and-random-alpha-numeric-string
- name: GITLAB_SECRETS_SECRET_KEY_BASE
value: long-and-random-alpha-numeric-string
- name: GITLAB_SECRETS_OTP_KEY_BASE
value: long-and-random-alpha-numeric-string
- name: GITLAB_ROOT_PASSWORD
value: admin321
- name: GITLAB_ROOT_EMAIL
- name: GITLAB_HOST
value: gitlab.wangxu.com
- name: GITLAB_PORT
value: "80"
- name: GITLAB_SSH_PORT
value: "22"
- name: GITLAB_NOTIFY_ON_BROKEN_BUILDS
value: "true"
- name: GITLAB_NOTIFY_PUSHER
value: "false"
- name: GITLAB_BACKUP_SCHEDULE
value: daily
- name: GITLAB_BACKUP_TIME
value: 01:00
- name: DB_TYPE
value: postgres
- name: DB_HOST
value: postgresql
- name: DB_PORT
value: "5432"
- name: DB_USER
value: gitlab
- name: DB_PASS
value: passw0rd
- name: DB_NAME
value: gitlab_production
- name: REDIS_HOST
value: redis
- name: REDIS_PORT
value: "6379"
ports:
- name: http
containerPort: 80
- name: ssh
containerPort: 22
volumeMounts:
- mountPath: /home/git/data
name: data
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 180
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
timeoutSeconds: 1
volumes:
- name: data
persistentVolumeClaim:
claimName: gitlab-pvc
---
apiVersion: v1
kind: Service
metadata:
name: gitlab
namespace: kube-ops
labels:
name: gitlab
spec:
ports:
- name: http
port: 80
targetPort: http
- name: ssh
port: 22
targetPort: ssh
nodePort: 30022
type: NodePort
selector:
name: gitlab
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gitlab
namespace: kube-ops
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: gitlab.wangxu.com
http:
paths:
- backend:
serviceName: gitlab
servicePort: 80
#apiVersion: extensions/v1beta1
#kind: Ingress
#metadata:
# name: gitlab
# namespace: kube-ops
# annotations:
# kubernetes.io/ingress.class: traefik
#spec:
# rules:
# - host: gtlab.wangxu.com
# http:
# paths:
# - backend:
# serviceName: gitlab
# servicePort: http
我们这里应用数据都做数据持久化,添加 PV/PVC 或者 StorageClass。
5 查看Pod的部署状态
#等5分钟
# kubectl get pods -n kube-ops| grep -E "gitlab|^redis-|postgresql"
gitlab-687c54659f-bm69j 1/1 Running 1 38m
postgresql-8644bdb9c5-qww4v 1/1 Running 0 107m
redis-6fcdb86497-2bshn 1/1 Running 0 100m
[root@k8s-master gitlab]#
6 页面访问
我们可以通过 Ingress 中定义的域名gitlab.wangxu.com
(需要做 DNS 解析或者在本地 /etc/hosts 中添加映射)来访问 Portal:
10.6.76.23 gitlab.wangxu.com
10.6.76.24 gitlab.wangxu.com
使用部署指定的用户密码root/admin321 登陆后请修改
如果在部署中反复操作,导致登录密码错误之类,请kubectl delete -f gitlab.yaml,并删除gitlab持久化数据目录再重新部署
设置成中文
7 配置ssh下载上传
点击Create a project
创建一个新的项目
创建完成后,我们可以添加本地用户的一个SSH-KEY
,这样我们就可以通过 SSH 来拉取或者推送代码了。SSH 公钥通常包含在~/.ssh/id_rsa.pub
文件中,并以ssh-rsa
开头。如果没有的话可以使用ssh-keygen
命令来生成,id_rsa.pub
里面的内容就是我们需要的 SSH 公钥,然后添加到 Gitlab 中。
由于平时使用的 ssh 默认是 22 端口,现在如果用默认的 22 端口去连接,是没办法和 Gitlab 容器中的 22 端口进行映射的,因为我们只是通过 Service 的 22 端口进行了映射,要想通过节点去进行 ssh 链接就需要在节点上一个端口和容器内部的22端口进行绑定,所以这里我们可以通过 NodePort 去映射 Gitlab 容器内部的22端口,比如我们将环境变量设置为GITLAB_SSH_PORT=30022
,将 Gitlab 的 Service 也设置为 NodePort 类型:
前面配置文件已经添加
# kubectl -n kube-ops get svc|grep gitlab
gitlab NodePort 10.108.77.201 <none> 80:30419/TCP,22:30022/TCP 11m
我们在项目上面 Clone 的时候使用 ssh 就会带上端口号了:
配置公钥
[root@k8s-master test]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:XP5breGwn4wNJrp9sPfmOGj2MeSzZrr23N6pxbsQW2o root@k8s-master
The key's randomart image is:
+---[RSA 2048]----+
| |
| |
| . |
| . o |
| S . .. . |
| .+ *. |
| .+XEoo.|
| o*+B^+=o|
| o=+XXO&+o|
+----[SHA256]-----+
[root@k8s-master test]# cd ~/.ssh/
[root@k8s-master .ssh]# ls
id_rsa id_rsa.pub known_hosts
[root@k8s-master .ssh]# cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQ/mJYwmzINjNKfFL+o91WqQk27ciaEAuZkrpJCUPGzkyOf8m3YawqR6K2U5AV14lVpRhkHBsPuMrYRNH9h41LOExXEd9w+sni340hjg4bL+N7yPygct3yr3vnEurh8+CuICldkw4MZNPxLiPoCYJrXtNHQd7pMW0vbf6OyRigOjRgQ8VO0oKNUJ/WKeneKUNcOceyJaIh3lSUvkkQYUKyip9v9gVaDRv7BAdanhwmQu0LWiCdZSguYEwq0+DTIiHr1/GyaZB1bdEgQO4Fp8sryNHeFWfPvIWiKEt0mo/YuIF5DttahEIxrdjOoEjG6c09DTumf7r9fWlChNWjfBm1 root@k8s-master
[root@k8s-master .ssh]#
下载
[root@k8s-master test]# git clone ssh://git@gitlab.wangxu.com:30022/root/gitlab-demo.git
正克隆到 'gitlab-demo'...
The authenticity of host '[gitlab.wangxu.com]:30022 ([10.6.76.24]:30022)' can't be established.
ECDSA key fingerprint is SHA256:wFtUeSNMyj0tav79o7IynrNlt7wNV57ADLbGRh0ZXgg.
ECDSA key fingerprint is MD5:cb:3f:6f:78:15:7f:11:9e:7b:75:46:23:7d:b1:90:2b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[gitlab.wangxu.com]:30022,[10.6.76.24]:30022' (ECDSA) to the list of known hosts.
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
接收对象中: 100% (3/3), done.
[root@k8s-master test]# ls
gitlab-demo
[root@k8s-master test]#
上传
上传
[root@k8s-master gitlab-demo]# echo '123' >> test.index
[root@k8s-master gitlab-demo]# git add .
[root@k8s-master gitlab-demo]# git commit -m 'add test.index'
*** Please tell me who you are.
Run
git config --global user.email "you@example.com"
git config --global user.name "Your Name"
to set your account's default identity.
Omit --global to set the identity only in this repository.
fatal: unable to auto-detect email address (got 'root@k8s-master.(none)')
[root@k8s-master gitlab-demo]# echo $?
128
[root@k8s-master gitlab-demo]# git config --global user.
[root@k8s-master gitlab-demo]# git config --global user.name "wx"
[root@k8s-master gitlab-demo]# git commit -m 'add test.index'
[master 2570c66] add test.index
1 file changed, 1 insertion(+)
create mode 100644 test.index
[root@k8s-master gitlab-demo]# git push origin master
Counting objects: 4, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 273 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To ssh://git@gitlab.wangxu.com:30022/root/gitlab-demo.git
631fb39..2570c66 master -> master