k8s笔记2(Harbor,ingress)

原创

山西空管技术支持 2022-10-13 15:34:56 博主文章分类：k8s ©著作权

文章标签 sed 用户名 vim 文章分类 运维

©著作权归作者所有：来自51CTO博客作者山西空管技术支持的原创作品，请联系作者获取转载授权，否则将追究法律责任

k8s笔记2(Harbor,ingress)_用户名

the overall architecture of the Harbor registry

chartmuseum/storage:支持多个存储作为registry and chart museum的后端存储进行数据持久性。Alibaba Cloud OSS Storage ,etcd (etcd.go) etcd（etcd. go）,Local filesystem (local.go) ,Minio (amazon.go, using custom endpoint and us-east-1),Netease Cloud NOS Storage (netease.go)......
k-v存储：Redis提供数据高速缓存功能，支持为作业服务临时持久化作业元数据。
Database：存储Harbor模型的相关元数据，如项目、用户、角色、复制策略、标签保留策略、扫描仪、图表和图像。采用PostgreSQL。
Proxy：由Nginx Server组成的反向代理，提供API路由功能。Harbor的组件，如核心、注册表、门户网站和令牌服务等，都在这个反向代理的背后。代理将来自浏览器和Docker客户端的请求转发到各种后端服务。
Core：Harbor的核心服务，主要提供以下功能:

API服务器：接受REST API请求并响应这些请求的HTTP服务器依赖于其子模块，例如“身份验证和授权”、“中间件”和“API处理程序”。Authentication & Authorization;令牌服务旨在根据项目的用户角色为每个docker推/拉命令发出令牌。如果从Docker客户端发送的请求中没有令牌，Registry将把请求重定向到令牌服务。
Config Manager：涵盖所有系统配置的管理，如身份验证类型设置、电子邮件设置和证书等。
Project Management:管理项目的基本数据和相应的元数据，创建这些数据是为了隔离被管理的工件。
配额管理器：管理项目的配额设置并在发生新推送时执行配额验证。
Chart Controller：将图表相关请求代理到后端chartmuseum，并提供多个扩展以改善图表管理体验。
Retention Manager:管理标签保留策略并执行和监控标签保留过程。
Content Trust: 在后端Notary提供的信任能力上添加扩展，支持内容信任流程顺畅，目前仅支持容器镜像签名。
Replication控制器：管理Replication策略和registry 适配器，触发和监控并发复制过程。Docker Hub、Huawei SWR、Ali ACR
Scan管理器：管理由不同提供商适配的多个配置的扫描仪，还为指定的工件提供扫描摘要和报告。
Notification管理器（webhook）：在Harbor中配置的一种机制，以便可以将Harbor中的工件状态更改填充到Harbor中配置的Webhook端点。
OCI工件管理器：管理整个Harbor注册表中所有OCI工件生命周期的核心组件。
Registry Driver：作为registry client SDK 实现，与底层registry 进行通信(docker distribution at this moment)。

Job Service: 通用作业执行队列服务，让其他组件/服务通过简单的restful API同时提交运行异步任务的请求。
Log collector: Log collector, responsible for collecting logs of other modules into a single place.
GC Controller：管理在线GC计划设置并启动和跟踪GC进度。
Chart Museum:提供图表管理和访问API的第3方图表存储库服务器。ChartMuseum is an open-source Helm Chart Repository server。
Docker Registry: 第三方注册表服务器，负责存储Docker镜像和处理Docker推/拉命令。
Notary:第3方内容信任服务器，负责安全地发布和验证内容。This repository comprises of a server and a client for running and interacting with trusted collections。发布者可以使用高度安全的密钥离线签署他们的内容。一旦发布者准备好提供内容，他们就可以将他们签名的可信集合推送到notary服务器。消费者通过安全通道获得发布者的公钥，然后可以与任何公证服务器或（不安全的）镜像通信，仅依赖发布者的密钥来确定接收内容的有效性和完整性。Docker Content Trust (DCT):目前仅支持容器镜像签名。

---

***helm upgrade 仓库内容不会丢失，但是uninstall后再install就恢复初始，数据丢失了。

***1年后证书过期：atc-harbor-notary-signer:7899: connection error: desc = "transport: x509: certificate has expired，notary-signer是READY，但是notary-server未READY，按https://github.com/goharbor/harbor/issues/13850操作不行，notary-signer和notary-server都是未READY，其它所有POD重启都READY。

atc-harbor-core失败时， docker push会提示503 Service Temporarily Unavailable

1、安装官方文档通过Helm部署Harbor（Harbor docs | Deploying Harbor with High Availability via Helm (goharbor.io)）

----->nodePort方式暴露服务；

----->按提示填写commonName

----->持久化存储：storageClass: "rook-ceph-block"

----->用户名admin/密码Harbor12345(！浏览器缓存功能导致账号登录失败，禁用缓存或换个浏览器正常)

distribution/token.md at main · distribution/distribution · GitHub

# vim values.yaml
expose:
  # Set how to expose the service. Set the type as "ingress", "clusterIP", "nodePort" or "loadBalancer"
  # and fill the information in the corresponding section
  type: nodePort
  tls:
    certSource: auto
    auto:
      # The common name used to generate the certificate, it's necessary
      # when the type isn't "ingress"
      commonName: "mizy"
  persistentVolumeClaim:
    registry:
      # Use the existing PVC which must be created manually before bound,
      # and specify the "subPath" if the PVC is shared with other components
      existingClaim: ""
      # Specify the "storageClass" used to provision the volume. Or the default
      # StorageClass will be used (the default).
      # Set it to "-" to disable dynamic provisioning
      storageClass: "rook-ceph-block"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 200Gi
      annotations: {}

2、将dockerhub上的镜像迁移到Harbor私有镜像仓库(镜像迁移：如何将dockerhub上的镜像迁移到Harbor私有镜像仓库中？_51CTO博客_docker 镜像迁移)

3、docker push（docker push的前提条件是要有Docker hub的账号）报错denied的原因：push的时候需要带上自己Docker Hub的用户名，用户名/镜像名:版本号（不指定版本号，默认latest）这样的形式，可以为镜像增加一个TAG。

# docker push tomcat
errors:
denied: requested access to the resource is denied
unauthorized: authentication required
# docker tag tomcat mizy/tomcat
# docker push mizy/tomcat
latest: digest: sha256:1c18f0ab6cb76865cacfe1ca0127ac98847a04ad6d4ca84496a5e5ac5b632955 size: 1788

4、关于证书:Harbor docs | Troubleshooting Harbor Installation (goharbor.io)

1）helm的/harbor/cert文件夹下

[root@k8s-master01 cert]# ls
tls.crt  tls.key

2）NodePort的服务harbor

[root@k8s-master01 ~]# k get svc harbor -n harbor
NAME     TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                                     AGE
harbor   NodePort   10.16.61.244   <none>        80:30002/TCP,443:30003/TCP,4443:30004/TCP   19h

3）服务的endpoint对应的pod,找到secret

# vim values.yaml
expose:
  # Set how to expose the service. Set the type as "ingress", "clusterIP", "nodePort" or "loadBalancer"
  # and fill the information in the corresponding section
  type: nodePort
  tls:
    certSource: auto
    auto:
      # The common name used to generate the certificate, it's necessary
      # when the type isn't "ingress"
      commonName: "mizy"
  persistentVolumeClaim:
    registry:
      # Use the existing PVC which must be created manually before bound,
      # and specify the "subPath" if the PVC is shared with other components
      existingClaim: ""
      # Specify the "storageClass" used to provision the volume. Or the default
      # StorageClass will be used (the default).
      # Set it to "-" to disable dynamic provisioning
      storageClass: "rook-ceph-block"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 200Gi
      annotations: {}

4）secret里是ca.crt和tls.crt，configMap挂载/etc/nginx/cert/tls.crt（tls.key）

# k edit secret mi-release-harbor-nginx -n harbor
data:
  ca.crt: LS0tLS1CR......
  tls.crt: LS0tLS1CRU......
# k edit cm mi-release-harbor-nginx -n harbor 
      server {
        listen 4443 ssl;
        listen [::]:4443 ssl;
        server_tokens off;
        # ssl
        ssl_certificate /etc/nginx/cert/tls.crt;
        ssl_certificate_key /etc/nginx/cert/tls.key;

5）进入nginx容器，查看证书，nginx容器里的tls.crt和helm的/harbor/cert文件夹下的不一样。

# k exec -ti mi-release-harbor-nginx-6c8fdb78b7-9h247 -n harbor -- bash
nginx [ / ]$ cd /etc/nginx/cert/
nginx [ /etc/nginx/cert ]$ ls
ca.crt  tls.crt  tls.key

5、docker登录harbor私有仓库

docker私有仓库 harbor私仓 harbor daemon.json配置 docker配置

Docker如果需要从非SSL源管理镜像，也就是说如果Harbor采用http协议的方式安装，需要配置Docker配置文件的insecury-registry参数。

[root@k8s-master01 cert]# ls
tls.crt  tls.key

OK!安装harbor：（type: nodePort）（tls:enabled: false）（externalURL: http://192.168.31.211:30002）

[root@k8s-master01 ~]# k get svc harbor -n harbor
NAME     TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                                     AGE
harbor   NodePort   10.16.61.244   <none>        80:30002/TCP,443:30003/TCP,4443:30004/TCP   19h

docker登录密码存放在config.json

# k edit svc harbor -n harbor
  selector:
    app: harbor
    component: nginx
    release: mi-release
  type: NodePort
[root@k8s-master01 ~]# k get ep  harbor -n harbor
NAME     ENDPOINTS                                               AGE
harbor   172.25.92.80:4443,172.25.92.80:8443,172.25.92.80:8080   19h
[root@k8s-master01 ~]# k edit ep  harbor -n harbor
    targetRef:
      kind: Pod
      name: mi-release-harbor-nginx-6c8fdb78b7-9h247
# k edit pod mi-release-harbor-nginx-6c8fdb78b7-9h247 -n harbor
  volumes:
  - configMap:
      defaultMode: 420
      name: mi-release-harbor-nginx
    name: config
  - name: certificate
    secret:
      defaultMode: 420
      secretName: mi-release-harbor-nginx

Harbor采用https的方式交换数据，Docker客户端处需要配置签署 harbor 证书的 CA 证书。Docker登录Harbor私有仓库 - 爱码网 (likecs.com)

Verify repository client with certificates | Docker Documentation

x509: certificate signed by unknown authority (harbor)_xiunai78的博客

在Docker客户端服务器上创建指定目录：/etc/docker/certs.d/[IP地址或域名]（Harbor地址，harbor.cfg文件中的hostname项）
# mkdir -p  /etc/docker/certs.d/192.168.31.211:30003
拷贝CA证书到上述目录中，------------->nginx容器里的tls.crt
docker login [IP地址或域名](Harbor地址,harbor.cfg文件中的hostname项)
//根据提示分别输入用户名和密码
最终docker目录结构：
/etc/docker/certs.d/
    └── yourdomain.com:port
       ├── yourdomain.com.cert  <-- Server certificate signed by CA
       ├── yourdomain.com.key   <-- Server key signed by CA
       └── ca.crt               <-- Certificate authority that signed the registry certificate

测试certificate记录，提示需要CA认证：Server certificate signed by CA

客户端验证证书的合法性，如果验证通过才会进行后续通信，否则根据错误情况不同做出提示和操作，合法性验证内容包括如下:

证书链的可信性 trusted certificate path;
证书是否吊销 revocation，有两类方式 - 离线 CRL 与在线 OCSP，不同的客户端行为会不同;
有效期 expiry date，证书是否在有效时间范围;
域名 domain，核查证书域名是否与当前的访问域名匹配;

由 PKI 体系 的内容可知，对端发来的证书签名是 CA 私钥加密的，接收到证书后，先读取证书中的相关的明文信息，采用相同的散列函数计算得到信息摘要，然后利用对应 CA 的公钥解密签名数据，对比证书的信息摘要，如果一致，则可以确认证书的合法性；然后去查询证书的吊销情况等。

# docker login https://core.harbor.domain:30003
Username: admin
Password: 
Error response from daemon: Get "https://core.harbor.domain:30003/v2/": x509: certificate is valid for mizy, not core.harbor.domain
# vim /etc/hosts
192.168.31.211 mizy
# docker login https://mizy:30003
Username: admin
Password: 
?Error response from daemon: Get "https://mizy:30003/v2/": x509: certificate signed by unknown authority
是不是/etc/docker/daemon.json里需要改成
"insecure-registries": [
    "mizy:30003"
--->不行！
[root@mizy ~]# systemctl daemon-reload
[root@mizy ~]# systemctl restart docker
[root@mizy ~]# docker login https://mizy:30003
Username: admin
Password: 
Error response from daemon: Get "https://mizy:30003/v2/": Get "https://core.harbor.domain/service/token?account=admin&client_id=docker&offline_token=true&service=harbor-registry": dial tcp: lookup core.harbor.domain on 114.114.114.114:53: no such host

distribution/token.md at main · distribution/distribution · GitHub

RFC 6750: The OAuth 2.0 Authorization Framework: Bearer Token Usage (rfc-editor.org)

Harbor仓库配置https访问 - flytoyou - 博客园 (cnblogs.com)

docker login https://192.168.75.100
x509: cannot validate certificate for 192.168.75.100 because it doesn't contain any IP SANs
排查步骤：
检查harbor.yml文件中hostname变量的值是否跟生成证书使用的一致

Containerd ctr、crictl、nerdctl 客户端命令

http://t.csdn.cn/TQyTy

7、helm安装ingress-nginx（controller的IP是192.168.31.218）

# vim values.yaml
  # Optionally change this to ClusterFirstWithHostNet in case you have 'hostNetwork: true'.
  # By default, while using host network, name resolution uses the host's DNS. If you wish nginx-controller
  # to keep resolving names inside the k8s network, use ClusterFirstWithHostNet.
  # Node节点的/etc/hosts在Pod里也是生效的。
  dnsPolicy: ClusterFirstWithHostNet
  # Required for use with CNI based kubernetes installations (such as ones set up by kubeadm),
  # since CNI and hostport don't mix yet. Can be deprecated once https://github.com/kubernetes/kubernetes/issues/23920
  # is merged
  hostNetwork: true
[root@k8s-master01 ingress-nginx]# k -n ingress get pod -owide
NAME                                 READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
atc-ingress-nginx-controller-7bgrw   1/1     Running   0          19d   192.168.31.218   k8s-node05   <none>           <none>
# k -n ingress get pod atc-ingress-nginx-controller-7bgrw -oyaml
  dnsPolicy: ClusterFirstWithHostNet
  enableServiceLinks: true
  hostNetwork: true
  nodeName: k8s-node05
  nodeSelector:
    ingress: "true"
    kubernetes.io/os: linux
# helm upgrade ingress-nginx -n ingress-nginx .     //helm install 按照视频修改value.yaml后uprade
NAME: ingress-nginx
LAST DEPLOYED: Tue Oct 18 14:30:26 2022
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'
An example Ingress that makes use of the controller:
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: 
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls
[root@k8s-master01 ingress-nginx]# netstat -lntp |grep 80           //直接在宿主机上监听端口
tcp        0      0 192.168.31.211:2380     0.0.0.0:*               LISTEN      1365/etcd           
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      31097/nginx: master 
tcp6       0      0 :::80                   :::*                    LISTEN      31097/nginx: master
#curl -H "Host:ingress.test.com" 192.168.31.211

8、WINDOWS系统C:\Windows\System32\drivers\etc\hosts添加：192.168.31.211 ingress.test.com，IE测试https://ingress.test.com/正常。

# vim /etc/hosts
192.168.31.211 ingress.test.com
----创建nginx，svc-nginx(ClusterIP)暴露80端口，创建ingress（web-ingress.yaml）----------
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-test
  namespace: ingress-nginx
spec:
  ingressClassName: nginx
  rules:
  - host: ingress.test.com
    http:
      paths:
      - backend:
          service:
            name: nginx
            port: 
              number: 80
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - ingress.test.com
    secretName: nginx-test-tls
 ----------secretName: nginx-test-tls---------------------
 openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=ingress.test.com" 
 kubectl -n ingress-nginx create secret tls nginx-test-tls --key=tls.key --cert=tls.crt
 # curl https://ingress.test.com -k     //测试OK

假如创建ingress报错（容器云平台No.8~kubernetes负载均衡之ingress-nginx - 空壳先生 - 博客园 (cnblogs.com)）

# kubectl  apply -f web-ingress.yaml
x509: certificate is valid for k8s-master002, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not ingress-nginx-controller-admission.ingress-nginx.svc
-----------解决办法1、把Webhook删了-----------------
# kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission