金庸武功之“北冥神功”--docker---nginx反向代理私有registry

原创

mgl0731 2016-11-07 18:11:25 博主文章分类：docker ©著作权

文章标签 服务器防火墙软件包 文章分类 服务器

©著作权归作者所有：来自51CTO博客作者mgl0731的原创作品，请联系作者获取转载授权，否则将追究法律责任

一。拓扑环境：

IP:192.168.93.202 docker registry服务器

IP:192.168.93.201 docker client服务器

关闭selinux及防火墙

修改/etc/hosts文件

192.168.93.202 docker.shengjing.com

hostnamectl set-hostname docker.shengjing.com (永久修改主机名）

安装依赖的软件包：

yum install gcc make pcre-devel pcre openssl-devel httpd-tools zlib-devel -y

生成根密钥：

先把
/etc/pki/CA/cacert.pem
/etc/pki/CA/index.txt
/etc/pki/CA/index.txt.attr
/etc/pki/CA/index.txt.old
/etc/pki/CA/serial
/etc/pki/CA/serial.old
删除掉!

cd /etc/pki/CA

(umask 077; openssl genrsa -out /etc/pki/CA/private/cakey.pem 4096)

[root@docker CA]# (umask 077; openssl genrsa -out /etc/pki/CA/private/cakey.pem 4096)

Generating RSA private key, 2048 bit long modulus

..................................................................................................................................+++

.......................................................+++

e is 65537 (0x10001)

生成根证书

openssl req -new -x509 -key /etc/pki/CA/private/cakey.pem -out /etc/pki/CA/cacert.pem -days 365

Generating RSA private key, 2048 bit long modulus

..................................................................................................................................+++

.......................................................+++

e is 65537 (0x10001)

[root@docker CA]# openssl req -new -x509 -key private/cakey.pem -out cacert.pem

You are about to be asked to enter information that will be incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

-----

Country Name (2 letter code) [XX]:CN

State or Province Name (full name) []:beijing

Locality Name (eg, city) [Default City]:beijing

Organization Name (eg, company) [Default Company Ltd]:sjwl

Organizational Unit Name (eg, section) []:sjwl

Common Name (eg, your name or your server's hostname) []:docker.shengjing.com

Email Address []:mengxl@shengjing360.com

会提示输入一些内容，因为是私有的，所以可以随便输入，最好记住能与后面保持一致,特别是"Common Name”。必须要和hostname显示的一致。mengxl@shengjing360.com要记住！
上面的自签证书cacert.pem应该生成在/etc/pki/CA下

[root@docker CA]# ls

cacert.pem certs crl newcerts private

为nginx web服务器生成ssl密钥

mkdir /usr/local/nginx/ssl

cd /usr/local/nginx/ssl

[root@docker ssl]#(umask 077; openssl genrsa -out /usr/local/nginx/ssl/nginx.key 2048)

Generating RSA private key, 2048 bit long modulus

.....................................................+++

....................................................................................................................+++

e is 65537 (0x10001)

查看nginx服务器的密钥

[root@docker ssl]# ls

nginx.key

为nginx生成证书签署请求

执行 openssl req -new -key /usr/local/nginx/ssl/nginx.key -out /usr/local/nginx/ssl/nginx.csr

You are about to be asked to enter information that will be incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

-----

Country Name (2 letter code) [XX]:CN

State or Province Name (full name) []:beijing

Locality Name (eg, city) [Default City]:beijing

Organization Name (eg, company) [Default Company Ltd]:sjwl

Organizational Unit Name (eg, section) []:sjwl

Common Name (eg, your name or your server's hostname) []:docker.shengjing.com

Email Address []:mengxl@shengjing360.com

Please enter the following 'extra' attributes

to be sent with your certificate request

A challenge password []:

An optional company name []:

同样会提示输入一些内容，Commone Name一定要是你要授予证书的服务器域名或主机名，challenge password不填。

私有CA根据请求来签发证书

[root@docker ssl]# touch /etc/pki/CA/index.txt

[root@docker ssl]# touch /etc/pki/CA/serial

[root@docker ssl]# echo 00 > /etc/pki/CA/serial

执行

[root@docker ssl]# openssl ca -in /usr/local/nginx/ssl/nginx.csr -out /etc/pki/CA/certs/nginx.crt -days 365

Using configuration from /etc/pki/tls/openssl.cnf

Check that the request matches the signature

Signature ok

Certificate Details:

Serial Number: 0 (0x0)

Validity

Not Before: Nov 7 16:28:22 2016 GMT

Not After : Nov 7 16:28:22 2017 GMT

Subject:

countryName = CN

stateOrProvinceName = beijing

organizationName = sjwl

organizationalUnitName = sjwl

commonName = docker.shengjing.com

emailAddress = mengxl@shengjing360.com

X509v3 extensions:

X509v3 Basic Constraints:

CA:FALSE

Netscape Comment:

OpenSSL Generated Certificate

X509v3 Subject Key Identifier:

04:91:10:DD:9E:37:81:66:5E:66:E4:CE:EB:02:E0:D3:27:FC:F7:7B

X509v3 Authority Key Identifier:

keyid:C8:F9:00:19:C0:61:7E:71:B8:16:FD:08:43:AD:82:F7:9E:BC:20:91

Certificate is to be certified until Nov 7 16:28:22 2017 GMT (365 days)

Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y

Write out database with 1 new entries

Data Base Updated

同样会提示输入一些内容，选择y就可以了!

查看nginx的证书

[root@docker ssl]# cd /etc/pki/CA/certs/

[root@docker certs]# ls

nginx.crt

cp /etc/pki/CA/certs/nginx.crt /usr/local/nginx/ssl/

查看证书中的信息：

~]# openssl x509 -in /etc/pki/CA/certs/nginx.crt -noout -serial -subject

安装,配置,运行nginx

(1) 添加组和用户

[root@docker ssl]# groupadd www -g 58

[root@docker ssl]# useradd -u 58 -g www www

cd /usr/local/src

wget http://nginx.org/download/nginx-1.11.2.tar.gz

tar -zxvf nginx-1.11.2.tar.gz

cd nginx-1.11.2

[root@docker nginx-1.11.2]# ./configure --user=www --group=www --prefix=/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module --with-pcre --with-http_addition_module --with-http_realip_module --with-http_flv_module

[root@docker nginx-1.11.2]# make && make install

vi /usr/local/nginx/conf/nginx.conf

user www;

worker_processes 4;

#error_log logs/error.log;

#error_log logs/error.log notice;

#error_log logs/error.log info;

pid logs/nginx.pid;

events {

worker_connections 4096;

}

http {

include mime.types;

default_type application/octet-stream;

#log_format main '$remote_addr - $remote_user [$time_local] "$request" '

# '$status $body_bytes_sent "$http_referer" '

# '"$http_user_agent" "$http_x_forwarded_for"';

#access_log logs/access.log main;

sendfile on;

#tcp_nopush on;

#keepalive_timeout 0;

keepalive_timeout 65;

#gzip on;

upstream registry {

server 192.168.93.202:5000;

}

server {

listen 80;

server_name localhost;

#charset koi8-r;

#access_log logs/host.access.log main;

location / {

root html;

index index.html index.htm;

}

#error_page 404 /404.html;

# redirect server error pages to the static page /50x.html

error_page 500 502 503 504 /50x.html;

location = /50x.html {

root html;

}

server {

listen 443 ;

server_name docker.shengjing.com;

add_header Docker-Distribution-Api-Version registry/2.0 always;

ssl on;

ssl_certificate /usr/local/nginx/ssl/nginx.crt;

ssl_certificate_key /usr/local/nginx/ssl/nginx.key;

ssl_session_cache shared:SSL:1m;

ssl_session_timeout 5m;

# ssl_ciphers HIGH:!aNULL:!MD5;

# ssl_prefer_server_ciphers on;

location / {

auth_basic "registry";

auth_basic_user_file /usr/local/nginx/conf/.htpasswd; #htpasswd的位置

root html;

index index.html index.htm;

#proxy_pass http://registry;

#proxy_set_header Host $http_host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header Authorization "";

client_body_buffer_size 128k;

proxy_connect_timeout 90;

proxy_send_timeout 90;

proxy_read_timeout 90;

proxy_buffer_size 8k;

proxy_buffers 4 32k;

proxy_busy_buffers_size 64k; #如果系统很忙的时候可以申请更大的proxy_buffers 官方推荐*2

proxy_temp_file_write_size 64k; #proxy缓存临时文件的大小

}

location /_ping {

auth_basic off;

proxy_pass http://registry;

}

location /v1/_ping {

auth_basic off;

proxy_pass http://registry;

}

验证配置：

/usr/local/nginx/sbin/nginx -t

启动nginx:

[root@docker conf]# /usr/local/nginx/sbin/nginx

[root@docker conf]# ss -ntpl

State Recv-Q Send-Q Local Address:Port Peer Address:Port

LISTEN 0 128 *:22 *:* users:(("sshd",pid=841,fd=3))

LISTEN 0 100 127.0.0.1:25 *:* users:(("master",pid=1408,fd=13))

LISTEN 0 128 *:443 *:* users:(("nginx",pid=4796,fd=6),("nginx",pid=4795,fd=6),("nginx",pid=4794,fd=6),("nginx",pid=4793,fd=6),("nginx",pid=4792,fd=6))

LISTEN 0 128 :::22 :::* users:(("sshd",pid=841,fd=4))

LISTEN 0 100 ::1:25 :::* users:(("master",pid=1408,fd=14))

[root@docker conf]# ps -ef | grep -i "nginx"

root 4792 1 0 12:07 ? 00:00:00 nginx: master process /usr/local/nginx/sbin/nginx

www 4793 4792 0 12:07 ? 00:00:00 nginx: worker process

www 4794 4792 0 12:07 ? 00:00:00 nginx: worker process

www 4795 4792 0 12:07 ? 00:00:00 nginx: worker process

www 4796 4792 0 12:07 ? 00:00:00 nginx: worker process

root 4804 1994 0 12:08 pts/0 00:00:00 grep --color=auto -i nginx

生成htpasswd

[root@localhost]# htpasswd -cb /usr/local/nginx/conf/.htpasswd admin admin

浏览器中：https://docker.shengjing.com

输入用户：admin

密码： admin

出现nginx测试页面

当把这条#proxy_pass http://registry;的注释#去掉

页面就报502，因为后端代理docker registry还没有启动

至此证明https认证完成

###############################################################################

搭建docker registry V2

[root@localhost ~]# systemctl stop docker

[root@localhost ~]# vim /etc/sysconfig/docker

修改后：

OPTIONS='--selinux-enabled --insecure-registry dockertest.shengjing.com'

添加--insecure-registry docker.shengjing.com 红色的是私有仓库的域名，更前面生成密钥时候的域名对应

配置密钥文件

mkdir -p /etc/docker/certs.d/hc.docker.io

cp /etc/pki/CA/cacert.pem /etc/docker/certs.d//hc.docker.io/ca-certificates.crt

cat /etc/pki/CA/cacert.pem >> /etc/pki/tls/certs/ca-bundle.crt

[root@localhost ~]#systemctl restart docker

[root@localhost ~]#vim config.yml

version: 0.1

log:

level: debug

formatter: text

fields:

service: registry

environment: staging

storage:

delete:

enabled: true

cache:

layerinfo: inmemory

filesystem:

rootdirectory: /var/lib/registry

http:

addr: :5000

secret: admin

[root@localhost ~]#mkdir data

docker run -d -p 5000:5000 --restart=always --name registry -v `pwd`/config.yml:/etc/docker/registry/config.yml -v `pwd`/data:/var/lib/registry registry:2

这里注意如果运行时报如下错误：

[root@docker ~]# docker run -d -p 5000:5000 --restart=always --name registry -v `pwd`/config.yml:/etc/docker/registry/config.yml -v `pwd`/data:/var/lib/registry registry:2

Unable to find p_w_picpath 'registry:2' locally

Trying to pull repository docker.io/library/registry ...

Get https://registry-1.docker.io/v2/library/registry/manifests/2: net/http: TLS handshake timeout

docker: Get https://registry-1.docker.io/v2/library/registry/manifests/2: net/http: TLS handshake timeout.

See '/usr/bin/docker-current run --help'.

没别的原因，就是网络问题，因为docker hub在国外，墙很厚啊，呵呵！多试几次就行了

[root@docker ~]# docker p_w_picpaths

REPOSITORY TAG IMAGE ID CREATED SIZE

docker.io/registry 2 c9bd19d022f6 2 weeks ago 33.27 MB

验证

[root@localhost ~]# curl https://admin:admin@dockertest.xxxx.com/v2/

结果返回{}

[root@localhost ~]docker login https://dockertest.xxxx.com

Username: admin

Password: #admin

Email: abcd@qq.com

WARNING: login credentials saved in /root/.docker/config.json

验证push镜像到私有仓库

在登录的前题下，本机验证

[root@localhost ~]# docker pull busybox

docker pull busybox

Using default tag: latest

Trying to pull repository docker.io/library/busybox ...

latest: Pulling from docker.io/library/busybox

56bec22e3559: Pull complete

Digest: sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912

Status: Downloaded newer p_w_picpath for docker.io/busybox:latest

[root@docker ~]# docker p_w_picpaths

REPOSITORY TAG IMAGE ID CREATED SIZE

docker.io/registry 2 c9bd19d022f6 2 weeks ago 33.27 MB

docker.io/busybox latest e02e811dd08f 4 weeks ago 1.093 MB

[root@docker ~]# docker tag e02e811dd08f docker.shengjing.com/busybox:v1

[root@docker ~]# docker p_w_picpaths

REPOSITORY TAG IMAGE ID CREATED SIZE

docker.io/registry 2 c9bd19d022f6 2 weeks ago 33.27 MB

docker.io/busybox latest e02e811dd08f 4 weeks ago 1.093 MB

docker.shengjing.com/busybox v1 e02e811dd08f 4 weeks ago 1.093 MB

[root@docker ~]# docker push docker.shengjing.com/busybox:v1

The push refers to a repository [docker.shengjing.com/busybox]

e88b3f82283b: Pushed

v1: digest: sha256:9393222c6789842b16bcf7306b6eb4b486d81a48d3b8b8f206589b5d1d5a6101 size: 505

[root@docker ~]# curl https://admin:admin@docker.shengjing.com/v2/_catalog

{"repositories":["busybox"]}

在client端测试

配置密钥

[root@localhost ~]#scp /etc/pki/CA/cacert.pem root@192.168.93.201:/root/

在client上（192.168.93.201）

vi /etc/sysconfig/docker

OPTIONS='--selinux-enabled --insecure-registry docker.shengjing.com'

mkdir -pv /etc/docker/certs.d/docker.shengjing.com

cp /root/cacert.pem /etc/docker/certs.d/docker.shengjing.com/

cat /etc/docker/certs.d/docker.shengjing.com/cacert.pem > /etc/pki/tls/certs/ca-bundle.crt

vi /etc/hosts

192.168.93.202 docker.shengjing.com

重启下服务器

第一次我登陆时报错，如下：

[root@localhost ~]# docker login https://docker.shengjing.com
Username: admin
Password:
Email:
Error response from daemon: invalid registry endpoint https://docker.shengjing.com/v0/: unable to ping registry endpoint https://docker.shengjing.com/v0/
v2 ping attempt failed with error: Get https://docker.shengjing.com/v2/: x509: certificate signed by unknown authority
v1 ping attempt failed with error: Get https://docker.shengjing.com/v1/_ping: x509: certificate signed by unknown authority. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry docker.shengjing.com` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/docker.shengjing.com/ca.crt