warden创建容器的过程

转载

mob604756e7abe8 2012-09-27 18:13:00

文章标签 ubuntu ide 初始化文件系统文件名 文章分类 代码人生

rv = unshare(CLONE_NEWNS);

unshare这个调用，可以把挂载的文件系统设置成只在新的挂载命名空间（mount namespace）中可见。

execvp(argv[0], argv);

execvp()会从PATH 环境变量所指的目录中查找符合参数file 的文件名，找到后便执行该文件，然后将第二个参数argv传给该欲执行的文件。

shopt -s nullglob

设置shell环境变量nullglob的值为on，nullglob为on时对于通配符匹配时，若匹配不到时为空（相对应的为通配符本身）。

int stat(const char *restrict pathname, struct stat *restrict buf);

提供文件名字，获取文件对应属性。

build-essential软件包

作用是提供编译程序必须软件包的列表信息，也就是说编译程序有了这个软件包，它才知道头文件在哪，才知道库函数在哪，还会下载依赖的软件包，最后才组成一个开发环境。

工具debootstrap

可以用于在系统的某个目录中安装一套基本系统，这个基本系统除了一些配置项外，与ubuntu安装程序在安装的第一阶段安装的内容基本相同。这项功能有许多有趣的功能，例如，你可以从某个定制版本的ubuntu Live光盘上通过这个命令快速的在硬盘上安装ubuntu而不需要ubuntu的安装程序，也可以把创建在硬盘上的基本系统目录作为某些涉及系统安全性服务的chroot运行环境，通过chroot进入该目录并调试和运行一些可能修改系统配置的应用程序，作为定制小型系统模板等等。

aufs

一种文件格式，可以mount到目录，同时控制只读和读写。

overlayfs

另一种文件格式，在ubuntu 11.04后开始替代aufs作为官方livecd的文件格式。

cgroup的初始化

初始化的脚本为：

cgroup_path=/sys/fs/cgroup
mount -t tmpfs none $cgroup_path
for subsystem in cpu cpuacct devices memory
do
mkdir -p $cgroup_path/$subsystem

if ! grep -q "${cgroup_path}/$subsystem " /proc/mounts
then
mount -t cgroup -o $subsystem none $cgroup_path/$subsystem
fi
done

上面的命令执行完后，可以看下挂载的结果：

# grep '^cgroup' /proc/mounts
cgroups /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0

网络的控制

Every container is assigned a network interface which is one side of a

virtual ethernet pair created on the host. The other side of the virtual

ethernet pair is only visible on the host (from the root namespace).

The pair is configured to use IPs in a small and static subnet. Traffic

from and to the container can be forwarded using NAT. Additionally, all

traffic can be filtered and shaped as needed, using readily available

tools such as `iptables`.网络初始化

echo 1 > /proc/sys/net/ipv4/ip_forward

# iptables-save | grep -E 'warden|\*'
*nat
:warden-instance-16al6hojp15 - [0:0]
:warden-prerouting - [0:0]
-A PREROUTING -i eth0 -j warden-prerouting
-A OUTPUT -o lo -j warden-prerouting
-A warden-prerouting -j warden-instance-16al6hojp15
*mangle
*filter
:warden-default - [0:0]
:warden-dispatch - [0:0]
:warden-instance-16al6hojp15 - [0:0]
-A INPUT -i w-+ -j warden-dispatch
-A FORWARD -i w-+ -j warden-dispatch
-A warden-dispatch -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A warden-dispatch -i w-16al6hojp15-0 -g warden-instance-16al6hojp15
-A warden-dispatch -j DROP
-A warden-instance-16al6hojp15 -g warden-default

停止apparmor：

/etc/init.d/apparmor teardown

# quotaon(8) exits with non-zero status when quotas are ENABLED
if quotaon -p $CONTAINER_DEPOT_MOUNT_POINT_PATH > /dev/null
then
mount -o remount,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 $CONTAINER_DEPOT_MOUNT_POINT_PATH
quotacheck -ugmb -F vfsv0 $CONTAINER_DEPOT_MOUNT_POINT_PATH
quotaon $CONTAINER_DEPOT_MOUNT_POINT_PATH
fi

文件系统

Every container gets a private root filesystem. This filesystem is

created by stacking a read-only filesytem and a read-write filesystem.

This is implemented by using `aufs` on Ubuntu versions from 10.04 up to

11.10, and `overlayfs` on Ubuntu 12.04.

The read-only filesystem contains the minimal set of Ubuntu packages and

Warden-specific modifications common to all containers. The read-write

filesystem stores files overriding container-specific settings when

necessary. Because all writes are applied to the read-write filesystem,

containers can share the same read-only base filesystem.

The read-write filesystem is created by formatting a large sparse file.

Because the size of this file is fixed, the filesystem that it contains

cannot grow beyond this initial size.

创建一个容器

set -o nounset

set -o errexit

shopt -s nullglob

cp -r skeleton "${target}"

unshare -m "${target}"/setup.sh

"${target}"/setup.sh脚本

1. 首先生成一个配置文件

id=16al6hojp15

network_netmask=255.255.255.252

network_host_ip=10.254.0.17

network_host_iface=w-16al6hojp15-0

network_container_ip=10.254.0.18

network_container_iface=w-16al6hojp15-1

user_uid=10004

2. 调用setup_fs()

mkdir -p rootfs ${target}mount -n -t overlayfs -o rw,upperdir=rootfs,lowerdir=${1} none ${target}

3. 调用prepare.sh 删除不需要的文件，并创建必要的设备

4. 配置系统参数

# cat etc/hosts

127.0.0.1 16al6hojp15 localhost

10.254.0.17 host

10.254.0.18 container

# cat etc/hostname

16al6hojp15

# cat etc/network/interfaces

auto lo

iface lo inet loopback

auto w-16al6hojp15-1

iface w-16al6hojp15-1 inet static

gateway 10.254.0.17

address 10.254.0.18

netmask 255.255.255.252

cp /etc/resolv.conf ${target}/etc/

chroot并添加用户：useradd -mU -u ${user_uid} -s /bin/bash vcap

# Copy override directory 其实就是几个etc文件和一个sbin/warden-stop.sh文件

cp -r override/* ${target}/

chmod 700 ${target}/sbin/warden-*

5. 配置ssh

6. 配置mesg

# The `mesg` tool modifies permissions on stdin. Warden regularly passes a

# custom stdin, which makes `mesg` complain that stdin is not a tty. Instead of

# removing all occurances of `mesg`, we simply bind it to /bin/true.

chroot <<EOS

rm /usr/bin/mesg

ln -s /bin/true /usr/bin/mesg

EOS

7. 物理机上添加网卡作为容器的网关：

ifconfig ${network_host_iface} ${network_host_ip} netmask ${network_netmask}

并添加tc规则控制带宽：

qdisc tbf 8002: dev w-16al6hojp15-0 root refcnt 2 rate 8192bit burst 9b lat 24.4ms

qdisc ingress ffff: dev w-16al6hojp15-0 parent ffff:fff1 ----------------

8. 配置cgroup

# Add new group for every subsystem
for system_path in /sys/fs/cgroup/*
do
instance_path=$system_path/instance-$id

mkdir -p $instance_path

if [ $(basename $system_path) == "cpuset" ]
then
cat $system_path/cpuset.cpus > $instance_path/cpuset.cpus
cat $system_path/cpuset.mems > $instance_path/cpuset.mems
fi

echo 1 > $instance_path/cgroup.clone_children
echo $PID > $instance_path/tasks
done

echo ${PPID} >> ppid

ip link add name ${network_host_iface} type veth peer name ${network_container_iface}
ip link set ${network_host_iface} netns 1
ip link set ${network_container_iface} netns ${PID}