1、初识分布式文件系统

在生产环境中,后端存储服务器服务器是一个很重要的环节。如果一旦前段服务器需要大量进行IO操作,会对后端存储的文件系统造成很大的压力。一旦后端的存储文件系统down机,那会导致整个生产环境奔溃。因为在生产环境中,后端的存储文件系统会成为一个单点故障。而为了避免因为单点故障导致整个生产环境无法正常使用,所有大家都会采用分布式文件系统,以避免因为单点故障导致生产环境奔溃。

  1. mogilefs组成部分

MogileFS3个部分组成:

(1) server:主要包括mogilefsd和mogstored两个应用程序。mogilefsd实现的是tracker,它通过数据库来保存元数据信息,包括站点domain、class、host等;mogstored是存储节点(store node),它其实是个WebDAV服务,默认监听在7500端口,接受客户端的文件存储请求。在MogileFS安装完后,要运行mogadm工具将所有的store node注册到mogilefsd的数据库里,mogilefsd会对这些节点进行管理和监控。

(2) utils(工具集):主要是MogileFS的一些管理工具,例如mogadm等。

(3) 客户端API:MogileFS的客户端API很多,例如Perl、PHP、Java、Python等,用这个模块可以编写客户端程序,实现文件的备份管理功能等。

存储主机(节点)

这个是 MogileFS 存储文件存放在这些机器上,也是 mogstored 节点,也叫 Storage Server,一台存储主要都要启动一个 mogstored 服务.扩容就是增加这些机器.

设备(device)

一个存储节点,以就是上面的主机,可以有多个 device, 就是用来存放文件的目录(例如挂载的目录),每个设备都有一个设备id,需要在 mogstored 的配置文件中的 docroot 配置的项目 指定的目录下面创建相应的设备的目录,目录名为 $docroot/dev$id,设备是不能删除的.只能将其设备的状态的值置为dead,当一个设备 dead 之后,就真的 dead了,里面的数据也无法恢复了,且这个dead了的设备的 id 也不能再用.

3、分布式文件系统需要的程序包

MogileFS-Server-2.46-2.el6.noarch.rpm

MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm

MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm

MogileFS-Utils-2.19-1.el6.noarch.rpm

Perlbal-1.78-1.el6.noarch.rpm

Perlbal-doc-1.78-1.el6.noarch.rpm

perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm

perl-IO-AIO-3.71-2.el6.x86_64

perl-MogileFS-Client-1.14-1.el6.noarch.rpm

perl-Net-Netmask-1.9015-8.el6.noarch.rpm

perl-Perlbal-1.78-1.el6.noarch.rpm

  1. 接着去安装所提供的所有的软件包

[root@mogilefs1 mogilefs]# yum install perl-IO-AIO perl-Net-Netmask perl-Sys-Syslog

[root@mogilefs1 mogilefs]# yum install MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm  

perl-MogileFS-Client-1.14-1.el6.noarch.rpm MogileFS-Server-2.46-2.el6.noarch.rpm MogileFS-Utils-2.19-1.el6.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm  perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm

或者在安装包的目录下把所有rpm一次性安装

[root@mogilefs1 mogilefs]# yum install *.rpm

  1. 接着去mysql数据库上进行授权

[root@mysql1 ~]# mysql

mysql> grant all on *.* to 'root'@'172.16.17.%' identified by 'xiaopeng';

Query OK, 0 rows affected (0.00 sec)

mysql> create database mogilefs;

Query OK, 1 row affected (0.03 sec)

mysql> grant all on mogilefs.* to 'moguser'@'172.16.17.%' identified by 'mogpass';

Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;

Query OK, 0 rows affected (0.00 sec)

[root@mysql1 ~]#

6、数据库授权完成之后,接着去mogilefs文件系统的主控制台上,对数据库进行初始化。(建议此命令慎用,如果mogilefs文件系统在运行的时候,小心使用。)

[root@mogilefs1 mogilefs]# mogdbsetup --dbhost=172.16.17.144 --dbrootpass='xiaopeng' --dbuser='moguser' --dbpass='mogpass'

This will attempt to setup or upgrade your MogileFS database.

It won't destroy existing data.

Run with --help for more information.  Run with --yes to shut up these prompts.

Continue? [N/y]: y

完成之后,可以去数据库确认初始化情况

[root@mogilefs1 mogilefs]#

mysql> use mogilefs

Database changed

mysql> show tables;

+----------------------+

| Tables_in_mogilefs   |

+----------------------+

  1. 数据库准备完成之后,接着去修改控制台文件

[root@mogilefs1 ~]# vim /etc/mogilefs/mogilefsd.conf

daemonize = 1

pidfile = /var/run/mogilefsd/mogilefsd.pid

db_dsn = DBI:mysql:mogilefs:host=172.16.100.68

db_user = moguser

db_pass = mogpass

listen = 0.0.0.0:7001

conf_port = 7001

query_jobs = 5

delete_jobs = 1

replicate_jobs = 2

reaper_jobs = 1

  1. 主配置文件修改完成之后,接着所有的存储节点都需要进行一下的配置

[root@mogilefs1 ~]# mkdir -pv /data/mogilefs/dev1

[root@mogilefs1 ~]# chown -R mogilefs.mogilefs /data/mogilefs/     

[root@mogilefs1 ~]# vim /etc/mogilefs/mogstored.conf

maxconns = 10000

httplisten = 0.0.0.0:7500

mgmtlisten = 0.0.0.0:7501

docroot = /data/mogilefs/

  1. 对于mogstord的启动文件(所有的节点都的mogstored的启动脚本文件需要加以下的几项)

为脚本添加

pidfile=/var/run/mogilefsd/mogstored.pid

echo $(pidof mogstored) > ${pidfile}

rm -f ${pidfile}

  1. 所有的配置修改完成之后,接着去启动服务器,进行配置资源

[root@mysql1 ~]# service mogstored start

Starting mogstored                                         [  OK  ]

[root@mysql1 ~]# ss -tanlp

[root@mogilefs1 init.d]# service mogilefsd start

Starting mogilefsd                                         [  OK  ]

[root@mogilefs1 ~]# mogadm check

Checking trackers...

  127.0.0.1:7001 ... OK

Checking hosts...

No devices found on tracker(s).

[root@mogilefs1 ~]# clear

追踪节点所能适应的管理命令

[root@mogilefs1 ~]# mogadm --tracker=172.16.17.141:7001

  mogadm check                     Check the state of the MogileFS world.

  mogadm stats                     Show MogileFS system statistics.   mogadm host ...

         host add ...              Add a host to MogileFS.

         host delete ...           Delete a host.

         host list                 List all hosts.

         host mark ...             Change the status of a host.  (equivalent to 'modify --status')

         host modify ...           Modify a host's properties.

  mogadm device ...

         device add ...            Add a device to a host.

         device list ...           List all devices, for each host.

         device mark ...           Mark a device as {alive,dead,down,drain,readonly}

         device modify ...         Modify a device's properties.

         device summary ...        List the summary of devices, for each host.

  mogadm domain ...

         domain add ...            Add a domain (namespace)

         domain delete ...         Delete a domain.

         domain list               List all hosts.

  mogadm class ...

         class add ...             Add a file class to a domain.

         class delete ...          Delete a file class from a domain.

         class list                List all classes, for each domain.

         class modify ...          Modify properties of a file class.

  mogadm slave ...

         slave add ...             Add a slave node for store usage

         slave delete ...          Delete a slave node for store usage

         slave list                List current store slave nodes.

         slave modify ...          Modify a slave node for store usage

  mogadm fsck ...

         fsck clearlog             Clear the fsck log

         fsck printlog             Display the fsck log

         fsck reset ...            Reset fsck position back to the beginning

         fsck start                Start (or resume) background fsck

         fsck status               Show fsck status

         fsck stop                 Stop (pause) background fsck

         fsck taillog              Tail the fsck log

  mogadm rebalance ...

         rebalance policy ...      Add or adjust the current policy

         rebalance reset           Reset an existing policy

         rebalance settings        Display rebalance settings

         rebalance start           Start a rebalance job

         rebalance status          Show status of current rebalance job

         rebalance stop            Stop a rebalance job

         rebalance test            Show what devices the current policy would match

  mogadm settings ...

         settings list             List all server settings

         settings set ...          Set server setting 'key' to 'value'.

  1. 首先需要把节点设备添加至管理设备中。使其能够正常管理

[root@mogilefs1 ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.140 --ip=172.16.17.140 --status=alive

[root@mogilefs1 ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.141 --ip=172.16.17.141 --status=alive

[root@mogilefs1 ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.144 --ip=172.16.17.144 --status=alive

[root@mogilefs1 ~]#

添加完成之后,去确认设备添加情况

[root@mogilefs1 ~]# mogadm host list

172.16.17.140 [1]: alive

  IP:       172.16.17.140:7500

172.16.17.141 [2]: alive

  IP:       172.16.17.141:7500

172.16.17.144 [3]: alive

  IP:       172.16.17.144:7500

12、后端存储的节点添加完成之后,接着去添加设备文件。

[root@mogilefs1 ~]# mogadm device add 172.16.17.140 1

[root@mogilefs1 ~]# mogadm device add 172.16.17.141 2

[root@mogilefs1 ~]# mogadm device add 172.16.17.144 3

[root@mogilefs1 ~]# mogadm device list

172.16.17.140 [1]: alive

                    used(G)    free(G)   total(G)  weight(%)

   dev1:   alive      0.280     18.280     18.561        100

172.16.17.141 [2]: alive

                    used(G)    free(G)   total(G)  weight(%)

   dev2:   alive      0.280     18.280     18.561        100

172.16.17.144 [3]: alive

                    used(G)    free(G)   total(G)  weight(%)

   dev3:   alive      1.671     16.890     18.561        100

再次检查mogadm的状态信息

[root@mogilefs1 ~]# mogadm check

Checking trackers...

  127.0.0.1:7001 ... OK

Checking hosts...

  [ 1] 172.16.17.140 ... OK

  [ 2] 172.16.17.141 ... OK

  [ 3] 172.16.17.144 ... OK

Checking devices...

  host device         size(G)    used(G)    free(G)   use%   ob state   I/O%

  ---- ------------ ---------- ---------- ---------- ------ ---------- -----

  [ 1] dev1            18.561      0.281     18.280   1.51%  writeable   0.0

  [ 2] dev2            18.561      0.281     18.280   1.51%  writeable   0.0

  [ 3] dev3            18.561      1.671     16.890   9.00%  writeable   1.7

  ---- ------------ ---------- ---------- ---------- ------

             total:    55.683      2.233     53.451   4.01%

[root@mogilefs1 ~]#

  1. 添加domain域空间存储目录

[root@mogilefs1 ~]# mogadm domain add files

[root@mogilefs1 ~]# mogadm domain add images

[root@mogilefs1 ~]# mogadm domain list

 domain               class                mindevcount   replpolicy   hashtype

-------------------- -------------------- ------------- ------------ -------

 files                default                   2        MultipleHosts() NONE   

 images               default                   2        MultipleHosts() NONE   

在创建domain的时候,根绝自己的需要进行创建。

  1. 创建好域名空间之后,如果需要有创建class类别。也能创建,这样能够分类别存储文件

基本使用方法

[root@mogilefs1 ~]# mogadm class help

  mogadm class add <domain> <class> [opts]           Add a file class to a domain.

      <class>              Name of class to add.

      <domain>             Domain to add class to.

      --hashtype=s         Hash algorithm string ('MD5', 'NONE').

      --mindevcount=i      Minimum number of replicas.

      --replpolicy=s       Replication policy string.

添加几个基本的类试试

[root@mogilefs1 ~]# mogadm class add images jpg

[root@mogilefs1 ~]# mogadm class add images png

[root@mogilefs1 ~]# mogadm class add images gif

[root@mogilefs1 ~]# mogadm class add files text --mindevcount=1

[root@mogilefs1 ~]# mogadm class add files html --replpolicy="MultipleHosts(3)"

[root@mogilefs1 ~]# mogadm class list

 domain               class                mindevcount   replpolicy   hashtype

 files                default                   2        MultipleHosts() NONE   

 files                html                      2        MultipleHosts(3) NONE   

 files                plaintext                 1        MultipleHosts() NONE   

 

 images               default                   2        MultipleHosts() NONE   

 images               gif                       2        MultipleHosts() NONE   

 images               jpeg                      2        MultipleHosts() NONE   

 images               png                       2        MultipleHosts() NONE   

  1. 如果基本的属性定义完成之后,接着可以尝试去上传资源试试

[root@mogilefs1 ~]# mogupload

Usage: /usr/bin/mogupload --trackers=host --domain=foo --key='/hello.jpg' --file='./hello.jpg'

[root@mogilefs1 ~]# mogupload --trackers=172.16.17.140 --domain=files --key='/fstab.txt' --file='/etc/fstab' --class=plaintext

上传完成之后,确认上传情况。能够,会生成http连接。但是连接会比较特殊。

[root@mogilefs1 ~]# mogfileinfo --trackers=172.16.17.140 --domain=files --key='/fstab.txt'

- file: /fstab.txt

     class:            plaintext

  devcount:                    1

    domain:                files

       fid:                    2

       key:           /fstab.txt

    length:                  921

 - http://172.16.17.141:7500/dev2/0/000/000/0000000002.fid

[root@mogilefs1 ~]#

wKiom1ca4vTCtxK4AAB7Fp4GPuI591.png 

[root@mogilefs1 mogilefs]# mogupload --trackers=172.16.17.140 --domain=images --key='/1.jpg' --file='1.jpg' --class=jpeg

[root@mogilefs1 mogilefs]# mogfileinfo --trackers=172.16.17.140 --domain=images --key='/1.jpg'

- file: /1.jpg

     class:                 jpeg

  devcount:                    2

    domain:               images

       fid:                    3

       key:               /1.jpg

    length:               170625

 - http://172.16.17.140:7500/dev1/0/000/000/0000000003.fid

 - http://172.16.17.141:7500/dev2/0/000/000/0000000003.fid

 

[root@mogilefs1 mogilefs]# mogstats --db_dsn="DBI:mysql:mogilefs:host=172.16.17.144" --db_user="moguser" --db_pass="mogpass" --verbose  --stats="all"

Statistics for devices...

  device     host                   files     status

  ---------- ---------------- ------------ ----------

  dev1       172.16.17.140          1      alive

  dev2       172.16.17.141          2      alive

  ---------- ---------------- ------------ ----------

Statistics for file ids...

  Max file id: 3

Statistics for files...

  domain               class           files    size (m)  fullsize (m)

  -------------------- ----------- ---------- ----------- -------------

  files                plaintext           1           0             0

  images               jpeg                1           0             0

  -------------------- ----------- ---------- ----------- -------------

Statistics for replication...

  domain               class        devcount      files

  -------------------- ----------- ---------- ----------

  files                plaintext           1          1

  images               jpeg                2          1

  -------------------- ----------- ---------- ----------

如果想要通过更加便捷的方式进行上传数据需要,程序员自己写API接口

 

前段基于nginx进行反代mogilefs文件系统

  1. 需要的安装包组

[root@backproxy ~]# ll

-rw-r--r--  1 root root 832104 Dec  4 15:32 nginx-1.8.0.tar.gz

-rw-r--r--  1 root root  11208 Dec  4 18:54 nginx_mogilefs_module-1.0.4.tar.gz

需要注意的一点是,对于nginx而言,如果想实现对后端的mogilefs服务器的代理需要nginx_mogilefs_module模块。否则不能实现此功能;同时,还需要注意的一点是模板的版本需要严格要求,模块的版本信息要和nginx版本信息匹配否则会报内核溢出之类的错误

2015/12/04 18:36:42 [alert] 1880#0: worker process 1890 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1892 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1894 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1896 exited on signal 11 (core dumped)

  1. 在安装之前需要解决需要的依赖环境

[root@backproxy ~]#  yum groupinstall "Development Tools" "Server Platform Deveopment"

[root@backproxy ~]# yum install openssl-devel pcre-devel

还需要创建一个nginx用户

[root@backproxy ~]# groupadd -r nginx

[root@backproxy ~]# useradd -r -g nginx nginx

  1. 接着解压安装包进行安装

[root@backproxy ~]# ./configure \

  --prefix=/usr \

  --sbin-path=/usr/sbin/nginx \

  --conf-path=/etc/nginx/nginx.conf \

  --error-log-path=/var/log/nginx/error.log \

  --http-log-path=/var/log/nginx/access.log \

  --pid-path=/var/run/nginx/nginx.pid  \

  --lock-path=/var/lock/nginx.lock \

  --user=nginx \

  --group=nginx \

  --with-http_ssl_module \

  --with-http_flv_module \

  --with-http_stub_status_module \

  --with-http_gzip_static_module \

  --http-client-body-temp-path=/var/tmp/nginx/client/ \

  --http-proxy-temp-path=/var/tmp/nginx/proxy/ \

  --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \

  --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \

  --http-scgi-temp-path=/var/tmp/nginx/scgi \

  --with-pcre \

  --with-debug \

  --add-module=

这里需要特别注意的一项是,在编译安装nginx的时候,需要为期指定mogilefs的模块相对于的文件--add-module=../nginx_mogilefs_module-1.0.4

编译完成之后进行

Make && make install

  1. nginx提供一个启动脚本如下

#!/bin/sh

# nginx - this script starts and stops the nginx daemon

# chkconfig:   - 85 15

# description:  Nginx is an HTTP(S) server, HTTP(S) reverse \

#               proxy and IMAP/POP3 proxy server

# processname: nginx

# config:      /etc/nginx/nginx.conf

# config:      /etc/sysconfig/nginx

# pidfile:     /var/run/nginx.pid

# Source function library.

. /etc/rc.d/init.d/functions

# Source networking configuration.

. /etc/sysconfig/network

# Check that networking is up.

[ "$NETWORKING" = "no" ] && exit 0

nginx="/usr/sbin/nginx"

prog=$(basename $nginx)

 

NGINX_CONF_FILE="/etc/nginx/nginx.conf"

 

[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx

 

lockfile=/var/lock/subsys/nginx

 

make_dirs() {

   # make required directories

   user=`nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -`

   options=`$nginx -V 2>&1 | grep 'configure arguments:'`

   for opt in $options; do

       if [ `echo $opt | grep '.*-temp-path'` ]; then

           value=`echo $opt | cut -d "=" -f 2`

           if [ ! -d "$value" ]; then

               # echo "creating" $value

               mkdir -p $value && chown -R $user $value

           fi

       fi

   done

}

 

start() {

    [ -x $nginx ] || exit 5

    [ -f $NGINX_CONF_FILE ] || exit 6

    make_dirs

    echo -n $"Starting $prog: "

    daemon $nginx -c $NGINX_CONF_FILE

    retval=$?

    echo

    [ $retval -eq 0 ] && touch $lockfile

    return $retval

}

 

stop() {

    echo -n $"Stopping $prog: "

    killproc $prog -QUIT

    retval=$?

    echo

    [ $retval -eq 0 ] && rm -f $lockfile

    return $retval

}

 

restart() {

    configtest || return $?

    stop

    sleep 1

    start

}

 

reload() {

    configtest || return $?

    echo -n $"Reloading $prog: "

    killproc $nginx -HUP

    RETVAL=$?

    echo

}

 

force_reload() {

    restart

}

 

configtest() {

  $nginx -t -c $NGINX_CONF_FILE

}

 

rh_status() {

    status $prog

}

 

rh_status_q() {

    rh_status >/dev/null 2>&1

}

 

case "$1" in

    start)

        rh_status_q && exit 0

        $1

        ;;

    stop)

        rh_status_q || exit 0

        $1

        ;;

    restart|configtest)

        $1

        ;;

    reload)

        rh_status_q || exit 7

        $1

        ;;

    force-reload)

        force_reload

        ;;

    status)

        rh_status

        ;;

    condrestart|try-restart)

        rh_status_q || exit 0

            ;;

    *)

        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"

        exit 2

esac

而后为此脚本赋予执行权限:

[root@backproxy ~]# chmod +x /etc/rc.d/init.d/nginx

添加至服务管理列表,并让其开机自动启动:

[root@backproxy ~]# chkconfig --add nginx

[root@backproxy ~]# chkconfig nginx on

而后就可以启动服务并测试了:

[root@backproxy ~]# service nginx start

  1. 接着去修改nginx的主配置文件

upstream trackers {

        server 172.16.17.140:7001 weight=1;

        server 172.16.17.141:7001 weight=1;

        server 172.16.17.144:7001 weight=1;

    }

Server {

        location /images {

            mogilefs_tracker trackers;

            mogilefs_domain images;

            mogilefs_methods GET;

            mogilefs_noverify on;

            mogilefs_pass {

                  proxy_pass $mogilefs_path;

                  proxy_hide_header Content-Type;

                  proxy_buffering off;

            }

        }

        location /files {

            mogilefs_tracker trackers;

            mogilefs_domain files;

            mogilefs_methods GET;

            mogilefs_noverify on;

            mogilefs_pass {

                  proxy_pass $mogilefs_path;

                  proxy_hide_header Content-Type;

                  proxy_buffering off;

            }

        }

}

  1. 配置结束之后,重启服务器可以去测试反代效果如何?

wKioL1ca5AjzuqrBAAA7uzO8LlY748.png 

查看反代日志

[root@backproxy ~]# tail -2 /var/log/nginx/access.log

172.16.1.100 - - [04/Dec/2015:22:18:20 +0800] "GET /images/love.jpg HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"

172.16.1.100 - - [04/Dec/2015:22:18:20 +0800] "GET /images/love.jpg HTTP/1.1" 200 190166 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"

[root@backproxy ~]#

  1. 接着尝试down机一台存储服务器测试效果如何

[root@mogilefs2 ~]# service mogstored stop

Stop mogstored                                         [  OK  ]

wKiom1ca41KBwQhLAAA7uzO8LlY056.png 

对于前段访问后端存储服务器不会出现任何问题

 

注意点:对于这个版本的mogilefs分布式文件系统,在centos7上,只能存储一份,不能有其他多余的副本