1,简介

FastDFS是一款类GoogleFS的开源分布式文件系统,它用纯C语言实现,支持Linux、FreeBSD、AIX等UNIX系统。它只能通过专有API对文件进行存取访问,不支持POSIX接口方式,不能mount使用。准确地讲,GoogleFS以及FastDFS、mogileFS、HDFS、TFS等类GoogleFS都不是系统级的分布式文件系统,而是应用级的分布式文件存储服务。

FastDFS是为互联网应用量身定做的分布式文件系统,充分考虑了冗余备份、负载均衡、线性扩容等机制,并注重高可用、高性能等指标。和现有的类Google FS分布式文件系统相比,FastDFS的架构和设计理念有其独到之处,主要体现在轻量级、分组方式和对等结构三个方面。

FastDFS是由国人余庆所开发,其项目地址:https://github.com/happyfish100 。

2,架构

2.1,架构图

FastDFS服务端有两个角色:跟踪器(tracker)和存储节点(storage)。跟踪器主要做调度工作,在访问上起负载均衡的作用。

分布式架构怎么部署在多个服务器上操作_nginx

2.2,模块介绍

  • tracker server:跟踪服务器,用来调度来自客户端的请求,且在内存中记录所有存储组和存储服务器的信息状态。
  • storage server:存储服务器,用来存储文件(data)和文件属性(metadata)。
  • client:客户端,业务请求发起方,通过专用接口基于TCP协议与tracker server和storage server进行交互。
  • group:组,也可称为卷,同组内上的文件是完全相同的。
  • 文件标识:包括两部分,组名(group)和文件名(含路径)
  • 文件相关属性:键值对(Key Value Pair)方式
  • 文件名:与原文件名并不相同。由storage server根据特定信息生成,并且可逆,文件名包含:源存储服务器的IP地址、文件创建时间戳、文件大小、随机数和文件扩展名等。

3,部署安装

3.1,安装包准备

执行下面的shell,下载安装包,分别是libfastcommon基础组件、fastdfs、nginx、fastdfs-nginx-module。

wget https://github.com/happyfish100/libfastcommon/archive/V1.0.7.tar.gz
wget https://github.com/happyfish100/fastdfs/archive/V5.05.tar.gz
wget http://nginx.org/download/nginx-1.11.8.tar.gz
wget http://jaist.dl.sourceforge.NET/project/fastdfs/FastDFS%20Nginx%20Module%20Source%20Code/fastdfs-nginx-module_v1.16.tar.gz

3.2,安装libfastcommon基础组件

  • 解压:tar -zxvf V1.0.7.tar.gz
  • 进入解压后目录:cd libfastcommon-1.0.7/
  • 编译:./make.sh
  • 安装:sudo ./make.sh install
  • 设置软链接,否则无法启动
    ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
    ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so

3.3,安装fastdfs

安装

  • 解压: tar -zxvf V5.05.tar.gz
  • 进入解压后目录:cd fastdfs-5.05/
  • 编译:./make.sh
  • 安装:sudo ./make.sh install
  • 设置软链接,否则无法启动
    ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so
    ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so
  • 目录规划
    /usr/bin存放有编译出来的文件,包含fdfs_trackerd、fdfs_storaged、fdfs_monitor、fdfs_upload_file等
    /etc/fdfs存放有配置文件

配置文件

  • 复制一份tracker配置文件:
    cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf
  • 编辑:vim /etc/fdfs/tracker.conf
# the base path to store data and log files
base_path=/home/kevin/fastdfs/tracker/data-and-log
  • 复制一份storage配置文件:cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf
  • 编辑:vim /etc/fdfs/storage.conf
# the base path to store data and log files
base_path=/home/kevin/fastdfs/storage

# store_path#, based 0, if store_path0 not exists, it's value is base_path
# the paths must be exist
store_path0=/home/kevin/fastdfs/storage/images-data0
store_path1=/home/kevin/fastdfs/storage/images-data1
store_path2=/home/kevin/fastdfs/storage/images-data2

# tracker_server can ocur more than once, and tracker_server format is
#  "host:port", host can be hostname or ip address
tracker_server=192.168.1.32:22122
  • 备注:storage (存储节点)服务部署,一般 storage 服务我们会单独装一台机子,但是这里为了方便我们安装在同一台。如果 storage 单独安装的话,那上面安装的所有步骤都要在走一遍,只是到了编辑配置文件的时候,编辑的是 storage.conf 而已。

启动测试

tracker server
  • 启动 tracker 服务:
    /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
  • 重启 tracker 服务:
    /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart
  • 查看是否有tracker 进程:
    ps aux | grep tracker
storage server
  • 启动 storage 服务:
    /usr/bin/fdfs_storaged /etc/fdfs/storage.conf,首次启动会很慢,因为它在创建预设存储文件的目录
  • 重启 storage 服务:
    /usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart
  • 查看是否有 storage 进程:
    ps aux | grep storage

3.4,client测试

  • 编辑/etc/fdfs/client.conf
# the base path to store log files
base_path=/home/kevin/fastdfs/client/data-and-log

# tracker_server can ocur more than once, and tracker_server format is
#  "host:port", host can be hostname or ip address
tracker_server=192.168.1.32:22122
  • 通过fdfs_test上传的图片,保存了两张图片
kevin@orange:~/fastdfs/temp$ wget http://img.tuku.cn/file_thumb/201504/m2015042221045424.jpg
kevin@orange:~/fastdfs/temp$ /usr/bin/fdfs_test /etc/fdfs/client.conf upload ./m2015042221045424.jpg
kevin@orange:~/fastdfs/storage/images-data0$ find . -name "*.jpg"
./data/00/00/wKgBIFmemtyAGcebAAAdgkSCZh4105.jpg
./data/00/00/wKgBIFmemtyAGcebAAAdgkSCZh4105_big.jpg
  • 通过fdfs_upload_file上传,保存一张原始图片
kevin@orange:~/fastdfs/temp$ /usr/bin/fdfs_upload_file /etc/fdfs/client.conf m2015042221045424.jpg
group1/M00/00/00/wKgBIFmenECAAOdbAAAdgkSCZh4066.jpg
kevin@orange:~/fastdfs/storage/images-data0$ find . -name "*.jpg" | xargs ls -l
-rw-r--r-- 1 root root 7554  8月 24 17:22 ./data/00/00/wKgBIFmemtyAGcebAAAdgkSCZh4105_big.jpg
-rw-r--r-- 1 root root 7554  8月 24 17:22 ./data/00/00/wKgBIFmemtyAGcebAAAdgkSCZh4105.jpg
-rw-r--r-- 1 root root 7554  8月 24 17:28 ./data/00/00/wKgBIFmenECAAOdbAAAdgkSCZh4066.jpg
  • 即使我们现在知道图片的访问地址我们也访问不了,因为我们还没装 FastDFS 的 Nginx 模块

3.5,安装nginx及其插件

安装

  • nginx插件的配置文件
# 修改配置文件中的无效路径,把local去掉
vim /opt/setups/FastDFS/fastdfs-nginx-module/src/config

CORE_INCS="$CORE_INCS /usr/local/include/fastdfs /usr/local/include/fastcommon/"
修改为
CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/"

复制文件:cp fastdfs-5.05/conf/http.conf /etc/fdfs
复制文件:cp fastdfs-5.05/conf/mime.types /etc/fdfs
  • 安装 Nginx 依赖包(Ubuntu)
sudo apt-get install openssl libssl-dev
sudo apt-get install libpcre3 libpcre3-dev  
sudo apt-get install zlib1g-dev
  • 通过命令,查看是否安装某些包
    dpkg -l | grep zlib
  • 创建nginx文件目录
    mkdir -p /usr/local/nginx /var/log/nginx /var/temp/nginx /var/lock/nginx
  • 生成配置,注意修改最后一行(–add-module参数)
./configure \
--prefix=/usr/local/nginx \
--pid-path=/var/local/nginx/nginx.pid \
--lock-path=/var/lock/nginx/nginx.lock \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--with-http_gzip_static_module \
--http-client-body-temp-path=/var/temp/nginx/client \
--http-proxy-temp-path=/var/temp/nginx/proxy \
--http-fastcgi-temp-path=/var/temp/nginx/fastcgi \
--http-uwsgi-temp-path=/var/temp/nginx/uwsgi \
--http-scgi-temp-path=/var/temp/nginx/scgi \
--add-module=/home/kevin/fastdfs/install/soft/fastdfs-nginx-module/src
  • 编译:make
  • 安装:sudo make install
  • 复制配置文件:cp fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs
  • 确认nginx的fastdfs插件是否安装成功
root@orange:/usr/local/nginx/conf# /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.11.8
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) 
configure arguments: --prefix=/usr/local/nginx --pid-path=/var/local/nginx/nginx.pid --lock-path=/var/lock/nginx/nginx.lock --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --with-http_gzip_static_module --http-client-body-temp-path=/var/temp/nginx/client --http-proxy-temp-path=/var/temp/nginx/proxy --http-fastcgi-temp-path=/var/temp/nginx/fastcgi --http-uwsgi-temp-path=/var/temp/nginx/uwsgi --http-scgi-temp-path=/var/temp/nginx/scgi --add-module=/home/kevin/fastdfs/install/soft/fastdfs-nginx-module/src
  • 备注:–prefix=/usr/local/nginx说明nginx目录是/usr/local/nginx,–add-module说明nginx的插件

配置文件

  • 编辑/etc/fdfs/mod_fastdfs.conf
# the base path to store log files
base_path=/home/kevin/fastdfs/fastdfs-nginx-module/data-and-log/

# FastDFS tracker_server can ocur more than once, and tracker_server format is
#  "host:port", host can be hostname or ip address
# valid only when load_fdfs_parameters_from_tracker is true
tracker_server=192.168.1.32:22122

# if the url / uri including the group name
# set to false when uri like /M00/00/00/xxx
# set to true when uri like ${group_name}/M00/00/00/xxx, such as group1/M00/xxx
# default value is false
url_have_group_name = true

# store_path#, based 0, if store_path0 not exists, it's value is base_path
# the paths must be exist
# must same as storage.conf
store_path0=/home/kevin/fastdfs/storage/images-data0
store_path1=/home/kevin/fastdfs/storage/images-data1
store_path2=/home/kevin/fastdfs/storage/images-data2
  • 编辑/usr/local/nginx/conf/nginx.conf
# 在http添加对其他配置文件的引用
include /usr/local/nginx/conf/conf.d/*.conf;
  • 在/usr/local/nginx/conf/目录下创建文件夹conf.d,并编辑fdfs.conf,完成nginx与fastdfs的对接
server {
            listen       8888;
            server_name  localhost;
            location ~/group[0-9]/ {
                ngx_fastdfs_module;
            }
            error_page   500 502 503 504  /50x.html;
            location = /50x.html {
            root   html;
            }
        }

4,测试

kevin@orange:~/fastdfs/temp$ fdfs_upload_file /etc/fdfs/client.conf meixi.jpg
group1/M00/00/00/wKgBIFmeyGWAJHacAACiyMy3SR8400.jpg
kevin@orange:~/fastdfs/temp$ fdfs_upload_file /etc/fdfs/client.conf  gnuplot.pdf 
group1/M00/00/00/wKgBIFmeyJiAFBM2ABhOlsdTgP0787.pdf
kevin@orange:~/fastdfs/temp$ fdfs_upload_file /etc/fdfs/client.conf my_multi.cnf 
group1/M00/00/00/wKgBIFmeyLeARRxPAAAHX_E2eqU506.cnf
kevin@orange:~/fastdfs/temp$ fdfs_upload_file /etc/fdfs/client.conf tiger.txt 
group1/M00/00/00/wKgBIFmeyLyAEhFzAAAFTfDtQ9M236.txt
  • 通过wget下载测试
wget http://192.168.1.32:8888/group1/M00/00/00/wKgBIFmeyGWAJHacAACiyMy3SR8400.jpg
wget http://192.168.1.32:8888/group1/M00/00/00/wKgBIFmeyJiAFBM2ABhOlsdTgP0787.pdf
wget http://192.168.1.32:8888/group1/M00/00/00/wKgBIFmeyLeARRxPAAAHX_E2eqU506.cnf
wget http://192.168.1.32:8888/group1/M00/00/00/wKgBIFmeyLyAEhFzAAAFTfDtQ9M236.txt
  • 通过浏览器直接访问

分布式架构怎么部署在多个服务器上操作_配置文件_02