FastDFS是一个国产开源的轻量级分布式文件系统,它对文件进行管理,功能包括:文件存储、文件同步、文件访问(文件上传、文件下载)等,解决了大容量存储和负载均衡的问题。特别适合以文件为载体的在线服务,如相册网站、视频网站等等。FastDFS服务端有两个角色:跟踪器(tracker)和存储节点(storage)。跟踪器主要做调度工作,在访问上起负载均衡的作用。


官方论坛: http://www.csource.org

下载地址: http://sourceforge.net/projects/fastdfs/files/


github软件仓库

libfastcommon  FastDFSFastDHT的通用函数库,地址:https://github.com/happyfish100/libfastcommon.git

FastDFS  FastDFS主代码,地址: https://github.com/happyfish100/fastdfs.git

fastdfs-nginx-module  FastDFSnginx模块,地址:https://github.com/happyfish100/fastdfs-nginx-module.git

nginx清除缓存模块 ngx_cache_purge-2.1 http://labs.frickle.com/nginx_ngx_cache_purge/


.安装

本安装使用 CentOS 6.6 x86_64版操作系统,按照以下网络结构进行部署:


1.下载安装文件


FastDFS 5.x 取消了对 libevent 的依赖,添加了对 libfastcommon 的依赖。

本部署说明用到的软件版本:

    libfastcommon v1.13

    FastDFS v5.06

    fastdfs-nginx-module v1.17

    可从上面的sourceforgegithub中下载,或者直接下载本文附带的压缩包。详细的安装说明可参照代码中的INSTALL


2.安装FastDFS

在每一台trackerstorage服务器上执行

# tar xjvf fdfs-5.06.tar.bz2

# yum install -y gcc perl   # 安装依赖的软件包

# cd ~/fdfs-5.06/libfastcommon && ./make.sh && ./make.sh install

# cd ~/fdfs-5.06/fastdfs && ./make.sh && ./make.sh install


确认make没有错误后,执行安装,64位系统默认会复制到/usr/lib64下。

这时候需要设置环境变量或者创建软链接

export LD_LIBRARY_PATH=/usr/lib64/

ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so


安装完成后,所有可执行文件在目录/usr/bin下,以fdfs_开头:

# ls /usr/bin/fdfs_*

/usr/bin/fdfs_appender_test   /usr/bin/fdfs_download_file  /usr/bin/fdfs_test1

/usr/bin/fdfs_appender_test1  /usr/bin/fdfs_file_info      /usr/bin/fdfs_trackerd

/usr/bin/fdfs_append_file     /usr/bin/fdfs_monitor        /usr/bin/fdfs_upload_appender

/usr/bin/fdfs_crc32           /usr/bin/fdfs_storaged       /usr/bin/fdfs_upload_file

/usr/bin/fdfs_delete_file     /usr/bin/fdfs_test



配置文件在目录/etc/fdfs下:

# ls /etc/fdfs/

client.conf.sample  http.conf  mime.types  storage.conf.sample  tracker.conf  tracker.conf.sample


.FastDFS配置

1.配置Tracker跟踪器

开放tracker监听端口访问(适用于CentOS 7,省略)

firewall-cmd --zone=public --add-port=22122/tcp --permanent

firewall-cmd --reload


修改配置文件

mkdir -p /data/fastdfs

cd /etc/fdfs

cp tracker.conf.sample tracker.conf

cp /root/fdfs/fastdfs/conf/http.conf .

cp /root/fdfs/fastdfs/conf/mime.types .

sed -i 's:base_path=.*:base_path=/data/fastdfs:g' tracker.conf

sed -i 's:http.server_port=.*:http.server_port=80:g' tracker.conf


启动tracker:

# /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf

# ps aux |grep fdfs

root     26350  0.0  0.0  78732  2840 ?        Sl   11:28   0:00 /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf

root     26358  0.0  0.0 103252   812 pts/0    S+   11:28   0:00 grep fdfs

# ss -tunlp |grep 26350

tcp    LISTEN     0      1024                   *:22122                 *:*      users:(("fdfs_trackerd",26350,5))



2.配置Storage存储节点:

开放tracker监听端口访问(省略)

firewall-cmd --zone=public --add-port=23000/tcp --permanent

firewall-cmd --reload


修改配置文件

mkdir -p /data/fastdfs

cd /etc/fdfs

cp storage.conf.sample storage.conf

cp /root/fdfs-5.06/fastdfs/conf/http.conf ./

cp /root/fdfs-5.06/fastdfs/conf/mime.types ./

sed -i 's:base_path=.*:base_path=/data/fastdfs:g' storage.conf

sed -i 's:store_path0=.*:store_path0=/data/fastdfs:g' storage.conf

sed -i 's/tracker_server=.*/tracker_server=192.168.23.72:22122/g' storage.conf

sed -i 's:http.server_port=.*:http.server_port=80:g' storage.conf


启动storage

/usr/bin/fdfs_storaged /etc/fdfs/storage.conf

检查启动进程:

# ps aux |grep fdfs

root     26285  0.5  1.7  80852 66756 ?        Sl   11:38   0:00 /usr/bin/fdfs_storaged /etc/fdfs/storage.conf

root     26295  0.0  0.0 103252   804 pts/0    S+   11:38   0:00 grep fdfs


检测监听端口:

# ss -tunlp |grep 26285

tcp    LISTEN     0      1024                   *:23000                 *:*      users:(("fdfs_storaged",26285,5))



3.client客户端配置

在tracker, storage之外的一台主机上安装FastDFS,然后执行:

mkdir -p /data/fastdfs

cd /etc/fdfs

cp client.conf.sample client.conf

sed -i 's:base_path=.*:base_path=/data/fastdfs:g' client.conf

sed -i 's/tracker_server=.*/tracker_server=192.168.23.72:22122/g' client.conf


FastDFS测试

上传测试:

#  /usr/bin/fdfs_test /etc/fdfs/client.conf upload  install.log      

This is FastDFS client test program v5.06


Copyright (C) 2008, Happy Fish / YuQing


FastDFS may be copied only under the terms of the GNU General

Public License V3, which may be found in the FastDFS source kit.

Please visit the FastDFS Home Page http://www.csource.org/ 

for more detail.


[2015-06-30 01:39:23] DEBUG - base_path=/data/fastdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0


tracker_query_storage_store_list_without_group:

        server 1. group_name=, ip_addr=192.168.23.73, port=23000

        server 2. group_name=, ip_addr=192.168.23.78, port=23000


group_name=group1, ip_addr=192.168.23.73, port=23000

storage_upload_by_filename

group_name=group1, remote_filename=M00/00/00/wKgXSVWR80-AKrL0AAAhTq9t0dk472.log

source ip address: 192.168.23.73

file timestamp=2015-06-30 09:39:27

file size=8526

file crc32=2943209945

example file url: http://192.168.23.73/group1/M00/00/00/wKgXSVWR80-AKrL0AAAhTq9t0dk472.log

storage_upload_slave_by_filename

group_name=group1, remote_filename=M00/00/00/wKgXSVWR80-AKrL0AAAhTq9t0dk472_big.log

source ip address: 192.168.23.73

file timestamp=2015-06-30 09:39:27

file size=8526

file crc32=2943209945

example file url: http://192.168.23.73/group1/M00/00/00/wKgXSVWR80-AKrL0AAAhTq9t0dk472_big.log


客户端监控:

[root@instance-mysql ~]# fdfs_monitor /etc/fdfs/client.conf

[2015-06-30 07:14:29] DEBUG - base_path=/data/fastdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0


server_count=1, server_index=0


tracker server is 192.168.23.72:22122


group count: 1


Group 1:

group name = group1

disk total space = 76102 MB

disk free space = 64333 MB

trunk free space = 0 MB

storage server count = 2

active server count = 2

storage server port = 23000

storage HTTP port = 80

store path count = 1

subdir count per path = 256

current write server index = 1

current trunk file id = 0


        Storage 1:

                id = 192.168.23.73

                ip_addr = 192.168.23.73  ACTIVE

                http domain =

                version = 5.06

                join time = 2015-06-29 11:38:37

                up time = 2015-06-29 11:38:37

                total storage = 76102 MB

                free storage = 70521 MB

                upload priority = 10

                store_path_count = 1

                subdir_count_per_path = 256

                storage_port = 23000

                storage_http_port = 80

                current_write_path = 0

                source storage id =

                if_trunk_server = 0

                connection.alloc_count = 256

                connection.current_count = 1

                connection.max_count = 3

                total_upload_count = 4

                success_upload_count = 4

                total_append_count = 0

                success_append_count = 0

                total_modify_count = 0

                success_modify_count = 0

                total_truncate_count = 0

                success_truncate_count = 0

                total_set_meta_count = 4

                success_set_meta_count = 4

                total_delete_count = 0

                success_delete_count = 0

                total_download_count = 0

                success_download_count = 0

                total_get_meta_count = 0

                success_get_meta_count = 0

                total_create_link_count = 0

                success_create_link_count = 0

                total_delete_link_count = 0

                success_delete_link_count = 0

                total_upload_bytes = 541576

                success_upload_bytes = 541576

                total_append_bytes = 0

                success_append_bytes = 0

                total_modify_bytes = 0

                success_modify_bytes = 0

                stotal_download_bytes = 0

                success_download_bytes = 0

                total_sync_in_bytes = 706836

                success_sync_in_bytes = 706836

                total_sync_out_bytes = 0

                success_sync_out_bytes = 0

                total_file_open_count = 8

                success_file_open_count = 8

                total_file_read_count = 0

                success_file_read_count = 0

                total_file_write_count = 12

                success_file_write_count = 12

                last_heart_beat_time = 2015-06-30 15:14:14

                last_source_update = 2015-06-30 13:56:47

                last_sync_update = 2015-06-30 13:56:29

                last_synced_timestamp = 2015-06-30 13:56:25 (0s delay)

        Storage 2:

                id = 192.168.23.78

                ip_addr = 192.168.23.78  ACTIVE

                http domain =

                version = 5.06

                join time = 2015-06-29 03:39:48

                up time = 2015-06-29 03:39:48

                total storage = 76102 MB

                free storage = 64333 MB

                upload priority = 10

                store_path_count = 1

                subdir_count_per_path = 256

                storage_port = 23000

                storage_http_port = 80

                current_write_path = 0

                source storage id = 192.168.23.73

                if_trunk_server = 0

                connection.alloc_count = 256

                connection.current_count = 1

                connection.max_count = 3

                total_upload_count = 2

                success_upload_count = 2

                total_append_count = 0

                success_append_count = 0

                total_modify_count = 0

                success_modify_count = 0

                total_truncate_count = 0

                success_truncate_count = 0

                total_set_meta_count = 2

                success_set_meta_count = 2

                total_delete_count = 0

                success_delete_count = 0

                total_download_count = 0

                success_download_count = 0

                total_get_meta_count = 0

                success_get_meta_count = 0

                total_create_link_count = 0

                success_create_link_count = 0

                total_delete_link_count = 0

                success_delete_link_count = 0

                total_upload_bytes = 706738

                success_upload_bytes = 706738

                total_append_bytes = 0

                success_append_bytes = 0

                total_modify_bytes = 0

                success_modify_bytes = 0

                stotal_download_bytes = 0

                success_download_bytes = 0

                total_sync_in_bytes = 541772

                success_sync_in_bytes = 541772

                total_sync_out_bytes = 0

                success_sync_out_bytes = 0

                total_file_open_count = 10

                success_file_open_count = 10

                total_file_read_count = 0

                success_file_read_count = 0

                total_file_write_count = 14

                success_file_write_count = 14

                last_heart_beat_time = 2015-06-30 15:14:12

                last_source_update = 2015-06-30 13:56:25

                last_sync_update = 2015-06-30 13:56:52

                last_synced_timestamp = 2015-06-30 13:56:47 (0s delay)


.安装nginx

配置storage的 nginx 服务器和tracker的反向代理服务器

防火墙开放http服务(适用于CentOS 7,此处省略)

firewall-cmd --permanent --zone=public --add-service=http

firewall-cmd --reload


1.安装并启用nginx  在三台服务器上都需要安装

安装nginx-1.9

编译环境:

# yum groupinstall Development Tools

# yum install openssl-devel libxslt-devel  gd-devel  perl-ExtUtils-Embed  libunwind  geoip-devel gperftools


创建nginx管理用户:

# useradd -M -s /sbin/nologin nginx


# id nginx

uid=500(nginx) gid=500(nginx) groups=500(nginx)


源码编译安装nginx:

官方提示说pcre模块版本在4.4 - 8.32 实际使用8.36 ,zlib模块源码要用1.1.3 — 1.2.7版本,实际使用1.2.8编译完成并未报错.

pcre-8.37.tar.gz

zlib-1.2.8.tar.gz


这里使用的是源码,而非编译后的安装目录哦!

解压pcre源码:

tar xvf pcre-8.37.tar.gz  -C /opt/test


解压zlib源码:

tar xvf zlib-1.2.8.tar.gz -C /opt/test


mkdir -pv /var/lib/nginx/tmp/{client_body,proxy,fastcgi,uwsgi,scgi}

mkdir /run/lock/subsys/ -p

mkdir -pv /var/log/nginx/


后续就能进行nginx-1.9的编译:

./configure --prefix=/usr/local/nginx --sbin-path=/usr/local/nginx/ --conf-path=/usr/local/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/var/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-pcre=/opt/test/pcre-8.37/ --with-zlib=/opt/test/zlib-1.2.8/ --with-mail --with-mail_ssl_module --with-http_spdy_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module --with-http_image_filter_module --with-http_geoip_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_stub_status_module --with-http_perl_module --with-mail --with-mail_ssl_module --with-google_perftools_module --add-module=/root/fdfs-5.06/fastdfs-nginx-module/src/   

编译完成后执行:

make && make install


设置nginx的环境变量:

echo "export PATH=/usr/local/nginx:$PATH" > /etc/profile.d/nginx.sh

source /etc/profile.d/nginx.sh


查看nginx的版本:

# nginx -v

nginx version: nginx/1.9.2


配置nginx启动脚本(省略)

chkconfig --add nginx

chkconfig nginx on


2.fastdfs模块配置

fastdfs-nginx-module用于解决同步延迟问题:


同组之间的服务器需要复制文件,有延迟的问题.

假设Tracker服务器将文件上传到了192.168.1.80,文件ID已经返回客户端,

这时,后台会将这个文件复制到192.168.1.30,如果复制没有完成,客户端就用这个ID在192.168.1.30取文件,肯定会出现错误

这个fastdfs-nginx-module可以重定向连接到源服务器取文件,避免客户端由于复制延迟的问题,出现错误。


fastdfs-nginx-module模块只需要安装到storage上。

cp /root/fdfs-5.06/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/

touch /data/fastdfs/logs/mod_fastdfs.log

chown nginx:nginx /data/fastdfs/logs/mod_fastdfs.log

vi /etc/fdfs/mod_fastdfs.conf

修改配置文件:

connect_timeout=2

network_timeout=30

base_path=/data/fastdfs                              #保存日志目录

load_fdfs_parameters_from_tracker=true

storage_sync_file_max_delay = 86400

use_storage_id = false

storage_ids_filename = storage_ids.conf

tracker_server=192.168.23.72:22122            #tracker 服务器的 IP 地址以及端口号

storage_server_port=23000                         #storage 服务器的端口号

group_name=group1                                  #当前服务器的 group 名

url_have_group_name = true                      #文件 url 中是否有 group 

store_path0=/data/fastdfs                          #存储路径

log_filename=/data/fastdfs/logs/mod_fastdfs.log  #日志文件名


3.Storage服务器nginx配置

server{} 里添加:对应的是url_have_group_name = true

location /group1/M00 {

   alias /data/fastdfs/data;

   ngx_fastdfs_module;

}


配置启用google-perftools:

如果使用googler开发的google-perftools优化Nginx和MySQL的内存管理,性能将会有一定程度的提升。特别是对高并发下的服务器,效果更明显

mkdir /tmp/tcmalloc

chmod 0777 /tmp/tcmalloc


配置nginx.conf,在pid文件下面加入:

#pid        logs/nginx.pid;

google_perftools_profiles /tmp/tcmalloc;


启动nginx服务:

# lsof -n | grep tcmalloc

nginx     13789   nginx   12w      REG              253,0        0    1837560 /tmp/tcmalloc.13789


    if ($request_filename ~* \.(png)|(jpg)|(txt)|(log)$) {  

                    set $sign 1;

                }  

                if ($args ~* ^flag=(.*)$) {  

                    set $flag $sign$1;  

                }  

                if ($flag = "1flag") {  

                    add_header Content-Disposition "attachment;";

                }  

注解:

    通过nginx提供的正则匹配判断get参数中的flag字段内容

    如果flag为download则下载图片,否则浏览器呈现图片

    指定指定文件的下载,防止信息泄露


完整的nginx.conf的配置文件:

# sed -e "s/#.*//g" nginx.conf |awk '{if (length !=0) print $0}' |sed '/^ *$/d'            

worker_processes  3;

google_perftools_profiles /tmp/tcmalloc;

events {

    worker_connections  1024;

}

http {

    include       mime.types;

    default_type  application/octet-stream;

    sendfile        on;

    keepalive_