分布式文件系统FastDFS介绍安装配置

原创

xhh_0168 2011-11-04 11:23:30 博主文章分类：FileSystem ©著作权

文章标签 负载均衡分布式文件系统高可用性 key-value fastdfs 文章分类 服务器

©著作权归作者所有：来自51CTO博客作者xhh_0168的原创作品，请联系作者获取转载授权，否则将追究法律责任

分布式文件系统FastDFS介绍安装配置引自：http://www.ttlsa.com/archives/301
一．介绍
FastDFS是一个开源的轻量级分布式文件系统，功能包括：文件存储、文件同步、文件访问（文件上传、文件下载）等，解决了大容量存储和负载均衡的问题。特别适合以文件为载体的在线服务，如相册网站、视频网站等等。
FastDFS有两个角色：跟踪器(tracker)和存储节点(storage)。
跟踪器主要做调度工作，在访问上起负载均衡的作用。
存储节点存储文件，完成文件管理的所有功能：存储、同步和提供存取接口，同时对文件的metadata进行管理。所谓文件的meta data就是文件的相关属性，以键值对（key valuepair）方式表示，如：width=1024，其中的key为width，value为1024。文件metadata是文件属性列表，可以包含多个键值对。
FastDFS架构图如下：

跟踪器和存储节点都可以由一台或多台服务器构成。跟踪器和存储节点中的服务器均可以随时增加或下线而不会影响线上服务。其中跟踪器中的所有服务器都是对等的，可以根据服务器的压力情况随时增加或减少。
为了支持大容量，存储节点（服务器）采用了分卷（或分组）的组织方式。存储系统由一个或多个卷组成，卷与卷之间的文件是相互独立的，所有卷的文件容量累加就是整个存储系统中的文件容量。一个卷可以由一台或多台存储服务器组成，一个卷下的存储服务器中的文件都是相同的，卷中的多台存储服务器起到了冗余备份和负载均衡的作用。在卷中增加服务器时，同步已有的文件由系统自动完成，同步完成后，系统自动将新增服务器切换到线上提供服务。当存储空间不足或即将耗尽时，可以动态添加卷。只需要增加一台或多台服务器，并将它们配置为一个新的卷，这样就扩大了存储系统的容量。
FastDFS中的文件标识分为两个部分：卷名和文件名，二者缺一不可。

上传文件交互过程：

　1. client询问tracker上传到的storage，不需要附加参数；
　2. tracker返回一台可用的storage；
　3. client直接和storage通讯完成文件上传。

下载文件交互过程：

　1. client询问tracker下载文件的storage，参数为文件标识（卷名和文件名）；
　2. tracker返回一台可用的storage；
　3. client直接和storage通讯完成文件下载。

同步机制
同一组内的storage server之间是对等的，文件上传、删除等操作可以在任意一台storage server上进行；
文件同步只在同组内的storage server之间进行，采用push方式，即源服务器同步给目标服务器；
源头数据才需要同步，备份数据不需要再次同步，否则就构成环路了；
上述第二条规则有个例外，就是新增加一台storage server时，由已有的一台storage server将已有的所有数据（包括源头数据和备份数据）同步给该新增服务器。

二．安装
# wget http://cloud.github.com/downloads/libevent/libevent/libevent-2.0.14-stable.tar.gz
# tar zxvf libevent-2.0.14-stable.tar.gz -C ../software/
# ./configure --prefix=/usr/local/libevent-2.0.14
# make
# make install

# wget http://fastdfs.googlecode.com/files/FastDFS_v3.02.tar.gz
# tar zxvf FastDFS_v3.02.tar.gz -C ../software/
# ./make.sh C_INCLUDE_PATH=/usr/local/libevent-2.0.14/include LIBRARY_PATH=/usr/local/libevent-2.0.14/lib
# ./make.sh install
【注意:】
如果要使fastdfs支持web和开机自动脚本(默认是不支持的)，需要修改make.sh文件
#WITH_HTTPD=1
#WITH_LINUX_SERVICE=1

配置文件说明
【tracker.conf】
# is this config file disabled
# false for enabled
# true for disabled
disabled=false //配置文件是否生效,false生效，true不生效

# bind an address of this host
# empty for bind all addresses of this host
bind_addr= //绑定ip

# the tracker server port
port=22122 //指定端口

# connect timeout in seconds
# default value is 30s
connect_timeout=30 //连接超时时间，针对socket套接字

# network timeout in seconds
# default value is 30s
network_timeout=60 //tracker server网络超时时间

# the base path to store data and log files
base_path=/data/fastdfs //目录地址，目录结构如下：
${base_path}
|__data
| |__storage_groups.dat：存储分组信息
| |__storage_servers.dat：存储服务器列表
|__logs
|__trackerd.log：tracker server日志文件
数据文件storage_groups.dat和storage_servers.dat中的记录之间以换行符（\n）分隔，字段之间以西文逗号（,）分隔。
storage_groups.dat中的字段依次为：
1. group_name：组名
2. storage_port：storage server端口号

storage_servers.dat中记录storage server相关信息，字段依次为：
1. group_name：所属组名
2. ip_addr：ip地址
3. status：状态
4. sync_src_ip_addr：向该storage server同步已有数据文件的源服务器
5. sync_until_timestamp：同步已有数据文件的截至时间（UNIX时间戳）
6. stat.total_upload_count：上传文件次数
7. stat.success_upload_count：成功上传文件次数
8. stat.total_set_meta_count：更改meta data次数
9. stat.success_set_meta_count：成功更改meta data次数
10. stat.total_delete_count：删除文件次数
11. stat.success_delete_count：成功删除文件次数
12. stat.total_download_count：下载文件次数
13. stat.success_download_count：成功下载文件次数
14. stat.total_get_meta_count：获取meta data次数
15. stat.success_get_meta_count：成功获取meta data次数
16. stat.last_source_update：最近一次源头更新时间（更新操作来自客户端）
17. stat.last_sync_update：最近一次同步更新时间（更新操作来自其他storage server的同步）

# max concurrent connections this server supported
max_connections=256 //最大连接数量

# work thread count, should <= max_connections
# default value is 4
# since V2.00
work_threads=4 //工作线程数量，通常设置为cpu数

# the method of selecting group to upload files
# 0: round robin
# 1: specify group
# 2: load balance, select the max free space group to upload file
store_lookup=2 //上传组方式：0轮询，1指定，2均衡负载，选择剩余空间最大的上传

# which group to upload file
# when store_lookup set to 1, must set store_group to the group name
store_group=group2 //当store_lookup=1时，必须设置此参数，指定上传的组名

# which storage server to upload file
# 0: round robin (default)
# 1: the first server order by ip address
# 2: the first server order by priority (the minimal)
store_server=0 //选择哪个storage server进行上传操作。一个文件上传后，这个storage server就相当于该文件的storage server源，会对同组的storage server进行同步。0轮询，1根据ip地址排序选择第一个，2根据优先级进行排序

# which path(means disk or mount point) of the storage server to upload file
# 0: round robin
# 2: load balance, select the max free space path to upload file
store_path=0 //选择storage server哪个目录进行上传，storage server可以有多个存放文件的base path。0轮询，2均衡负载，选择剩余空间最大的目录

# which storage server to download file
# 0: round robin (default)
# 1: the source storage server which the current file uploaded to
download_server=0 //选择哪个storage server作为下载服务器。0轮询1选择原始源storage server

# reserved storage space for system or other applications.
# if the free(available) space of any stoarge server in
# a group <= reserved_storage_space,
# no file can be uploaded to this group.
# bytes unit can be one of follows:
### G or g for gigabyte(GB)
### M or m for megabyte(MB)
### K or k for kilobyte(KB)
### no unit for byte(B)
reserved_storage_space = 4GB //storage server预留空间大小

#standard log level as syslog, case insensitive, value list:
### emerg for emergency
### alert
### crit for critical
### error
### warn for warning
### notice
### info
### debug
log_level=info //日志级别

#unix group name to run this program,
#not set (empty) means run by the group of current user
run_by_group= //运行用户组

#unix username to run this program,
#not set (empty) means run by current user
run_by_user= //运行用户

# allow_hosts can ocur more than once, host can be hostname or ip address,
# "*" means match all ip addresses, can use range like this: 10.0.1.[1-15,20] or
# host[01-08,20-25].domain.com, for example:
# allow_hosts=10.0.1.[1-15,20]
# allow_hosts=host[01-08,20-25].domain.com
allow_hosts=* //允许连接此tracker server的ip范围

# sync log buff to disk every interval seconds
# default value is 10 seconds
sync_log_buff_interval = 10 //同步日志到磁盘的时间间隔，tracker server日志先写入到内存。

# check storage server alive interval seconds
check_active_interval = 120 //检测storage server存活时间间隔

# thread stack size, should >= 64KB
# default value is 64KB
thread_stack_size = 64KB //线程堆大小

# auto adjust when the ip address of the storage server changed
# default value is true
storage_ip_changed_auto_adjust = true //当storage server IP地址改变时，集群是否自动调整。需要storage server重启以便完成自动调整

# storage sync file max delay seconds
# default value is 86400 seconds (one day)
# since V2.00
storage_sync_file_max_delay = 86400 //storage server之间同步文件的最大延时时间

# the max time of storage sync a file
# default value is 300 seconds
# since V2.00
storage_sync_file_max_time = 300 //storage server同步一个文件的最大时间

# if use a trunk file to store several small files
# default value is false
# since V3.00
use_trunk_file = false //是否使用trunk文件存储几个小文件

# the min slot size, should <= 4KB
# default value is 256 bytes
# since V3.00
slot_min_size = 256 //slot最小大小

# the max slot size, should > slot_min_size
# store the upload file to trunk file when it's size <= this value
# default value is 16MB
# since V3.00
slot_max_size = 16MB //slot最大大小

# the trunk file size, should >= 4MB
# default value is 64MB
# since V3.00
trunk_file_size = 64MB //trunk文件大小

# HTTP settings
http.disabled=false //HTTP服务是否生效

# HTTP port on this tracker server
http.server_port=8080 //HTTP服务端口

# check storage HTTP server alive interval seconds
# <= 0 for never check
# default value is 30
http.check_alive_interval=30 //检测HTTP服务器存活时间间隔

# check storage HTTP server alive type, values are:
#   tcp : connect to the storge server with HTTP port only,
#        do not request and get response
#   http: storage check alive url must return http status 200
# default value is tcp
http.check_alive_type=tcp //检测类型

# check storage HTTP server alive uri/url
# NOTE: storage embed HTTP server support uri: /status.html
http.check_alive_uri=/status.html //检测的uri

# if need find content type from file extension name
http.need_find_content_type=true //是否需要从文件扩展名找内容类型

#use "#include" directive to include http other settings
##include http.conf //引用http.conf文件

【storage.conf】
# is this config file disabled
# false for enabled
# true for disabled
disabled=false //配置文件是否生效,false生效，true不生效

# the name of the group this storage server belongs to
group_name=group1 //指定此storage server所属组

# bind an address of this host
# empty for bind all addresses of this host
bind_addr= //绑定ip

# if bind an address of this host when connect to other servers
# (this storage server as a client)
# true for binding the address configed by above parameter: "bind_addr"
# false for binding any address of this host
client_bind=true //当指定bind_addr时，此参数才有效

# the storage server port
port=23000 //指定端口

# connect timeout in seconds
# default value is 30s
connect_timeout=30 //连接超时时间，针对socket

# network timeout in seconds
# default value is 30s
network_timeout=60 //storage server网络超时时间

# heart beat interval in seconds
heart_beat_interval=30 //心跳时间间隔

# disk usage report interval in seconds
stat_report_interval=60 //storage server向tracker server报告磁盘剩余大小的时间间隔

# the base path to store data and log files
base_path=/data/fastdfs //目录地址，目录结构如下：
${base_path}
|__data
| |__.data_init_flag：当前storage server初始化信息
| |__storage_stat.dat：当前storage server统计信息
| |__sync：存放数据同步相关文件
| | |__binlog.index：当前的binlog文件索引号
| | |__binlog.###：存放更新操作记录（日志）
| | |__${ip_addr}_${port}.mark：存放同步的完成情况
| |
| |__一级目录：256个存放数据文件的目录，如：00, 1F
| |__二级目录：256个存放数据文件的目录
|__logs
|__storaged.log：storage server日志文件

# max concurrent connections server supported
# max_connections worker threads start when this service startup
max_connections=256 //最大连接数

# the buff size to recv / send data
# default value is 64KB
# since V2.00
buff_size = 256KB //接收发送数据缓冲区大小

# work thread count, should <= max_connections
# work thread deal network io
# default value is 4
# since V2.00
work_threads=4 //工作线程

# if disk read / write separated
## false for mixed read and write
## true for separated read and write
# default value is true
# since V2.00
disk_rw_separated = true //磁盘读写是否分离

# if read / write file directly
# if set to true, open file will add the O_DIRECT flag to avoid file caching
# by the file system. be careful to set this parameter.
# default value is false
disk_rw_direct = false //是否直接读写文件

# disk reader thread count per store base path
# for mixed read / write, this parameter can be 0
# default value is 1
# since V2.00
disk_reader_threads = 1 //单个base path的读线程数，读写分离时，读线程数=disk_reader_threads*store_path_count；读写混合时，读写线程数=(disk_reader_threads+disk_writer_threads)*store_path_count

# disk writer thread count per store base path
# for mixed read / write, this parameter can be 0
# default value is 1
# since V2.00
disk_writer_threads = 1 //单个base path的写线程数，写分离时，写线程数=disk_writer_threads*store_path_count；读写混合时，读写线程数=(disk_reader_threads+disk_writer_threads)*store_path_count

# when no entry to sync, try read binlog again after X milliseconds
# 0 for try again immediately (not need to wait)
sync_wait_msec=50 //同步文件时，如果从binlog中没有读到要同步的文件，X毫秒后重新读取，0表示不需要等待立即再次尝试读取

# after sync a file, usleep milliseconds
# 0 for sync successively (never call usleep)
sync_interval=0 //同步完上一个文件后，再同步下一个文件的时间间隔，0表示连续同步

# storage sync start time of a day, time format: Hour:Minute
# Hour from 0 to 23, Minute from 0 to 59
sync_start_time=00:00 //storage server同步开始时间

# storage sync end time of a day, time format: Hour:Minute
# Hour from 0 to 23, Minute from 0 to 59
sync_end_time=23:59 //storage server同步结束时间

# write to the mark file after sync N files
# default value is 500
write_mark_file_freq=500 //同步N个文件后写入到mark文件

# path(disk or mount point) count, default value is 1
store_path_count=1 //storage server支持多个路劲，指定存放文件的base path数目

# store_path#, based 0, if store_path0 not exists, it's value is base_path
# the paths must be exist
store_path0=/data/fastdfs
#store_path1=/data/fastdfs2 //存放数据文件目录

# subdir_count * subdir_count directories will be auto created under each
# store_path (disk), value can be 1 to 256, default value is 256
subdir_count_per_path=256 //存放文件的目录个数。Fastdfs存储文件时，采用两级目录

# tracker_server can ocur more than once, and tracker_server format is
# "host:port", host can be hostname or ip address
tracker_server=192.168.209.121:22122 //指定tracker server地址

#unix group name to run this program,
#not set (empty) means run by the group of current user
run_by_group=

#unix username to run this program,
#not set (empty) means run by current user
run_by_user=

# the mode of the files distributed to the data path
# 0: round robin(default)
# 1: random, distributted by hash code
file_distribute_path_mode=0 //文件分散存储方式，0轮流，1随机

# valid when file_distribute_to_path is set to 0 (round robin),
# when the written file count reaches this number, then rotate to next path
# default value is 100
file_distribute_rotate_count=100 //当file_distribute_to_path=0时，此参数有效。当一个目录下的文件数量达到该值时，后续上传的文件存储到下一个目录中

# call fsync to disk when write big file
# 0: never call fsync
# other: call fsync when written bytes >= this bytes
# default value is 0 (never call fsync)
fsync_after_written_bytes=0 //当写入大文件时，每写入N个字节，调用一次系统函数fsync将内容强制同步到磁盘。0表示不调用fsync

# sync log buff to disk every interval seconds
# default value is 10 seconds
sync_log_buff_interval=10 //同步日志到磁盘的时间间隔

# sync binlog buff / cache to disk every interval seconds
# this parameter is valid when write_to_binlog set to 1
# default value is 60 seconds
sync_binlog_buff_interval=10 //同步binlog到磁盘的时间间隔

# sync storage stat info to disk every interval seconds
# default value is 300 seconds
sync_stat_file_interval=300 //把storage的状态信息同步到磁盘的时间间隔

# thread stack size, should >= 512KB
# default value is 512KB
thread_stack_size=512KB //线程堆大小

# the priority as a source server for uploading file.
# the lower this value, the higher its uploading priority.
# default value is 10
upload_priority=10 //上传优先级，值越小优先级越高

# the NIC alias prefix, such as eth in Linux, you can see it by ifconfig -a
# multi aliases split by comma. empty value means auto set by OS type
# default values is empty
if_alias_prefix=

# if check file duplicate, when set to true, use FastDHT to store file indexes
# 1 or yes: need check
# 0 or no: do not check
# default value is 0
check_file_duplicate=0 //是否检测上传文件已经存在。如果设置为true需要使用fastdht。如果已经存在，则建立一个符号链接以节省磁盘空间

# namespace for storing file indexes (key-value pairs)
# this item must be set when check_file_duplicate is true / on
key_namespace=FastDFS //当check_file_duplicate=1时，在fastdht中的命名空间

# set keep_alive to 1 to enable persistent connection with FastDHT servers
# default value is 0 (short connection)
keep_alive=0 //与fastdht server的连接方式。0短连接，1长连接

# you can use "#include filename" (not include double quotes) directive to
# load FastDHT server list, when the filename is a relative path such as
# pure filename, the base path is the base path of current/this config file.
# must set FastDHT server list when check_file_duplicate is true / on
# please see INSTALL of FastDHT for detail
##include /home/yuqing/fastdht/conf/fdht_servers.conf

#HTTP settings
http.disabled=false

# use the ip address of this storage server if domain_name is empty,
# else this domain name will ocur in the url redirected by the tracker server
http.domain_name=

# the port of the web server on this storage server
http.server_port=8888

http.trunk_size=256KB

# if need find content type from file extension name
http.need_find_content_type=true

#use "#include" directive to include HTTP other settings
##include http.conf

配置实例
【tracker server】
disabled=false
bind_addr=
port=22122
connect_timeout=30
network_timeout=60
base_path=/data/fastdfs
max_connections=256
work_threads=4
store_lookup=2
store_group=group2
store_server=0
store_path=0
download_server=0
reserved_storage_space = 4GB
log_level=info
run_by_group=
run_by_user=
allow_hosts=*
sync_log_buff_interval = 10
check_active_interval = 120
thread_stack_size = 64KB
storage_ip_changed_auto_adjust = true
storage_sync_file_max_delay = 86400
storage_sync_file_max_time = 300
use_trunk_file = false
slot_min_size = 256
slot_max_size = 16MB
trunk_file_size = 64MB

【storage server】
disabled=false
group_name=group1
bind_addr=192.168.1.101
client_bind=true
port=23000
connect_timeout=30
network_timeout=60
heart_beat_interval=30
stat_report_interval=60
base_path=/data/fastdfs
max_connections=256
buff_size = 256KB
work_threads=4
disk_rw_separated = true
disk_rw_direct = false
disk_reader_threads = 1
disk_writer_threads = 1
sync_wait_msec=50
sync_interval=0
sync_start_time=00:00
sync_end_time=23:59
write_mark_file_freq=500
store_path_count=1
store_path0=/data/fastdfs
subdir_count_per_path=256
tracker_server=192.168.1.114:22122
log_level=info
run_by_group=
run_by_user=
allow_hosts=*
file_distribute_path_mode=0
file_distribute_rotate_count=100
fsync_after_written_bytes=0
sync_log_buff_interval=10
sync_binlog_buff_interval=10
sync_stat_file_interval=300
thread_stack_size=512KB
upload_priority=10
if_alias_prefix=
check_file_duplicate=0
key_namespace=FastDFS
keep_alive=0

启动
【tracker server】
# /usr/local/bin/fdfs_trackerd /etc/fdfs/tracker.conf
【storage server】
# /usr/local/bin/fdfs_storaged /etc/fdfs/storage.conf

安装nginx的fastdfs-nginx-module模块
# wget http://nginx.org/download/nginx-1.0.8.tar.gz
# tar zxvf nginx-1.0.8.tar.gz -C ../software/
# wget http://fastdfs-nginx-module.googlecode.com/files/fastdfs-nginx-module_v1.08.tar.gz
# tar zxvf fastdfs-nginx-module_v1.08.tar.gz -C ../software/
# apt-get install libpcre3 libpcre3-dev openssl libssl-dev
# ./configure --prefix=/usr/local/nginx-1.0.8 --user=nginx --group=nginx --add-module=/usr/local/src/software/fastdfs-nginx-module/src/
# make
# make install
# cp mod_fastdfs.conf /etc/fdfs/
# vi nginx.conf添加如下内容
location /M00 {
            root /data/fastdfs/data;
            ngx_fastdfs_module;
        }
# ln -s /data/fastdfs/data /data/fastdfs/data/M00
# vim mod_fastdfs.conf
connect_timeout=2
network_timeout=30
base_path=/data/fastdfs
tracker_server=192.168.1.114:22122
storage_server_port=23000
group_name=group1
url_have_group_name = false
store_path_count=1
store_path0=/data/fastdfs
log_level=debug
log_filename=
response_mode=redirect
if_alias_prefix=
http.need_find_content_type=false
# /usr/local/nginx-1.0.8/sbin/nginx -c /usr/local/nginx-1.0.8/conf/nginx.conf
ngx_http_fastdfs_set pid=28525
[2011-10-22 16:39:53] INFO - fastdfs apache / nginx module v1.08, response_mode=redirect, base_path=/data/fastdfs, path_count=1, connect_timeout=2, network_timeout=30, tracker_server_count=1, storage_server_port=23000, group_name=group1, if_alias_prefix=, local_host_ip_count=2, need_find_content_type=0, default_content_type=, anti_steal_token=0, token_ttl=0s, anti_steal_secret_key length=0, token_check_fail content_type=, token_check_fail buff length=0, storage_sync_file_max_delay=86400s

上传文件
# vim client.conf
connect_timeout=30
network_timeout=60
base_path=/data/fastdfs
tracker_server=192.168.1.114:22122
log_level=info

# /usr/local/bin/fdfs_test /etc/fdfs/client.conf upload .bashrc

[2011-10-22 17:24:49] INFO - base_path=/data/fastdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0

tracker_query_storage_store_list_without_group:
server 1. group_name=group1, ip_addr=192.168.1.101, port=23000

group_name=group1, ip_addr=192.168.1.101, port=23000
storage_upload_by_filename
group_name=group1, remote_filename=M00/00/00/wKgBZU6ii-GamB5eAAALcAZ5KwI.bashrc
source ip address: 192.168.1.101
file timestamp=2011-10-22 17:24:49
file size=2928
file crc32=108604162
file url: http://192.168.1.114/group1/M00/00/00/wKgBZU6ii-GamB5eAAALcAZ5KwI.bashrc
storage_upload_slave_by_filename
group_name=group1, remote_filename=M00/00/00/wKgBZU6ii-GamB5eAAALcAZ5KwI_big.bashrc
source ip address: 192.168.1.101
file timestamp=2011-10-22 17:24:49
file size=2928
file crc32=108604162
file url: http://192.168.1.114/group1/M00/00/00/wKgBZU6ii-GamB5eAAALcAZ5KwI_big.bashrc

用浏览器访问
http://192.168.1.101/M00/00/00/wKgBZU6ii-GamB5eAAALcAZ5KwI_big.bashrc