NoSQL数据库之Memcached的认识及安装使用,本章节从基本概念、工作原理、yum安装、编译安装等角度介绍 Memcached,让我们对 NoSQL 类数据库有基本的了解。

1. 概述

1.1 NoSQL介绍

NoSQL是对 Not Only SQL、非传统关系型数据库的统称。NoSQL一词诞生于1998年,2009年这个词汇被再次提出指非关系型、分布式、不提供ACID的数据库设计模式。随着互联网时代的到来,数据爆发式增长,数据库技术发展日新月异,要适应新的业务需求。而随着移动互联网、物联网的到来,大数据的技术中NoSQL也同样重要。

数据库排名:https://db-engines.com/en/ranking

NoSQL 分类

  • Key-value Store k/v数据库
  • 性能好 O(1) , 如: redis、memcached
  • Document Store 文档数据库
  • mongodb、CouchDB
  • Column Store 列存数据库,Column-Oriented DB
  • HBase、Cassandra,大数据领域应用广泛
  • Graph DB 图数据库
  • Neo4j
  • Time Series 时序数据库
  • InfluxDB、Prometheus

1.2 认识NoSQL之Memcached

Memcached 只支持能序列化的数据类型,不支持持久化,基于Key-Value的内存缓存系统。

memcached 虽然没有像redis所具备的数据持久化功能,比如RDB和AOF都没有,但是可以通过做集群同步的方式,让各memcached服务器的数据进行同步,从而实现数据的一致性,即保证各memcached 的数据是一样的,即使有任何一台 memcached 发生故障,只要集群中有一台 memcached 可用就不会出现数据丢失,当其他memcached 重新加入到集群的时候,可以自动从有数据的memcached 当中自动获取数据并提供服务。

Memcached 借助了操作系统的 libevent 工具做高效的读写。libevent是个程序库,它将Linux的epoll、BSD类操作系统的kqueue等事件处理功能封装成统一的接口。即使对服务器的连接数增加,也能发挥高性能。memcached使用这个libevent库,因此能在Linux、BSD、Solaris等操作系统上发挥其高性能。

Memcached 支持最大的内存存储对象为1M,超过1M的数据可以使用客户端压缩或拆分报包放到多个key中,比较大的数据在进行读取的时候需要消耗的时间比较长,memcached 最适合保存用户的session实现session共享 Memcached存储数据时, Memcached会去申请1MB的内存, 把该块内存称为一个slab, 也称为一个page。

Memcached 支持多种开发语言,包括:JAVA,C,Python,PHP,C#,Ruby,Perl等。

Memcached 官网:http://memcached.org/

NoSQL数据库之Memcached的认识及安装使用_memcached


1.3 Memcached 和 Redis 比较

比较 项

Redis

memcached

支持的数据结构

哈希、列表、集合、有序集

纯kev-value

持久化

高可用

redis支持集群功能,可以实现主动复制,读写分离。官方也提供了sentinel集群管理工具,能够实现主从服务监控,故障自动转移,这一切,对于客户端都是透明的,无需程序改动,也无需人工介入

需要二次开发

存储value容量

最大512M

最大1M

内存分配

临时申请空间,可能导致碎片

预分配内存池的方式管理内存,能够省去内存分配时间

虚拟内存使用

有自己的VM机制,理论上能够存储比物理内存更多的数据,当数据超量时,会引发swap,把冷数据刷到磁盘上

所有的数据存储在物理内存里

网络模型

非阻塞IO复用模型,提供一些非KV存储之外的排序,聚合功能,在执行这些功能时,复杂的CPU计算,会阻塞整个IO调度

非阻塞IO复用模型

水平扩展

redis cluster 可以横向扩展

暂无

多线程

Redis6.0之前是只支持单线程

Memcached支持多线程,CPU利用方面Memcache优于Redis

过期策略

有专门线程,清除缓存数据

懒淘汰机制:每次往缓存放入数据的时候,都会存一个时间,在读取的时候要和设置的时间做TTL比较来判断是否过期

单机QPS

约10W

约60W

源代码可读性

代码清爽简洁

代码清爽简洁可能是考虑了太多的扩展性,多系统的兼容性,代码不清爽

使用场景

复杂数据结构、有持久化、高可用需求、value存储内容较大

纯KV,数据量非常大,并发量非常大的业务

2. Memcached 工作机制

2.1 内存分配机制

应用程序运行需要使用内存存储数据,但对于一个缓存系统来说,申请内存、释放内存将十分频繁,非常容易导致大量内存碎片,最后导致无连续可用内存可用。

Memcached采用了Slab Allocator机制来分配、管理内存。

  • Page:分配给Slab的内存空间,默认为1MB,分配后就得到一个Slab。Slab分配之后内存按照固定字节大小等分成chunk。
  • Chunk:用于缓存记录k/v值的内存空间。Memcached会根据数据大小选择存到哪一个chunk中,假设chunk有128bytes、64bytes等多种,数据只有100bytes存储在128bytes中,存在少许浪费。Chunk最大就是Page的大小,即一个Page中就一个Chunk
  • Slab Class:Slab按照Chunk的大小分组,就组成不同的Slab Class, 第一个Chunk大小为 96B的Slab为Class1,Chunk 120B为Class 2,如果有100bytes要存,那么Memcached会选择下图中SlabClass 2 存储,因为它是120bytes的Chunk。Slab之间的差异可以使用Growth Factor 控制,默认1.25。

NoSQL数据库之Memcached的认识及安装使用_安装 使用_02

2.2 懒过期 Lazy Expiration

memcached不会监视数据是否过期,而是在取数据时才看是否过期,如果过期,把数据有效期限标识为0,并不清除该数据。以后可以覆盖该位置存储其它数据。

2.3 LRU

当内存不足时,memcached会使用LRU(Least Recently Used)机制来查找可用空间,分配给新记录使用。

2.4 集群

  • Memcached集群,称为基于客户端的分布式集群,即由客户端实现集群功能,即Memcached本身不支持集群。
  • Memcached集群内部并不互相通信,一切都需要客户端连接到Memcached服务器后自行组织这些节点,并决定数据存储的节点。

3. 安装及使用

3.1 yum 安装

[root@CentOS84-IP08 ]#dnf info memcached
[root@CentOS84-IP08 ]#dnf -y install memcached
[root@CentOS84-IP08 ]#rpm -ql memcached
/etc/sysconfig/memcached
/usr/bin/memcached
/usr/bin/memcached-tool
/usr/lib/.build-id
/usr/lib/.build-id/25
/usr/lib/.build-id/25/9ffe16926831a542c5c1a8fbdfdf063429439e
/usr/lib/systemd/system/memcached.service
/usr/share/doc/memcached
/usr/share/doc/memcached/AUTHORS
/usr/share/doc/memcached/CONTRIBUTORS
/usr/share/doc/memcached/COPYING
/usr/share/doc/memcached/ChangeLog
/usr/share/doc/memcached/NEWS
/usr/share/doc/memcached/README.md
/usr/share/doc/memcached/new_lru.txt
/usr/share/doc/memcached/protocol-binary-range.txt
/usr/share/doc/memcached/protocol-binary.txt
/usr/share/doc/memcached/protocol.txt
/usr/share/doc/memcached/readme.txt
/usr/share/doc/memcached/storage.txt
/usr/share/doc/memcached/threads.txt
/usr/share/doc/memcached/tls.txt
/usr/share/man/man1/memcached-tool.1.gz
/usr/share/man/man1/memcached.1.gz
[root@CentOS84-IP08 ]#rpm -qa memcached
memcached-1.5.22-2.el8.x86_64
[root@CentOS84-IP08 ]#rpm -qi memcached
Name : memcached
Epoch : 0
Version : 1.5.22
Release : 2.el8
Architecture: x86_64
Install Date: Thu 14 Apr 2022 07:30:40 AM CST
Group : System Environment/Daemons
Size : 414767
License : BSD
Signature : RSA/SHA256, Thu 18 Jun 2020 01:57:20 AM CST, Key ID 05b555b38483c65d
Source RPM : memcached-1.5.22-2.el8.src.rpm
Build Date : Thu 18 Jun 2020 01:19:21 AM CST
Build Host : x86-02.mbox.centos.org
Relocations : (not relocatable)
Packager : CentOS Buildsys <bugs@centos.org>
Vendor : CentOS
URL : https://www.memcached.org/
Summary : High Performance, Distributed Memory Object Cache
Description :
memcached is a high-performance, distributed memory object caching
system, generic in nature, but intended for use in speeding up dynamic
web applications by alleviating database load.

[root@CentOS84-IP08 ]#cat /etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-l 127.0.0.1,::1"

# 查看yum安装的默认服务启动文件去除注释行后实际生效的信息
[root@CentOS84-IP08 ]#grep -Ev "^#|^$" /usr/lib/systemd/system/memcached.service
[Unit]
Description=memcached daemon
Before=httpd.service
After=network.target
[Service]
EnvironmentFile=/etc/sysconfig/memcached
ExecStart=/usr/bin/memcached -p ${PORT} -u ${USER} -m ${CACHESIZE} -c ${MAXCONN} $OPTIONS
PrivateTmp=true
ProtectSystem=full
NoNewPrivileges=true
PrivateDevices=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
[Install]
WantedBy=multi-user.target
[root@CentOS84-IP08 ]#

# 查看yum安装的默认服务启动文件全部信息
[root@CentOS84-IP08 ]#cat /usr/lib/systemd/system/memcached.service
# It's not recommended to modify this file in-place, because it will be
# overwritten during upgrades. If you want to customize, the best
# way is to use the "systemctl edit" command to create an override unit.
#
# For example, to pass additional options, create an override unit
# (as is done by systemctl edit) and enter the following:
#
# [Service]
# Environment=OPTIONS="-l 127.0.0.1,::1"


[Unit]
Description=memcached daemon
Before=httpd.service
After=network.target

[Service]
EnvironmentFile=/etc/sysconfig/memcached
ExecStart=/usr/bin/memcached -p ${PORT} -u ${USER} -m ${CACHESIZE} -c ${MAXCONN} $OPTIONS

# Set up a new file system namespace and mounts private /tmp and /var/tmp
# directories so this service cannot access the global directories and
# other processes cannot access this service's directories.
PrivateTmp=true

# Mounts the /usr, /boot, and /etc directories read-only for processes
# invoked by this unit.
ProtectSystem=full

# Ensures that the service process and all its children can never gain new
# privileges
NoNewPrivileges=true

# Sets up a new /dev namespace for the executed processes and only adds API
# pseudo devices such as /dev/null, /dev/zero or /dev/random (as well as
# the pseudo TTY subsystem) to it, but no physical devices such as /dev/sda.
PrivateDevices=true

# Required for dropping privileges and running as a different user
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE

# Restricts the set of socket address families accessible to the processes
# of this unit. Protects against vulnerabilities such as CVE-2016-8655
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX


# Some security features are not in the older versions of systemd used by
# e.g. RHEL7/CentOS 7. The below settings are automatically edited at package
# build time to uncomment them if the target platform supports them.

# Attempts to create memory mappings that are writable and executable at
# the same time, or to change existing memory mappings to become executable
# are prohibited.
##safer##MemoryDenyWriteExecute=true

# Explicit module loading will be denied. This allows to turn off module
# load and unload operations on modular kernels. It is recommended to turn
# this on for most services that do not need special file systems or extra
# kernel modules to work.
##safer##ProtectKernelModules=true

# Kernel variables accessible through /proc/sys, /sys, /proc/sysrq-trigger,
# /proc/latency_stats, /proc/acpi, /proc/timer_stats, /proc/fs and /proc/irq
# will be made read-only to all processes of the unit. Usually, tunable
# kernel variables should only be written at boot-time, with the sysctl.d(5)
# mechanism. Almost no services need to write to these at runtime; it is hence
# recommended to turn this on for most services.
##safer##ProtectKernelTunables=true

# The Linux Control Groups (cgroups(7)) hierarchies accessible through
# /sys/fs/cgroup will be made read-only to all processes of the unit.
# Except for container managers no services should require write access
# to the control groups hierarchies; it is hence recommended to turn this
# on for most services
##safer##ProtectControlGroups=true

# Any attempts to enable realtime scheduling in a process of the unit are
# refused.
##safer##RestrictRealtime=true

# Takes away the ability to create or manage any kind of namespace
##safer##RestrictNamespaces=true

[Install]
WantedBy=multi-user.target

# 认识memcached 的一些常规信息
[root@CentOS84-IP08 ]#getent passwd memcached
memcached:x:975:974:Memcached daemon:/run/memcached:/sbin/nologin
[root@CentOS84-IP08 ]#systemctl enable --now memcached
Created symlink /etc/systemd/system/multi-user.target.wants/memcached.service → /usr/lib/systemd/system/memcached.service.
[root@CentOS84-IP08 ]#pstree -p |grep memcached
|-memcached(40702)-+-{memcached}(40703)
| |-{memcached}(40704)
| |-{memcached}(40705)
| |-{memcached}(40706)
| |-{memcached}(40707)
| |-{memcached}(40708)
| |-{memcached}(40709)
| |-{memcached}(40710)
| `-{memcached}(40711)

[root@CentOS84-IP08 ]#ss -ntlup|grep memcached
tcp LISTEN 0 1024 127.0.0.1:11211 0.0.0.0:* users:(("memcached",pid=40702,fd=27))
tcp LISTEN 0 1024 [::1]:11211 [::]:* users:(("memcached",pid=40702,fd=28))

# 修改短空监听到非127本地端口,确保其他的主机可以通过网络访问
[root@CentOS84-IP08 ]#vim /etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
#OPTIONS="-l 127.0.0.1,::1"
[root@CentOS84-IP08 ]#systemctl restart memcached
[root@CentOS84-IP08 ]#ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 1024 0.0.0.0:11211 0.0.0.0:*
LISTEN 0 1024 [::]:11211 [::]:*

[root@CentOS84-IP08 ]#vim /etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
#OPTIONS="-l 127.0.0.1,::1"
OPTIONS=""
[root@CentOS84-IP08 ]#systemctl restart memcached
[root@CentOS84-IP08 ]#ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 1024 0.0.0.0:11211 0.0.0.0:*
LISTEN 0 1024 [::]:11211 [::]:*

[root@CentOS84-IP08 ]#

3.2 memcached 编译安装

####手动编译安装的过程及修改成脚本后,通过脚本自动编译安装 memcached的过程都在下面提供了

######################################################################
## 手动编译安装过程
# 依赖包及源码包
[root@centos7 <sub>]#yum -y install gcc libevent-devel
[root@centos7 </sub>]#wget http://memcached.org/files/memcached-1.6.6.tar.gz
[root@centos7 <sub>]#tar xvf memcached-1.6.6.tar.gz
[root@centos7 </sub>]#cd memcached-1.6.6/
[root@centos7 memcached-1.6.6]#./configure --prefix=/apps/memcached
[root@centos7 memcached-1.6.6]#make && make install
[root@centos7 <sub>]#tree /apps/memcached/
/apps/memcached/
├── bin
│ └── memcached
├── include
│ └── memcached
│ └── protocol_binary.h
└── share
└── man
└── man1
└── memcached.1
6 directories, 3 files

[root@centos7 </sub>]#echo 'PATH=/apps/memcached/bin:$PATH' >
/etc/profile.d/memcached.sh
[root@centos7 <sub>]#. /etc/profile.d/memcached.sh
# 准备用户
[root@centos7 </sub>]#useradd -r -s /sbin/nologin memcached
[root@centos7 <sub>]#cat /etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-l 127.0.0.1,::1"

#默认前台执行
[root@centos7 </sub>]#memcached -u memcached -m 2048 -c 65536 -f 2 -vv

#以台方式执行
[root@centos7 <sub>]#memcached -u memcached -m 2048 -c 65536 -d

#准备service文件
[root@centos7 </sub>]#cat /lib/systemd/system/memcached.service
[Unit]
Description=memcached daemon
Before=httpd.service
After=network.target
[Service]
EnvironmentFile=/etc/sysconfig/memcached
ExecStart=/apps/memcached/bin/memcached -p ${PORT} -u ${USER} -m ${CACHESIZE} -c
${MAXCONN} $OPTIONS
[Install]
WantedBy=multi-user.target
[root@centos7 <sub>]#systemctl daemon-reload
[root@centos7 </sub>]#systemctl enable --now memcached.service
[root@centos7 <sub>]#ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 127.0.0.1:11211 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 [::1]:11211 [::]:*
LISTEN 0 128 [::]:22 [::]:*
LISTEN 0 100 [::1]:25 [::]:*
[root@centos7 </sub>]#memcached --version
memcached 1.6.6


######################################################################
#### 上面的步骤合并成的脚本,自动编译安装memcached 过程

#!/bin/bash
MEMCACHED=memcached-1.6.15
INSTALL_DIR=/apps/memcached

if which memcached &>/dev/null ; then
echo -e "\033[31mYou have installed memcached on your system,please uninstall it first!\033[0m"
exit 1
fi

yum -y install gcc libevent-devel
wget http://memcached.org/files/$MEMCACHED.tar.gz
tar xvf $MEMCACHED.tar.gz
cd $MEMCACHED/
./configure --prefix=$INSTALL_DIR
make && make install

echo PATH=$INSTALL_DIR/bin:'$PATH' > /etc/profile.d/memcached.sh
. /etc/profile.d/memcached.sh

useradd -r -s /sbin/nologin memcached

cat > /etc/sysconfig/memcached <<EOF
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS=""
EOF


cat > /lib/systemd/system/memcached.service <<EOF
[Unit]
Description=memcached daemon
Before=httpd.service
After=network.target
[Service]
EnvironmentFile=/etc/sysconfig/memcached
ExecStart=$INSTALL_DIR/bin/memcached -p \${PORT} -u \${USER} -m \${CACHESIZE} -c \${MAXCONN} \$OPTIONS
[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now memcached.service
. /etc/init.d/functions
action "Memcached编译安装成功" || action "Memcached编译安装失败"

# 检查语法,并执行
[root@CentOS84-IP18 ]#bash -n install-memcached.sh
[root@CentOS84-IP18 ]#bash install-memcached.sh

[root@CentOS84-IP18 ]#ss -tln
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 1024 0.0.0.0:11211 0.0.0.0:*
LISTEN 0 1024 [::]:11211 [::]:*
[root@CentOS84-IP18 ]#

[root@CentOS84-IP18 ]#memcached --version
memcached 1.6.15
[root@CentOS84-IP18 ]#


3.2 memcached 使用

五种基本 memcached 命令执行最简单的操作。这些命令和操作包括:

  • set
  • add
  • replace
  • get
  • delete
#前三个命令是用于操作存储在 memcached 中的键值对的标准修改命令,都使用如下所示的语法:
command <key> <flags> <expiration time> <bytes>
<value>
#参数说明如下:
command set/add/replace
key key 用于查找缓存值
flags 可以包括键值对的整型参数,客户机使用它存储关于键值对的额外信息
expiration time 在缓存中保存键值对的时间长度(以秒为单位,0 表示永远)
bytes 在缓存中存储的字节数
value 存储的值(始终位于第二行)
#增加key,过期时间为秒,bytes为存储数据的字节数
add key flags exptime bytes
# 基本操作等
[root@CentOS84-IP18 ]#telnet localhost 11211
Trying ::1...
Connected to localhost.
Escape character is '^]'.
stats
.........

# 因子2倍 运行
[root@CentOS84-IP18 ]#memcached -u memcached -m 2048 -c 65536 -f 2 -vv
slab class 1: chunk size 96 perslab 10922
slab class 2: chunk size 192 perslab 5461
slab class 3: chunk size 384 perslab 2730
slab class 4: chunk size 768 perslab 1365
slab class 5: chunk size 1536 perslab 682
slab class 6: chunk size 3072 perslab 341
slab class 7: chunk size 6144 perslab 170
slab class 8: chunk size 12288 perslab 85
slab class 9: chunk size 24576 perslab 42
slab class 10: chunk size 49152 perslab 21
slab class 11: chunk size 98304 perslab 10
slab class 12: chunk size 196608 perslab 5
slab class 13: chunk size 524288 perslab 2
failed to listen on TCP port 11211: Address already in use
[root@CentOS84-IP18 ]#

# 后台方式运行
[root@CentOS84-IP18 ]#memcached -u memcached -m 2048 -c 65536 -d

# 因子 1.25
[root@CentOS84-IP18 ]#memcached -u memcached -m 2048 -c 65536 -f 1.25 -vv
slab class 1: chunk size 96 perslab 10922
slab class 2: chunk size 120 perslab 8738
slab class 3: chunk size 152 perslab 6898
slab class 4: chunk size 192 perslab 5461
slab class 5: chunk size 240 perslab 4369
slab class 6: chunk size 304 perslab 3449
slab class 7: chunk size 384 perslab 2730
slab class 8: chunk size 480 perslab 2184
slab class 9: chunk size 600 perslab 1747
slab class 10: chunk size 752 perslab 1394
slab class 11: chunk size 944 perslab 1110
slab class 12: chunk size 1184 perslab 885
slab class 13: chunk size 1480 perslab 708
slab class 14: chunk size 1856 perslab 564
slab class 15: chunk size 2320 perslab 451
slab class 16: chunk size 2904 perslab 361
slab class 17: chunk size 3632 perslab 288
slab class 18: chunk size 4544 perslab 230
slab class 19: chunk size 5680 perslab 184
slab class 20: chunk size 7104 perslab 147
slab class 21: chunk size 8880 perslab 118
slab class 22: chunk size 11104 perslab 94
slab class 23: chunk size 13880 perslab 75
slab class 24: chunk size 17352 perslab 60
slab class 25: chunk size 21696 perslab 48
slab class 26: chunk size 27120 perslab 38
slab class 27: chunk size 33904 perslab 30
slab class 28: chunk size 42384 perslab 24
slab class 29: chunk size 52984 perslab 19
slab class 30: chunk size 66232 perslab 15
slab class 31: chunk size 82792 perslab 12
slab class 32: chunk size 103496 perslab 10
slab class 33: chunk size 129376 perslab 8
slab class 34: chunk size 161720 perslab 6
slab class 35: chunk size 202152 perslab 5
slab class 36: chunk size 252696 perslab 4
slab class 37: chunk size 315872 perslab 3
slab class 38: chunk size 394840 perslab 2
slab class 39: chunk size 524288 perslab 2
failed to listen on TCP port 11211: Address already in use