bcache手册

  • 测试平台@fedora-26;
  • centos-7 内核不包含bcache模块;bcache_on_CentOS-7
  • SSD=sdb, 后端设备=LVM, 本文后端做成3T*6的RAID10;
yum install -y bcache-tools
modprobe bcache
lsmod |grep bcache

This will typically be your mechanical drive. The backing device can be a whole device, a partition or any other standard block device.

OPTION1-创建一个常规的LVM卷作为后端设备:

# General LVM
pvcreate /dev/sd{c..h}
vgcreate vg_data /dev/sd{c..h}
lvcreate -n lv_data -l +100%FREE vg_data

OPTION2-如果服务器上有多个一样型号的硬盘,可以做成LVM的RAID10以提高后端设备的连续读写性能和冗余:

# RAID-10,3T*6
vgcreate vg_data /dev/sd{c..h}
lvcreate --type raid10 --stripes 3 -l +100%FREE --nosync --name lv_data vg_data

创建backing设备/dev/bcache01)

make-bcache -B /dev/vg_data/lv_data

创建backing设备的时候报错:

[root@r630 ~]# make-bcache -B /dev/vg_data/lv_data
Already a bcache device on /dev/vg_data/lv_data, overwrite with --wipe-bcache
[root@r630 ~]# make-bcache -B /dev/vg_data/lv_data --wipe-bcache
Can't open dev /dev/vg_data/lv_data: Device or resource busy

处理:

#这时候应该可以看到这个lv的superblock还有信息,这就是原因.
bcache-super-show /dev/vg_data/lv_data
wipefs -af /dev/vg_data/lv_data
dd if=/dev/zero of=/dev/vg_data/lv_data bs=1M count=512
reboot

This will typically be your SSD. The cache device can be a whole device, a partition or any other standard block device.

# The block size should match the backing devices sector size which will usually be either 512 or 4k. 
# The bucket size should match the erase block size of the caching device with the intent of reducing write amplification. 
# 这个不知道为啥加了参数不能关联
# make-bcache --wipe-bcache --block 4k --bucket 2M -C /dev/sdb
make-bcache -C /dev/sdb
# 有可能对应的设备已经有一些元数据,需要使用wipefs清理掉
wipefs -af /dev/sdb

完成后就多了一个cache设备 /sys/fs/bcache/<CacheSetUUID>

# 如果设备注册过了或者设备有误会报错-bash: echo: write error: Invalid argument
echo /dev/sdb > /sys/fs/bcache/register
echo /dev/vg_data/lv_data > /sys/fs/bcache/register

注册超速缓存设备和后备设备后,需要将后备设备挂接到相关的超速缓存集才能启用超速缓存:2)

# 查看cache设备
ls -d /sys/fs/bcache/*-*-* | cut -f5 -d/
# 查看后端设备
ll /dev/bcache*
# 关联cache设备和backing设备
echo cache_set_uuid > /sys/block/bcache0/bcache/attach

像普通块设备一样使用bcache设备,挂载之后还可以用fio来测试一把性能,一定会有彩蛋的!

mkfs.xfs -f /dev/bcache0
mount /dev/bcache0 /mnt

bcache 设备使用 sysfs 接口来储存其运行时配置值。这样,您便可以更改 bcache 后备设备和超速缓存磁盘的行为,或查看其使用统计数字,此链接页面尾端有SYSFS的详尽解释,主要查看 SYSFS - BACKING DEVICE、SYSFS - BACKING DEVICE STATS 和 SYSFS - CACHE DEVICE 部分。

7.1 配置信息

  • backing device: 在 /sys/block/bcache<n>/bcache/ 目录下;
  • cache device: 在 /sys/fs/bcache/<CSET-UUID>/ 目录下;
  • /sys中配置信息的改变是暂时的,重启会失效;

如果想要在启动时设置当前的配置,需要创建/etc/tmpfiles.d/bcache.conf:3)

CacheSetUUID=`ls -d /sys/fs/bcache/*-*-* | cut -f5 -d/`
cat > /etc/tmpfiles.d/bcache.conf <<EOF
w /sys/block/bcache0/bcache/cache_mode  - - - - writeback
w /sys/fs/bcache/$CacheSetUUID/congested_read_threshold_us  - - - - 0
w /sys/fs/bcache/$CacheSetUUID/congested_write_threshold_us  - - - - 0
w /sys/block/bcache0/bcache/sequential_cutoff  - - - - 4M
EOF

7.2 检查bcache状态

cat /sys/block/bcache0/bcache/state
  • no cache: this means you have not attached a caching device to your backing bcache device;
  • clean: this means everything is ok. The cache is clean;
  • dirty: this means everything is setup fine and that you have enabled writeback and that the cache is dirty;
  • inconsistent: you are in trouble because the backing device is not in sync with the caching device;
# 查看设备关联情况,有几个后端设备就有几个bcache号
bcache-status
lsblk

7.3 性能优化

# 修改bcache设备的缓存策略为writeback(性能最好)
cat /sys/block/bcache0/bcache/cache_mode
echo writeback > /sys/block/bcache0/bcache/cache_mode
# 关闭IO跟踪
echo 0 > /sys/fs/bcache/$CacheSetUUID/congested_read_threshold_us
echo 0 > /sys/fs/bcache/$CacheSetUUID/congested_write_threshold_us

By default, bcache doesn't cache everything, it tries to skip sequential IO . Because you really want to be caching the random IO, and if you copy a 10 gigabyte file you probably don't want that pushing 10 gigabytes of randomly accessed data out of your cache. But if you want to benchmark reads from cache, and you start out with fio writing an 8 gigabyte test file - so you want to disable that.

# Disable it for benchmark sequential IO
echo 0 > /sys/block/bcache0/bcache/sequential_cutoff
# To set it back to the default (4 mb)
echo 4M > /sys/block/bcache0/bcache/sequential_cutoff

7.4 bcache 维护操作

7.4.1 分离cache设备

# 查找 CacheSetUUID
ls -d /sys/fs/bcache/*-*-* | cut -f5 -d/
echo <CacheSetUUID> > /sys/block/bcache0/bcache/detach
# 注销/删除 cache 设备
echo 1 > /sys/fs/bcache/4d754740-fdbc-4a84-853f-b748389e1c60/unregister
# 查看block设备的布局变化
lsblk
# state 为none即为分离成功
cat /sys/block/bcache0/bcache/state
# 此时 bcache-status 输出中 "Cache Mode" 变为 "Various" 即为成功
bcache-status

bcache-status 输出:

数据在从SSD往backing回写
detach后block设备布局中cache设备关联消失
detacheed后CacheMode应该为"Various"

7.4.2 添加cache设备

如果unregister了cache设备,还需增加并cache设备:

make-bcache -C /dev/sdb
echo <CacheSetUUID> > /sys/block/bcache0/bcache/attach
# 查找 CacheSetUUID
ls -d /sys/fs/bcache/*-*-* | cut -f5 -d/
echo <CacheSetUUID> > /sys/block/bcache0/bcache/attach
# 查看block设备的布局变化
lsblk
# state 应该为clean
cat /sys/block/bcache0/bcache/state
bcache-status

cache设备已经关联起来

7.4.3 停止backing设备

echo 1 > /sys/block/bcache0/bcache/stop
lsblk

7.4.4 强制运行后端设备

即使cache设备掉盘,注意使用writeback模式可能导致data loss:

echo 1 > /sys/block/bcache0/bcache/running


1)
此时多了一个/dev/bcache0设备
2)
Udev rules will take care of this on reboot and will only need to be done once.
3)
必须放在该目录,文件名随意.
  • storage/缓存技术/bcache/bcache手册.txt
  • 最后更改: 2019/04/16 18:31
  • (外部编辑)