OpenStack集成ceph

Ceph 不支持 QCOW2 格式的虚拟机磁盘,所以如果想要在 Ceph 中启动虚拟机( 临时后端或者从卷启动),Glance 镜像必须是 RAW 格式。

如果集成的ceph集群完全是给OpenStack使用,从降低复杂度和性能方面的考虑,建议关闭cephx认证!

vim my-cluster/ceph.conf

auth_cluster_required = none
auth_service_required = none
auth_client_required = none
# 推送配置到各个节点
ceph-deploy --overwrite-conf admin ceph01 ceph02 ceph03
# 重启所有ceph节点
systemctl restart ceph.target
# 验证cephx是否关闭
ceph --admin-daemon /var/run/ceph/ceph-mon.ceph01.asok config show |grep auth.*_required
# 验证ceph健康状态
ceph -s

OpenStack 有3个地方可以和ceph rbd结合:

  • images: 管理实例的image,image相对恒定,OpenStack把它们当作二进制文件、并以此格式下载;
  • volumes: 块设备,也就是通常说的“云数据盘”,OpenStack用它们引导虚拟机、或挂载到运行中的虚拟机上,OpenStack用cinder服务管理volumes;
  • vms: 实例的系统盘, 默认情况下启动一台虚拟机时,它的系统盘位于hypervisor:/var/lib/nova/instances/<uuid>,在此将其存入ceph;

ceph mon上创建pools:1)

# 存放"云盘"
ceph osd pool create volumes 128
# 存放镜像(ubuntu,centos,windows等系统镜像)
ceph osd pool create images 128
# 存放实例本身的系统盘
ceph osd pool create vms 128
# backups
ceph osd pool create backups 128
# 查看pools
ceph osd lspools

初始化这些rbd类型的pools:

rbd pool init volumes
rbd pool init images
rbd pool init vms
rbd pool init backups

3.1 安装ceph软件包

复制ceph节点上的ceph.repo:

ssh $ip tee /etc/yum.repos.d/ceph.repo < /etc/yum.repos.d/ceph.repo

glance-api,nova-compute,cinder-volume,cinder-backup node:

yum install -y python-ceph ceph-common python-rbd

3.2 分发ceph.conf

分发ceph.conf配置文件,ceph client才能访问ceph集群:

ssh $ip sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf

3.3 创建账号及授权

开启cephx才需要操作此章节.

3.3.1 配置

为了方便和简化配置,这里并没有单独为volumes,images等创建账号,而是统一创建openstack账号:

ceph auth get-or-create client.openstack mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=images, allow rwx pool=vms, allow rwx pool=backups'

将keyring复制到以下节点:

  • glance node;
  • cinder node;
  • cinder-backup node;
  • compute node;
ceph auth get-or-create client.openstack | ssh $ip sudo tee /etc/ceph/ceph.client.openstack.keyring
ssh $ip sudo chmod o+r /etc/ceph/ceph.client.openstack.keyring

3.3.2 libvirt注入secret

所有计算节点上的 UUID 不一定非要一样,但考虑到平台的一致性最好使用同一个UUID,建议直接使用ceph cluster的uuid.

secret key注入到每个compute node的libvirt:

cat > secret.xml <<EOF
<secret ephemeral='no' private='no'>
  <uuid>457eb676-33da-42ec-9a8c-9293d545c337</uuid>
  <usage type='ceph'>
    <name>client.openstack secret</name>
  </usage>
</secret>
EOF
 
virsh secret-define --file secret.xml
virsh secret-set-value --secret 457eb676-33da-42ec-9a8c-9293d545c337 --base64 `ceph auth get-key client.openstack`
virsh secret-list

3.3.3 验证ceph访问

在每个节点上用client.openstack用户访问ceph:

ceph -s --name client.openstack

3.4 验证是否能访问ceph集群

在controller node,compute node等节点安装了ceph client软件包,并配置了/etc/ceph/ceph.conf的情况下,就能访问ceph集群了:

ceph -s

openstack image delete <uuid>的方式删除现有的image。。

4.1 glance集成ceph

4.1.1 配置

glance可使用多种后端存储image,默认使用ceph rbd:

vim /etc/glance/glance-api.conf
 
[DEFAULT]
# image的写时复制克隆
show_image_direct_url = True
 
[glance_store]
stores = rbd
default_store = rbd
rbd_store_pool = images
# rbd_store_user = openstack	#cephx
rbd_store_ceph_conf = /etc/ceph/ceph.conf
rbd_store_chunk_size = 8
 
[paste_deploy]
# disable caching of images in /var/lib/glance/image-cache/
flavor = keystone

4.1.2 重启

systemctl restart openstack-glance-api.service openstack-glance-registry.service

4.1.3 验证操作

. admin-openrc
# download
wget http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
qemu-img convert -O raw cirros-0.4.0-x86_64-disk.img cirros-0.4.0-x86_64-disk.raw
# create image
glance image-create --name="cirros" \
  --visibility=public \
  --disk-format=raw\
  --container-format=bare \
  --file cirros-0.4.0-x86_64-disk.raw \
  --progress
 
# list images
openstack image list
# 去ceph池验证:object
rados [--name client.openstack] -p images ls
# list images
rbd [--name client.openstack] -p images ls [-l]

4.2 cinder集成ceph

OpenStack 需要一个驱动和 ceph 块设备交互,还得指定块设备所在的存储池名.

安装cinder

4.2.1 配置

vim /etc/cinder/cinder.conf
 
[DEFAULT]
enabled_backends = ceph
# 这条没搞清楚含义,先注释
#default_volume_type = ceph
# 如果cinder配置了多后端[DEFAULT]节中必须有'glance_api_version = 2'
glance_api_version = 2
 
# 文件尾端添加,[ceph]就是上文的enabled_backends = ceph
[ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1

如果使用了cephx认证,还需要配置用户及其密钥的uuid(前述文档中注入了libvirt):

[ceph]
rbd_user = openstack
rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337

4.2.2 配置cinder backup

ref

Cinder Backup 需要一个特定的守护进程,不要忘记安装它.编辑 Cinder Backup 节点的 /etc/cinder/cinder.conf 添加:

backup_driver = cinder.backup.drivers.ceph
backup_ceph_conf = /etc/ceph/ceph.conf
backup_ceph_user = openstack
backup_ceph_chunk_size = 134217728
backup_ceph_pool = backups
backup_ceph_stripe_unit = 0
backup_ceph_stripe_count = 0
restore_discard_excess_bytes = true

4.2.3 重启

systemctl restart openstack-cinder-volume

4.2.4 验证

# 创建volume
cinder create --display-name volume_test 10
# 去ceph池验证:object
rados -p volumes ls [--name client.openstack]
# list
rbd -p volumes ls [--name client.openstack]
# 查看cinder服务
cinder-manage service list

为了挂载cinder块设备(块设备或者启动卷),必须告诉cpmpute node 的 nova 挂载设备时使用的ceph配置信息。如果ceph集群开启了cephx认证,还需要额外指明ceph用户名和secret uuid,libvirt会使用该用户来和ceph集群进行连接和认证.

编辑所有compute node上的/etc/nova/nova.conf,添加如下内容:

[libvirt]
# 确保热迁移能顺利进行使用如下flag
live_migration_flag="VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST,VIR_MIGRATE_TUNNELLED"
# 禁止注入
inject_password = false
inject_key = false
inject_partition = -2
images_rbd_ceph_conf = /etc/ceph/ceph.conf
images_type=rbd
images_rbd_pool=vms
disk_cachemodes ="network=writeback"

如果启用了cephx,还要添加:

rbd_user = openstack
rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337

5.1 套接字 & cache (optional)

启用管理套接字对于故障排查来说大有好处,给每个使用ceph块设备的虚拟机分配一个套接字有助于排查性能问题和/或异常行为.

vim /etc/ceph/ceph.conf

[client]
    rbd cache = true
    rbd cache writethrough until flush = true
    admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok
    log file = /var/log/qemu/qemu-guest-$pid.log
    rbd concurrent management ops = 20

调整这些路径的权限:

mkdir -p /var/run/ceph/guests/ /var/log/qemu/
chown qemu:libvirtd /var/run/ceph/guests /var/log/qemu/

访问套接字:

ceph daemon /var/run/ceph/ceph-client.cinder.19195.32310016.asok help

要激活Ceph块设备驱动、并把块设备存储池名载入配置,必须重启OpenStack:

# controller node
systemctl restart openstack-glance-api
systemctl restart openstack-cinder-volume
systemctl restart openstack-cinder-backup
# compute node
systemctl restart openstack-nova-compute

  • virtualization/openstack/cinder/openstack_ceph.txt
  • 最后更改: 2019/04/16 18:31
  • (外部编辑)