Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle the rbd image not found error #204

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Handle the rbd image not found error #204

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 35 additions & 32 deletions ceph_iscsi_config/lun.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1231,38 +1231,41 @@ def define_luns(logger, config, target):
for disk_key in pool_disks:

pool, image_name = disk_key.split('/')
with rbd.Image(ioctx, image_name) as rbd_image:

disk_config = config.config['disks'][disk_key]
backstore = disk_config['backstore']
backstore_object_name = disk_config['backstore_object_name']

lun = LUN(logger, pool, image_name,
rbd_image.size(), local_gw, backstore,
backstore_object_name)

if lun.error:
raise CephiSCSIError("Error defining rbd image {}"
.format(disk_key))

so = lun.allocate()
if lun.error:
raise CephiSCSIError("Unable to register {} "
"with LIO: {}"
.format(disk_key,
lun.error_msg))

# If not in use by another target on this gw
# clean up stale locks.
if so.status != 'activated':
RBDDev.rbd_lock_cleanup(logger, ips,
rbd_image)

target._map_lun(config, so, target_disks[disk_key])
if target.error:
raise CephiSCSIError("Mapping for {} failed: {}"
.format(disk_key,
target.error_msg))
try:
with rbd.Image(ioctx, image_name) as rbd_image:

disk_config = config.config['disks'][disk_key]
backstore = disk_config['backstore']
backstore_object_name = disk_config['backstore_object_name']

lun = LUN(logger, pool, image_name,
rbd_image.size(), local_gw, backstore,
backstore_object_name)

if lun.error:
raise CephiSCSIError("Error defining rbd image {}"
.format(disk_key))

so = lun.allocate()
if lun.error:
raise CephiSCSIError("Unable to register {} "
"with LIO: {}"
.format(disk_key,
lun.error_msg))

# If not in use by another target on this gw
# clean up stale locks.
if so.status != 'activated':
RBDDev.rbd_lock_cleanup(logger, ips,
rbd_image)

target._map_lun(config, so, target_disks[disk_key])
if target.error:
raise CephiSCSIError("Mapping for {} failed: {}"
.format(disk_key,
target.error_msg))
except rbd.ImageNotFound:
logger.debug("Rbd image not found: '{}' ".format(image_name))

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know the answer to this, maybe someone more experienced with ceph-iscsi and rbd. From what it looks like. It's grabbing the list of disks from the ceph-iscsi config, and then opening up the pool and searching for the key/object (disk) in ceph. So the question is what is the expected behaviour if the image in the pool is gone?

Do we just log to debug? that seems like it's easily missed, or do be need to do something more noticeable for ops?

I'd assume we'd either have to use:
logger.warning

So it appears in more logs... and might even be colour coded depending on the log analyser.

Or maybe it should throw a CephISCSIError, but I assume that'll stop processing and not actually fix things, minus stop a full blown stacktrace.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original intention was to fail the service so that it could easily be seen that somehow the configuration is invalid and needs to be resolved. Just hiding it like this will still potentially leave dangling LUNs in the configuration. In this case, the easy "solution" is to just recreate the missing images and remove them from Ceph iSCSI first.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@matthewoliver @dillaman Thanks for clarifying.
Cloud you point me how to recreate missing images when the rbd-target-api is failed?
do you mean that we need to use cephcli to create a rbd image as the ceph-iscsi service is dead.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, just use rbd create --size 1G (??) --image-format exclusive-lock volumes/<image name>


def rados_pool(conf=None, pool=None):
Expand Down