ceph_cephfs resource tries to mount again (mons order changed)

[klamontagne]

I currently have the following mounted fs in my fstab:

1.2.3.1:6789,1.2.3.3:6789,1.2.3.4:6789,1.2.3.2:6789:/ /media/ceph ceph _netdev,name=cephfs.mymachine,secretfile=/etc/ceph/ceph.client.cephfs.mymachine.secret 0 0

chef-client tries to mount the "new" fs and mount exits unsuccessfuly:

[2014-12-03T14:50:19-05:00] ERROR: mount[mount /media/ceph] (/var/chef/cache/cookbooks/ceph/providers/cephfs.rb line 35) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '16'
---- Begin output of mount -t ceph -o _netdev,name=cephfs.mymachine,secretfile=/etc/ceph/ceph.client.cephfs.mymachine.secret 1.2.3.1:6789,1.2.3.4:6789,1.2.3.3:6789,1.2.3.2:6789:/ /media/ceph ----
STDOUT: mount error 16 = Device or resource busy
STDERR:
---- End output of mount -t ceph -o _netdev,name=cephfs.mymachine,secretfile=/etc/ceph/ceph.client.cephfs.mymachine.secret 1.2.3.1:6789,1.2.3.4:6789,1.2.3.3:6789,1.2.3.2:6789:/ /media/ceph ----
Ran mount -t ceph -o _netdev,name=cephfs.mymachine,secretfile=/etc/ceph/ceph.client.cephfs.mymachine.secret 1.2.3.1:6789,1.2.3.4:6789,1.2.3.3:6789,1.2.3.2:6789:/ /media/ceph returned 16
[2014-12-03T14:50:19-05:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

It seems that the order of mons is not kept between runs. (1,3,4,2 vs 1,4,3,2)

[hufman]

Feel free to send a PR that sorts the mons, as a workaround. The cookbook should learn how to delete the old line from fstab and add a new line with the new mons in case the actual mons change, but I'm not sure how to do that.

[klamontagne]

The line in fstab could be tagged with an option or a comment, but I can't think of a way to do that with the mount resource. The mount point itself could be considered such a tag.

There are a couple of "clever" ways to work around the problem in a wrapper cookbook, such as

• Generate a unique name for the mountpoint and use symlinks;
• save the mons list in a separate attribute and remove the mount if it changed;
• Use a set of hostnames for mon_adresses and change the coresponding IPs without ever touching the mount.

[hufman]

The :mount action could be disabled if it manually checks that there is something mounted at that mountpoint. That will prevent problems of busy mounts when switching mons around.
Before the mount:enable action happens, it could clip out any existing ceph mounts for the same mountpoint from the fstab file.
Changing from kcephfs to fuse.ceph, for example, wouldn't be very automated. A user could make their own wrapper cookbook to run once and unmount the old mode. Otherwise, I think it would be safe to just edit fstab and then rely on reboots or have the user manually remount. I think that's about all it can do towards #171

What do you think?

[klamontagne]

I do indeed have a basic workaround in my wrapper for this recurring problem.

mountpath = Pathname.new(node['ceph']['cephfs_mount'])

ceph_cephfs node['ceph']['cephfs_mount'] do
  use_fuse node['ceph']['cephfs_use_fuse']
  if mountpath.mountpoint?
    action :enable
  else
    action [:enable, :mount]
  end
end