microceph tries to safe-to-destroy before rebalance is completed

[peppepetra]

I am removing an osd with extended timeout but that failed with:

root@machine2:~# microceph disk remove --timeout 3000 osd.2
Removing osd.2, timeout 3000s
Error: Failed to remove disk: osd.2 failed to reach safe-to-destroy

Checking the status I see that the recovery hasn't completed yet

root@machine2:~# ceph -s
  cluster:
    id:     e84b0370-654d-4fd4-8c0f-4fc11b284779
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum machine3,machine2,machine1 (age 2d)
    mgr: machine2(active, since 2d), standbys: machine3, machine1
    osd: 17 osds: 16 up (since 7m), 16 in (since 7m); 1 remapped pgs
    rgw: 3 daemons active (3 hosts, 1 zones)
 
  data:
    pools:   9 pools, 257 pgs
    objects: 75.21k objects, 219 GiB
    usage:   557 GiB used, 15 TiB / 16 TiB avail
    pgs:     1177/225633 objects misplaced (0.522%)
             256 active+clean
             1   active+remapped+backfilling
 
  io:
    client:   341 B/s rd, 2.3 MiB/s wr, 218 op/s rd, 261 op/s wr
    recovery: 7.2 MiB/s, 2 objects/s

and indeed running the safe-to-destroy while 1 pgs is backfilling fails with

root@machine2:~# ceph osd safe-to-destroy osd.2
Error EAGAIN: OSD(s) 2 have no reported stats, and not all PGs are active+clean; we cannot draw any conclusions. 

It seems like microceph is not honoring the extended timeout.

[sabaini]

PR #235 should improve this with a longer timeout and retry