SLOW TRANSACTION in RegisterControlStream when enrolling new device

[apmillett]

Expected behavior:
When registering a new resource (Ubuntu or Debian in my case), after running the install-node.sh on the node concerned, one expects the Next button on the Teleport server to un-grey to complete the registration.

Current behavior:
The Next button never appears so I can't complete the registration of new resources.

Bug details:

• Teleport version - 16.2.1

• Recreation steps

Register a new device (Debian or Ubuntu).

• Debug logs

On teleport, eventually, the following is displayed -

WARN [SQLITE] "SLOW TRANSACTION: 1.289639611s, goroutine 6053 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x5e\ngithub.com/gravitational/teleport/lib/backend/lite.(*Backend).inTransaction.func1()\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:946 +0x47\ngithub.com/gravitational/teleport/lib/backend/lite.(*Backend).inTransaction(0xc00173c160, {0x55b1446b4a58, 0xc004c28cc0}, 0xc003a1d108)\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:997 +0x1f5\ngithub.com/gravitational/teleport/lib/backend/lite.(*Backend).Put(0xc00173c160, {0x55b1446b4a58, 0xc004c28cc0}, {{0xc004235c50, 0x2f, 0x30}, {0xc002b3e140, 0x136, 0x136}, {0x2db81ebd, ...}, ...})\n\tgithub.com/gravitational/teleport/lib/backend/lite/lite.go:510 +0x125\ngithub.com/gravitational/teleport/lib/backend.(*Sanitizer).Put(0xc000d0ed30, {0x55b1446b4a58, 0xc004c28cc0}, {{0xc004235c50, 0x2f, 0x30}, {0xc002b3e140, 0x136, 0x136}, {0x2db81ebd, ...}, ...})\n\tgithub.com/gravitational/teleport/lib/backend/sanitize.go:98 +0x126\ngithub.com/gravitational/teleport/lib/backend.(*Reporter).Put(0xc0015c6640, {0x55b1446b4a90, 0xc00128b1d0}, {{0xc004235c50, 0x2f, 0x30}, {0xc002b3e140, 0x136, 0x136}, {0x2db81ebd, ...}, ...})\n\tgithub.com/gravitational/teleport/lib/backend/report.go:202 +0x509\ngithub.com/gravitational/teleport/lib/services/local.(*PresenceService).UpsertInstance(0xc00173a2c0, {0x55b1446b4a90, 0xc00128b1d0}, {0x55b14471e518, 0xc005411520})\n\tgithub.com/gravitational/teleport/lib/services/local/inventory.go:113 +0x4c6\ngithub.com/gravitational/teleport/lib/inventory.(*Controller).heartbeatInstanceState.func2()\n\tgithub.com/gravitational/teleport/lib/inventory/controller.go:452 +0x43\ngithub.com/gravitational/teleport/lib/inventory.(*Controller).heartbeatInstanceState.func1(0xc005411520?)\n\tgithub.com/gravitational/teleport/lib/inventory/controller.go:438 +0x7d\ngithub.com/gravitational/teleport/lib/inventory.(*Controller).heartbeatInstanceState(0xc00173c210, 0xc003140800, {0xc004b0b908?, 0x0?, 0x55b149e068a0?})\n\tgithub.com/gravitational/teleport/lib/inventory/controller.go:451 +0x3b4\ngithub.com/gravitational/teleport/lib/inventory.(*Controller).handleControlStream(0xc00173c210, 0xc003140800)\n\tgithub.com/gravitational/teleport/lib/inventory/controller.go:394 +0x771\ncreated by github.com/gravitational/teleport/lib/inventory.(*Controller).RegisterControlStream in goroutine 6049\n\tgithub.com/gravitational/teleport/lib/inventory/controller.go:273 +0x306\n." lite/lite.go:946

On the newly registered node -

root@node00:~# systemctl status teleport
● teleport.service - Teleport Service
     Loaded: loaded (/lib/systemd/system/teleport.service; enabled; preset: enabled)
     Active: active (running) since Fri 2024-09-06 14:01:20 BST; 21h ago
   Main PID: 1354036 (teleport)
      Tasks: 15 (limit: 4644)
     Memory: 73.6M
        CPU: 2min 49.395s
     CGroup: /system.slice/teleport.service
             └─1354036 /usr/local/bin/teleport start --config /etc/teleport.yaml --pid-file=/run/teleport.pid

Sep 06 14:01:21 node00 teleport[1354036]: 2024-09-06T14:01:21+01:00 INFO [UPLOAD:1]  Creating directory. pid:1354036.1 directory:/var/lib/teleport/log/upload/streaming/default service/service.go:3133
Sep 06 14:01:21 node00 teleport[1354036]: 2024-09-06T14:01:21+01:00 INFO [UPLOAD:1]  Creating directory. pid:1354036.1 directory:/var/lib/teleport/log service/service.go:3133
Sep 06 14:01:21 node00 teleport[1354036]: 2024-09-06T14:01:21+01:00 INFO [UPLOAD:1]  Creating directory. pid:1354036.1 directory:/var/lib/teleport/log/upload service/service.go:3133
Sep 06 14:01:21 node00 teleport[1354036]: 2024-09-06T14:01:21+01:00 INFO [UPLOAD:1]  Creating directory. pid:1354036.1 directory:/var/lib/teleport/log/upload/corrupted service/service.go:3133
Sep 06 14:01:21 node00 teleport[1354036]: 2024-09-06T14:01:21+01:00 INFO [UPLOAD:1]  Creating directory. pid:1354036.1 directory:/var/lib/teleport/log/upload/corrupted/default service/service.go:3133
Sep 06 14:01:21 node00 teleport[1354036]: 2024-09-06T14:01:21+01:00 INFO [UPLOAD]    uploader will scan /var/lib/teleport/log/upload/streaming/default every 5s filesessions/fileasync.go:196
Sep 06 14:01:21 node00 teleport[1354036]: 2024-09-06T14:01:21+01:00 INFO [UPLOAD:1:] upload completer will run every 5m0s events/complete.go:167
Sep 06 14:01:21 node00 teleport[1354036]: 2024-09-06T14:01:21+01:00 INFO [NODE:1:CA] Cache "node" first init succeeded. cache/cache.go:1022
Sep 06 14:01:21 node00 teleport[1354036]: 2024-09-06T14:01:21+01:00 INFO [NODE:1]    Service is starting in tunnel mode. pid:1354036.1 service/service.go:2981
Sep 06 14:01:21 node00 teleport[1354036]: 2024-09-06T14:01:21+01:00 INFO [PROC:1]    The new service has started successfully. Starting syncing rotation status. pid:1354036.1 max_retry_period:4m16s service/connect.go:700

The node also has a good connection on port 22 to the Teleport proxy/auth server -

root@node00:~# netstat -antp | grep ":22"
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1033525/sshd: /usr/ 
tcp        0      0 192.168.22.3:22         192.168.16.23:52896     ESTABLISHED 1765326/sshd: root@ 

The install log on the resource is as follows -

root@node00:~# sudo bash -c "$(curl -fsSL https://teleport.domain.com/scripts/0d6b081d71de8e087a80e5da43c36f5a/install-node.sh)"
2024-09-07 11:52:03 BST [teleport-installer] TELEPORT_VERSION: 16.2.1
2024-09-07 11:52:03 BST [teleport-installer] TARGET_HOSTNAME: teleport.domain.com
2024-09-07 11:52:03 BST [teleport-installer] TARGET_PORT: 443
2024-09-07 11:52:03 BST [teleport-installer] JOIN_TOKEN: 0d6b081d71de8e087a80e5da43c36f5a
2024-09-07 11:52:03 BST [teleport-installer] CA_PIN_HASHES: sha256:81730f2235e76e3cbaec37484d0f1446b517c60c7ce33875c2f29865e0308c0d
2024-09-07 11:52:03 BST [teleport-installer] Checking TCP connectivity to Teleport server (teleport.domain.com:443)
2024-09-07 11:52:03 BST [teleport-installer] Connectivity to Teleport server (via nc) looks good
2024-09-07 11:52:03 BST [teleport-installer] Detected host: linux-gnu, using Teleport binary type linux
2024-09-07 11:52:03 BST [teleport-installer] Detected arch: x86_64, using Teleport arch amd64
2024-09-07 11:52:03 BST [teleport-installer] Detected distro type: debian
2024-09-07 11:52:03 BST [teleport-installer] Using Teleport distribution: deb
2024-09-07 11:52:03 BST [teleport-installer] Created temp dir /tmp/teleport-6wQUQ4renz
2024-09-07 11:52:03 BST [teleport-installer] Installing repo for distro debian.
Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
Hit:2 https://apt.releases.teleport.dev/debian bookworm InRelease             
Hit:3 http://mirror.ox.ac.uk/debian bookworm InRelease                        
Hit:4 http://mirror.ox.ac.uk/debian bookworm-updates InRelease               
Hit:5 https://artifacts.elastic.co/packages/8.x/apt stable InRelease
Reading package lists... Done
trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  teleport
0 upgraded, 1 newly installed, 0 to remove and 39 not upgraded.
Need to get 0 B/167 MB of archives.
After this operation, 630 MB of additional disk space will be used.
Selecting previously unselected package teleport.
(Reading database ... 67168 files and directories currently installed.)
Preparing to unpack .../teleport_16.2.1_amd64.deb ...
Unpacking teleport (16.2.1) ...
Setting up teleport (16.2.1) ...
Processing triggers for runit (2.1.2-54) ...
2024-09-07 11:52:21 BST [teleport-installer] Found: Teleport v16.2.1 git:v16.2.1-0-g4534d83 go1.22.7
2024-09-07 11:52:21 BST [teleport-installer] Writing Teleport node service config to /etc/teleport.yaml
2024-09-07 11:52:21 BST [teleport-installer] Host is using systemd
2024-09-07 11:52:21 BST [teleport-installer] Starting Teleport via systemd. It will automatically be started whenever the system reboots.

Teleport has been started.

View its status with 'sudo systemctl status teleport.service'
View Teleport logs using 'sudo journalctl -u teleport.service'
To stop Teleport, run 'sudo systemctl stop teleport.service'
To start Teleport again if you stop it, run 'sudo systemctl start teleport.service'

You can see this node connected in the Teleport web UI or 'tsh ls' with the name 'node00'
Find more details on how to use Teleport here: https://goteleport.com/docs/user-manual/

On the Teleport server -

2024-09-07T11:52:29.780810+01:00 teleport00 teleport[485]: 2024-09-07T11:52:29+01:00 INFO [AUTH]      Instance "node00" [9c270d0c-04e9-46e4-8545-810f61a9b170] has joined the cluster. role=Node auth/join.go:474
2024-09-07T11:52:29.781517+01:00 teleport00 teleport[485]: 2024-09-07T11:52:29+01:00 INFO [AUDIT]     instance.join addr.remote:192.168.22.3 cluster_name:teleport.domain.com code:TJ002I ei:0 event:instance.join host_id:9c270d0c-04e9-46e4-8545-810f61a9b170 method:token node_name:node00 role:Node success:true time:2024-09-07T10:52:29.78Z token_expires:2024-09-07T14:43:19.187336615Z token_name:************************43c36f5a uid:5bc95e3e-2012-425d-8422-6787ccf6e4ba events/emitter.go:288
2024-09-07T11:52:30.487446+01:00 teleport00 teleport[485]: 2024-09-07T11:52:30+01:00 INFO [AUTH]      Instance "node00" [9c270d0c-04e9-46e4-8545-810f61a9b170] has joined the cluster. role=Instance, systemRoles=[Node] auth/join.go:472
2024-09-07T11:52:30.487709+01:00 teleport00 teleport[485]: 2024-09-07T11:52:30+01:00 INFO [AUDIT]     instance.join addr.remote:192.168.22.3 cluster_name:teleport.domain.com code:TJ002I ei:0 event:instance.join host_id:9c270d0c-04e9-46e4-8545-810f61a9b170 method:token node_name:node00 role:Instance success:true time:2024-09-07T10:52:30.487Z token_expires:2024-09-07T14:43:19.187336615Z token_name:************************43c36f5a uid:c7532ecb-2360-484b-a063-d9c779239765 events/emitter.go:288

Eventually, the SLOW TRANSACTION will appear.

[strideynet]

SLOW TRANSACTION can be indicative of poor database performance. Which storage backend are you using?

[apmillett]

Hi,

The storage back-end is Ceph on a multi-node Proxmox cluster.

Running iostat, we don't see anything untoward within the VM itself -

root@teleport00:~# iostat -x 10 /dev/sda
Linux 6.1.0-25-amd64 (teleport00) 	12/09/24 	_x86_64_	(4 CPU)
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.54    0.00    0.39    2.41    0.00   96.66
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.03      2.22     0.01  13.70    5.77    68.71   13.23    116.32    10.71  44.74   14.58     8.79    0.02    166.86     0.00   0.00    0.03  7324.52    5.47    1.27    0.20   9.26
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.50    0.00    0.55    0.95    0.00   98.00
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   15.70    137.60    12.80  44.91    4.25     8.76    0.00      0.00     0.00   0.00    0.00     0.00    6.60    0.20    0.07   4.08
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.63    0.00    0.55    0.70    0.00   98.12
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   13.60    118.00    10.50  43.57    4.24     8.68    0.00      0.00     0.00   0.00    0.00     0.00    4.80    0.19    0.06   2.88
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.50    0.00    0.40    0.65    0.00   98.45
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   17.20    117.20     6.90  28.63    6.43     6.81    0.00      0.00     0.00   0.00    0.00     0.00    3.00    0.20    0.11   2.80
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.50    0.00    0.43    1.05    0.00   98.02
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   14.30    124.40    11.70  45.00    4.48     8.70    0.00      0.00     0.00   0.00    0.00     0.00    6.00    0.17    0.07   4.36
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.63    0.00    0.58    1.08    0.00   97.72
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   18.30    170.40    17.80  49.31    3.95     9.31    0.00      0.00     0.00   0.00    0.00     0.00    7.90    0.19    0.07   4.40
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.43    0.00    0.33    0.58    0.00   98.67
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00    9.10     74.40     6.30  40.91    4.80     8.18    0.00      0.00     0.00   0.00    0.00     0.00    3.00    0.20    0.04   2.48
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.65    0.00    0.43    0.63    0.00   98.30
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00    9.60     86.40     8.60  47.25    4.28     9.00    0.00      0.00     0.00   0.00    0.00     0.00    4.40    0.23    0.04   2.60
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.65    0.00    0.48    0.63    0.00   98.24
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   13.00    113.60    10.20  43.97    3.72     8.74    0.00      0.00     0.00   0.00    0.00     0.00    4.90    0.18    0.05   2.56
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.65    0.00    0.48    1.15    0.00   97.72
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   16.50    146.00    13.70  45.36    5.05     8.85    0.00      0.00     0.00   0.00    0.00     0.00    7.20    0.21    0.08   5.12
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.48    0.00    0.40    1.58    0.00   97.54
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   13.90    121.60    11.60  45.49    7.60     8.75    0.00      0.00     0.00   0.00    0.00     0.00    6.00    0.18    0.11   7.32
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.50    0.00    0.48    0.68    0.03   98.32
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   12.50    114.40    11.40  47.70    4.08     9.15    0.00      0.00     0.00   0.00    0.00     0.00    5.40    0.19    0.05   3.08
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.43    0.00    0.40    0.58    0.00   98.60
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   11.10     99.60     9.50  46.12    3.36     8.97    0.00      0.00     0.00   0.00    0.00     0.00    4.80    0.17    0.04   2.28
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.55    0.00    0.43    0.83    0.00   98.20
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   13.50    114.00     9.80  42.06    4.87     8.44    0.00      0.00     0.00   0.00    0.00     0.00    5.40    0.19    0.07   4.32
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.68    0.00    0.48    0.85    0.00   97.99
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   14.00    122.00    11.50  45.10    3.76     8.71    0.00      0.00     0.00   0.00    0.00     0.00    6.00    0.18    0.05   3.48
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.58    0.00    0.40    2.51    0.00   96.52
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   13.40    122.40    11.50  46.18    9.16     9.13    0.00      0.00     0.00   0.00    0.00     0.00    6.00    0.20    0.12  10.08
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.53    0.00    0.43    0.73    0.00   98.32
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   11.90    105.20     9.90  45.41    3.90     8.84    0.00      0.00     0.00   0.00    0.00     0.00    5.40    0.20    0.05   2.88
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.90    0.00    0.43    1.63    0.00   97.04
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   23.48    212.99    19.58  45.48    4.74     9.07    0.00      0.00     0.00   0.00    0.00     0.00    9.59    0.20    0.11   7.07
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.45    0.00    0.43    0.58    0.00   98.54
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00    7.90     71.60     6.80  46.26    4.32     9.06    0.00      0.00     0.00   0.00    0.00     0.00    3.60    0.19    0.03   2.48
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.23    0.00    0.58    1.70    0.00   96.49
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   15.30    139.60    12.40  44.77    6.08     9.12    0.00      0.00     0.00   0.00    0.00     0.00    6.60    0.18    0.09   6.68
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.26    0.00    0.73    1.41    0.00   95.61
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   22.10    193.60    18.10  45.02    4.48     8.76    0.00      0.00     0.00   0.00    0.00     0.00    8.40    0.19    0.10   5.56
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.13    0.00    0.50    0.88    0.03   97.47
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   18.20    141.60    10.50  36.59    7.72     7.78    0.00      0.00     0.00   0.00    0.00     0.00    5.40    0.20    0.14   5.16
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.93    0.00    0.43    0.90    0.00   97.74
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   13.80    120.80    11.00  44.35    4.62     8.75    0.00      0.00     0.00   0.00    0.00     0.00    5.40    0.22    0.06   3.80
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.95    0.00    0.45    0.80    0.00   97.79
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   15.20    129.60    12.10  44.32    4.05     8.53    0.00      0.00     0.00   0.00    0.00     0.00    6.00    0.20    0.06   3.56
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.60    0.00    0.53    0.60    0.00   98.27
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   11.40     86.00     7.30  39.04    5.74     7.54    0.00      0.00     0.00   0.00    0.00     0.00    3.20    0.19    0.07   2.56
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.60    0.00    0.45    1.13    0.00   97.82
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   17.60    156.40    15.10  46.18    4.08     8.89    0.00      0.00     0.00   0.00    0.00     0.00    7.80    0.18    0.07   4.64
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.93    0.00    0.65    0.90    0.00   97.52
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   14.60    126.40    11.90  44.91    4.12     8.66    0.00      0.00     0.00   0.00    0.00     0.00    6.00    0.20    0.06   3.76
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.60    0.00    0.43    1.03    0.00   97.95
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   11.00     93.20     8.40  43.30    6.06     8.47    0.00      0.00     0.00   0.00    0.00     0.00    4.80    0.19    0.07   4.28
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.05    0.00    0.68    1.25    0.00   97.02
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   16.60    148.00    13.80  45.39    4.66     8.92    0.00      0.00     0.00   0.00    0.00     0.00    7.20    0.19    0.08   5.04
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.70    0.00    0.48    0.98    0.00   97.84
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   16.20    138.40    13.20  44.90    4.48     8.54    0.00      0.00     0.00   0.00    0.00     0.00    6.20    0.19    0.07   4.40
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.53    0.00    0.43    0.63    0.03   98.40
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00    7.80     66.80     5.90  43.07    6.77     8.56    0.00      0.00     0.00   0.00    0.00     0.00    3.00    0.17    0.05   3.76
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.70    0.00    0.38    1.16    0.00   97.76
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   18.20    142.40    11.70  39.13    5.47     7.82    0.00      0.00     0.00   0.00    0.00     0.00    6.00    0.18    0.10   5.08
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.68    0.00    0.53    0.68    0.00   98.12
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.00      0.00     0.00   0.00    0.00     0.00   10.50     93.20     8.50  44.74    4.57     8.88    0.00      0.00     0.00   0.00    0.00     0.00    4.20    0.21    0.05   2.76

Is there anything else you'd like to see the output of?

I'm not entirely sure when this issue started to be fair, it may have been around since 15.x, however it's only that we've started trying to add new resources that the lack of progress on adding a resource has become apparent.

[apmillett]

Downgrading back to 15.4.19 resolved my issue.