Slow virtual machine list

[notprogrammer]

Hello,

while using the Cockpit I ran into a problem with the list of virtual machines loading slowly. The update didn't help. I read in the discussions that Cockpit has a limit for managing 20 machines. Can you please confirm if this limit actually exists? I couldn't find it directly in the documentation. If so, is it possible to somehow increase or optimize it? He needs to manage up to 60 virtual servers in the cockpit, and we don't want to wait for tens of seconds to load the list of virtual servers.

This problem of virtual server list loading too slowly only occurs when the list is loaded for the first time after login. Every next load is already immediate. I am sending screenshots of what it looks like to log out and log in again on the screenshots attached to this thread as attachments.

BTW: It loads slowly for us even on a server where there are only 19 VMs running + 2 VMs turned off.

journalctl -u cockpit:

writes only cockpit-tls: gnutls_handshake failed: A TLS fatal alert, which I associate with untrusted self-signed
journalctl -u libvirtd:

-- No entries --

The event log says this:

13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:05:00.1/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:01:00.1/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:02:00.1/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:05:00.0/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:02:00.0/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:01:00.0/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:05:00.1/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:01:00.1/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:02:00.1/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:05:00.0/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:02:00.0/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:01:00.0/vpd': Permission denied
virtnodevd
13:56
Failed to open a VPD file '/sys/bus/pci/devices/0000:05:00.1/vpd': Permission denied
virtnodevd

Sample stat:

# stat /sys/bus/pci/devices/0000:05:00.1/vpd
 File: /sys/bus/pci/devices/0000:05:00.1/vpd
 Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 15h/21d Inode: 10166 Links: 1
Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:sysfs_t:s0
Access: 2024-10-16 14:45:12.595980287 +0200
Modify: 2024-10-16 14:45:12.595980287 +0200
Change: 2024-10-16 14:45:12.595980287 +0200
 Birth:-
# stat /sys/bus/pci/devices/0000:01:00.1/vpd
 File: /sys/bus/pci/devices/0000:01:00.1/vpd
 Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 15h/21d Inode: 9971 Links: 1
Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:sysfs_t:s0
Access: 2024-10-16 14:45:12.405975751 +0200
Modify: 2024-10-16 14:45:12.405975751 +0200
Change: 2024-10-16 14:45:12.405975751 +0200
 Birth:-
#

Does anyone have experience with multiple VMs in the cockpit please? Also, does the VM list take a long time to load? What could it be and how to solve it?

Thank you for your help.

[mvollmer]

Yes, cockpit-machines feels quite slow when "Loading resources". I'll have a look.

[mvollmer]

Yes, cockpit-machines feels quite slow when "Loading resources". I'll have a look.

Bad news: In my tests, these are the times from logging into Cockpit until the list of VMs is displayed:

   1 VM: 2.7 seconds
101 VMs: 3.5 seconds

So I would argue that 2.7 seconds is slow, but it doesn't get much worse when the number of VMs grows.

Could you share more about what exactly you are doing and when the slowness is there and when it isn't?

This problem of virtual server list loading too slowly only occurs when the list is loaded for the first time after login.

That's a login into Cockpit, right? There is no reboot of the physical server involved or other things, right? What kind of network is there between the web browser that runs Cockpit and the machine that you log into? (Cockpit loads the full XML for each VM, not just the name. If the network is slow enough, this will be noticeable.)

Every next load is already immediate.

What is a "next load"? Do you reload the browser? Or switch between views in the Cockpit application?

[notprogrammer]

The slowness is far away after logging in from my and Kleg's PC at home (we live 150 km apart). I am 100 km from the data center, my colleague is 50 km from the data center. The first loading of the list after login takes time, but every subsequent loading of the list is already instantaneous. But as soon as I log out and log in again, it takes a long time again. "next load" meant switching to another tab in the cockpit and returning to the virtual servers tab.

Yes, that meant logging out and logging in as a user or root user in the cockpit. We restart the server only sometimes after an update, which takes approx. once a year. However, we tried the update to see if it would speed up loading. After the update, we rebooted the server and immediately after the reboot, the list of servers loaded quickly, but that only lasted a while, after a while the list started to be slow again.

I tried removing the surrounding hosts from the cockpit and logging in as root instead of user. And the problem persists. It takes a long time to load the list for the first time after logging in. every other load is now instant until I log out and log in again, when the initial load of the list takes a long time again.

[mvollmer]

Could you run the following script to do some benchmarking? It dumps all the XML of all VMs, which is pretty much Cockpit is doing at start up.

cmd=""
for name in $(virsh list --all --name); do
  cmd="$cmd dumpxml $name; dumpxml --inactive $name; "
done
echo $cmd | virsh

Please run it in the Terminal of the Cockpit instance where you see the slow list loading, so that the same amount of data travels over the same network and also has to go through Cockpit. Doing that in the server that you have setup yesterday just for benchmarking should already be helpful.

I assume your VMs are on the "system" connection of libvirt. Is that true?

With my test setup, 100 shut-off mini VMs, this script completes in about 0.2 seconds. Cockpit itself takes about 1 second to get all its initial data (which is slightly more than just VMs) and also includes XML parsing.

My goal here is to figure out whether the slowness comes from producing and transferring this data, or whether there is something really special in your setup that makes Cockpit very slow.

[notprogrammer]

As I already wrote virsh commands are immediate.

I try this:

#!/bin/bash

# Start time measurement
start_time=$(date +%s)

# Initialize the command variable
cmd=""

# Initialize counters
running_count=0
stopped_count=0

# Iterate over all virtual machines
for name in $(virsh list --all --name); do
  # Check the state of each virtual machine
  state=$(virsh domstate "$name")
  if [[ $state == "running" ]]; then
    ((running_count++)) # Increment running counter
  elif [[ $state == "shut off" ]]; then
    ((stopped_count++)) # Increment stopped counter
  fi

  # Add commands to dump XML configurations
  cmd="$cmd dumpxml $name; dumpxml --inactive $name; "
done

# Execute the commands
echo $cmd | virsh

# End time measurement
end_time=$(date +%s)

# Calculate the total duration
duration=$((end_time - start_time))

# Display the results
echo "Processing took $duration seconds."
echo "Number of running servers: $running_count"
echo "Number of stopped servers: $stopped_count"

Result:

Processing took 1 seconds.
Number of running servers: 18
Number of stopped servers: 3

[mvollmer]

As I already wrote virsh commands are immediate.

Yes, but I didn't know which ones. :-) There is a difference in traffic volume between "virsh list" and "virsh dumpxml" and I wanted to rule out that the slowness you experience is due to a slow network or slow production of that data.

Processing took 1 seconds.
Number of running servers: 18
Number of stopped servers: 3

Thanks! I have to ask, sorry: Did you run this in the "Terminal" page of the Cockpit instance that is slow when loading the "Virtual Machines" page?

Just to make super sure, I am talking about this:

[mvollmer]

Ok, so looks like the network is not to blame. I think we have to do some profiling in Cockpit itself.

Please do this:

• Open the developer console of the Cockpit window (Control-Shift-I in Firefox and Chrome) and enter window.debugging = "machines":

• Then reload the page (Control R). You should get a lot of output in the console similar to this:

• Save all the messages and send them to me somehow. Saving can be done in Firefox with "Save all messages to file" in the context menu:

Attaching them here is fine if you are comfortable with that, or email to [email protected]. Hopefully the timestamps in those messages give a clue as to what exactly is slow.

Thanks!

[notprogrammer]

Sorry for the delay in responding.

I understand your concerns. I assure you that I used the terms inside the web UI cockpit.

I have now sent you what you requested by email.

[mvollmer]

Thanks!

The console text unfortunately has no timestamps in it... I had no idea that one needs to enable them explicitly, sorry! But this is still very useful, together with the GIF.

Is the GIF in real time? When I play it, it is 2 minutes long, and the list of VMs appears in Firefox after 1 minute. Is that how long it took in reality?

But from what I can see, everything is slow. :-( I don't see any particular thing that would block the loading process unexpectedly. I also didn't see anything out of the ordinary in the debugging messages.

The error messages in the Chromium console are indicative: a couple of DBus calls have timed out. This means that inter-process communication on medusa was very very slow, without anything outside of the medusa machine being involved, such as the network to the machine that runs the browser. (There should be a process called "cockpit-bridge" running on medusa, and that process makes these DBus calls to a libvirt daemon that is also running on medusa. The libvirt daemon takes more than 30 seconds to reply to a few of the requests, which is unexpected and indicates that medusa might be overloaded.)

No such calls have timed out for the session in the Firefox browser.

So, hmm. I think medusa might just be slow when producing the data Cockpit is asking for.

Cockpit is not helping: It makes most of its requests in parallel which helps hide network latency but does not help when the server is already at its limits.

I am afraid this is about as far as I can help you...

I am certainly interested in improving the performance of Cockpit with many VMs in a slow environment, but I am afraid there is no easy fix right now. We would have to load less at initialization time and load the rest on demand, for example.

If/when I start working on this, would you be willing to try out some development versions of the "Virtual Machines" page of Cockpit?

[notprogrammer]

@mvollmer

I may have some important news for you. On all 3 of our servers, where loading the list (the first time the page loads after logging in) was taking several minutes, I ran a single command that fixed it. On all 3 servers, the server list now loads in seconds at most, not minutes.

The mysterious command that fixed it: systemctl restart libvirt-dbus

However, I'm sure this is only a temporary solution. The last time the server list loaded quickly after a complete reboot was for about 1-2 weeks, then it started to be too slow again.

How can I help to fix the next version? I am willing and able to send you by email any logs you may want.

[mvollmer]

The mysterious command that fixed it: systemctl restart libvirt-dbus

Oh, interesting! So a long running libvirt-dbus starts to get slow. I will investigate a bit how it behaves.

I think it might be designed to timeout and quit on its own, and then get restarted automatically when a DBus request comes in from a client. Do you keep Cockpit open permanently? That might prevent libvirt-dbus from ever timing out on its own.

In any case, can you watch libvirt-dbus a bit? Does it consume more and more memory over time, for example? I am sure the libvirt people will be interested in your observations.

[mvollmer]

In the mean time, restarting libvirt-dbus (and maybe reloading Cockpit in the browser) from time to time should be harmless. But it's certainly annoying.

[notprogrammer]

@mvollmer

Can you please send me a script that I can set up as an automatic task that will collect the required memory consumption information for you?

My logs look like this:

# journalctl -u libvirt-dbus
Oct 29 20:33:36 server systemd[1]: Starting Libvirt DBus Service...
Oct 29 20:33:36 server systemd[1]: Started Libvirt DBus Service.
Nov 25 13:33:20 server systemd[1]: Stopping Libvirt DBus Service...
Nov 25 13:33:23 server systemd[1]: libvirt-dbus.service: Deactivated successfully.
Nov 25 13:33:23 server systemd[1]: Stopped Libvirt DBus Service.
Nov 25 13:33:23 server systemd[1]: libvirt-dbus.service: Consumed 2h 7min 13.450s CPU time.
Nov 25 13:33:23 server systemd[1]: Starting Libvirt DBus Service...
Nov 25 13:33:23 server systemd[1]: Started Libvirt DBus Service.
#