Performance differences on RHEL vs Windows: RHEL performance is abysmal

[jaypatrick]

We are in the process of finding a suitable linux distribution to use for AGH instances (to save on licensing costs vs Windows, plus the way AGH handles permissions on Windows is unacceptable for anybody but home users or hobbyists) and we've noticed that the performance on RHEL is 10x-20x worse than on Windows.

The average response times on Windows were consistently in the 100-200ms range, whereas on RHEL, we're seeing an average of 2500ms (that's 2.5 seconds, which is pretty awful.) Many requests end up timing out.

The topology is 2 instances of AGH on RHEL sitting behind 2 domain controllers which have the RHEL instances set up as forwarders. AGH on RHEL is set up to use DoH to private AdGuard DNS servers.

We cannot tell which hop the delay is happening on: The hop from DC to AGH/RHEL or the hop from AGH/RHEL to AdGuard DNS servers. All settings are identical (literally used the same .yaml config except for the upstream server addresses) yet consistently we are seeing this kind of performance difference.

Any ideas as to what might be going on? We can't continue using Windows due to licensing costs. These are two identically configured instances of RHEL running as VMs in Hyper-V on Windows Server 2022 Datacenter (this is our test environment) that are set up to only serve AGH DNS requests.