hotsos on arm64 nodes is taking too long in some cases #850
Assignees
Comments
|
Is it the same when you grab a sosreport from that node and feed it to the hotsos? If so, an example sosreport would be useful to dig into this issue. |
|
I think I have some ideas as to what is contributing to the slowness and I'm working on a solution atm. Will have something up soon. |
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 2, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches for a given file once. Related-to: canonical#850
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 2, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches for a given file once. Related-to: canonical#850
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 2, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches for a given file once. Related-to: canonical#850
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 3, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches for a given file once. Related-to: canonical#850
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 3, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches for a given file once. Related-to: canonical#850
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 3, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches for a given file once. Related-to: canonical#850
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 6, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches for a given file once. Related-to: canonical#850
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 6, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches for a given file once. Related-to: canonical#850
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 6, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches prior to executing event callbacks so that the results are shared with all and avoiding any repeated searches of the same file. Also removes some extraneous regex from some search patterns. Related-To: canonical#850
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 6, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches prior to executing event callbacks so that the results are shared with all and avoiding any repeated searches of the same file. Also removes some extraneous regex from some search patterns. Related-To: canonical#850
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 7, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches prior to executing event callbacks so that the results are shared with all and avoiding any repeated searches of the same file. Also removes some extraneous regex from some search patterns. Related-To: canonical#850
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 7, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches prior to executing event callbacks so that the results are shared with all and avoiding any repeated searches of the same file. Also removes extraneous regex from some search patterns. Related-To: canonical#850
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 8, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches prior to executing event callbacks so that the results are shared with all and avoiding any repeated searches of the same file. Also removes extraneous regex from some search patterns. Related-To: canonical#850
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 8, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches prior to executing event callbacks so that the results are shared with all and avoiding any repeated searches of the same file. Also removes extraneous regex from some search patterns. Related-To: canonical#850
dosaboy
added a commit
to dosaboy/hotsos
that referenced
this issue
May 8, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches prior to executing event callbacks so that the results are shared with all and avoiding any repeated searches of the same file. Also removes extraneous regex from some search patterns. Related-To: canonical#850
dosaboy
added a commit
that referenced
this issue
May 8, 2024
To avoid searching the same file more than once we now gather all searches from all events and executed all searches prior to executing event callbacks so that the results are shared with all and avoiding any repeated searches of the same file. Also removes extraneous regex from some search patterns. Related-To: #850
|
I believe we should have a substantial improvement with some patches landed recently including the one above and also 0045208 and some recent changes to search. I have run some tests against small and large sosreports and am definitely seeing improvements. There is definitely a lot of room for improvement still, particularly around the speed of search execution and I now we have further changes queued up for that but these recent patch hopefully remove some slowness reduce memory consumption. |
|
We now perform all event and scenario searches (per-plugin) in one go and also allow any other searches to be performed in this way. This will remove any duplicate searches or searching of long files more than once which should result in a net improvement. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In investigating an issue with stale ports in ovn, I found that one some arm64 nodes, ovn has a lot (anywhere between 30-800) stale port entries on a host with no associated tap devices. This requires manual cleanup using ovs-vsctl like so
ovs-vsctl del-port br-int ...When trying to run hotsos on this affected node (with stale ports), hotsos takes a really long time and had to be killed.
On a non-affected node (that has 2 ports in ovn however), hotsos runs in a few seconds.
The hotsos command I am using is
hotsos --openvswitch --kernel --openstack --systemhotsos version:
1.17.0+1276~ubuntu22.04.1The text was updated successfully, but these errors were encountered: