Make proxy dump print out more meaningful information.

[thananon]

Details

Proxy dump is a debugging tool to print out the current state and outstanding operations of the proxy. Proxy dump was not working properly prior to this patch. This PR made some modification to proxy dump code. This does not affect the communication logic in anyway.

Example:

[0-27|1780| Coll->Send 0G/2]  (send to rank 0 / state: waiting on GPU / on channel: 2)
| `-> [0-71|1782| Coll->Send 0/2]
| `-> [0-150|1784| Coll->Send 0/2]
| `-> [0-45|1786| Coll->Send 0/2]
| `-> [0-61|1788| Coll->Send 0/2]
| `-> [0-68|1790| Coll->Send 0/2]
| `-> [0-62|1792| Coll->Send 0/2]

Note that NCCL 2.23.4 made improvements in this area. We should revisit this when we eventually sync to that NCCL version.

This proves to be useful for debugging large scale job for us, and our partner are asking us to put this in as soon as possible.

Work item: : HPEXA-63 / LWPCOMMLIBS-559

What were the changes?
Change how proxy dump prints out information (added more info).

Why were the changes made?
This proxy dump is useful in debugging large scale multi-node hang.

How was the outcome achieved?
printf

Additional Documentation:
Note that NCCL 2.23.4 made improvements in this area. We should revisit this when we eventually sync to that NCCL version.

Proxy dump is not used in normal circumstance. This only affects debug instance when we set NCCL_DUMP_PROXY_SIGNAL.

Approval Checklist

Do not approve until these items are satisfied.

• Verify the CHANGELOG has been updated, if
  • there are any NCCL API version changes,
  • any changes impact library users, and/or
  • any changes impact any other ROCm library.

[wenkaidu]

what's wrong with original "op->pattern == ncclPatternRecv" checking?

[wenkaidu]

There are many "patterns". Why remove the original checking?

[thananon]

ncclPatternSend/Recv is used in only ncclSend/ncclRecv. In this PR, we want to see all the send/recv inside a collective operation (such as allreduce) and this pattern check prevents it.

[wenkaidu]

I am not sure I understand the reason. If this PR is about dump proxy information, it should not change proxy code handling itself. Unless original proxy code was wrong here? We can add proxy dump at all if/else if/else conditions without changing logic of proxy code.