Log all gpu rank stdout/err to MosaicML platform

[jjanezhang]

Log all gpu rank stdout/err to MosaicML platform

We are no longer differentiating between stderr and stdout for gpu rank logs.

If run is sent by Mosaic platform, redirect GPU local rank 1-7 logs to gpu_x.txt in Mosaic logs directory for each node rank.

On mosaic platform

When mosaic env var is false

[siriuslee]

Platform parts look good if the stdout / stderr change is acceptable

[dakinggg]

Does this change mean that runs on the mosaic platform will no longer show the run logs via mcli logs?

[jjanezhang]

Does this change mean that runs on the mosaic platform will no longer show the run logs via mcli logs?

@dakinggg We will still be able to see the gpu 0 rank logs (+ failed gpu rank logs when the run fails) using mcli logs for non finetuning runs. The only diff is that when your run fails, you will get the full chronological logs of all failed gpu ranks instead of separated stdout and stderr. Additionally, you will be able to get individual gpu rank logs regardless of whether the gpu rank failed or not.

[mvpatel2000]

Instead of killing the existing flags, can we just capture the stream and duplicate it if platform env exists?

[mvpatel2000]

LGTM!