[BUG] GPU tolerations are not correctly passed to MPI workers

[gdabisias]

Describe the bug

When defining an MPITask, the tolerations that are set as default for GPU usage in the deployment charts, are not applied to the created workers.
i.e. this configuration is not applied

configmap:
  k8s:
    plugins:
      k8s:
        "resource-tolerations":
          "nvidia.com/gpu":
            key: "flyte/GPU"
            operator: "Exists"
            effect: "NoSchedule"

This has also an additional bad side effect: the MPI launcher will end up on a GPU node, even if it could just run on a cpu node, wasting resources.

Expected behavior

The GPU tolerations should be applied automatically to MPI workers when GPUs are requested.
Currently this can be fixed by passing a flytekit.PodTemplate to the MPI Task, but this makes the code very cumbersome
e.g.

flytekit.PodTemplate(
        V1PodSpec(
            tolerations=[
                V1Toleration(
                    key="flyte/GPU",
                    operator="Exists",
                    effect="NoSchedule",
                ),
            ]
        )
    )

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[gdabisias]

Notice that this issue is similar to
#4378

[jeevb]

Confirmed that this can be reproduced with:

@task(
    task_config=MPIJob(
        launcher=Launcher(
            replicas=1,
        ),
        worker=Worker(
            replicas=1,
            limits=Resources(cpu="2", gpu="1"),
        ),
    ),
    retries=3,
    cache=True,
    cache_version="0.1",
    requests=Resources(cpu="1", mem="1000Mi"),
    limits=Resources(cpu="2"),
)

The tolerations are applied based on the universal resources specified in the @task decorator, but not for the launcher/worker-specific resource specification.