Change Release Base Images to

[KuuCi]

This is to speed up release process by making us not need to reinstall all of transformer_engine. Docker caching doesn't work here because of

llm-foundry/Dockerfile

Line 15 in 8e78eb5

ADD https://raw.githubusercontent.com/mosaicml/llm-foundry/$BRANCH_NAME/setup.py setup.py

, where if the branch name is different than main (and it will be since we expect it to be whatever branch is associated with the release), cache will be invaliadted.

Testing:

[mvpatel2000]

how is base image the same image ur building...?

[KuuCi]

They're different images, the images built here are
mosaicml/llm-foundry:release_${TAG_NAME}" and mosaicml/llm-foundry:release_latest"
to keep up with foundry releases

[snarayan21]

wait yeah, how are we building off the foundry images? that should be the result of this right

[KuuCi]

ohh. Huh, I guess the behavior is slightly weird.

1. https://github.com/mosaicml/llm-foundry/blob/main/.github/workflows/docker.yaml is what builds mosaicml/llm-foundry:2.4.0_cu124-latest (which is largely cached) -- not necessarily the same pr (could be the previous pr)
2. https://github.com/mosaicml/llm-foundry/blob/main/.github/workflows/release.yaml builds from that image. We don't cache here because it's almost guarenteed cache miss when we clone from our specific branch + pip install, but that doesn't matter bc the dependencies are mostly installed from the mosaicml/llm-foundry:2.4.0_cu124-latest image, and any that aren't install will be in dockerfile so we get net speed up
3. We keep the foundry repo

but like isn't that fine?

[snarayan21]

But currently the release workflow builds on the pytorch image, not the foundry image...i guess the foundry image does exist though...

[mvpatel2000]

I don't think its worth optimizing release workflow to add this complication. This feels a bit weird to me to build on top of the foundry build image...

[snarayan21]

^ agree

[snarayan21]

This can have some unintended complications -- for example when doing foundry release, suppose I create the release branch but don't publish the release and kick off the release workflow yet. Then, in between, someone else merges some breaking PR. Then I launch this new release workflow which builds on the cached latest foundry image that contains the bad commit....right?

[KuuCi]

Ok, sounds fair, will close pr

[snarayan21]

wait yeah, how are we building off the foundry images? that should be the result of this right