Refactor distribution build job

[rishabh6788]

Description

This PR moves the docker build stage into parallel processing stages.
The stage will wait on x64 and arm64 tar builds to complete, and once the artifact urls for both are available it will process the docker build job.

Issues Resolved

#5173

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.32%. Comparing base (d74cbaf) to head (87f2ce0).
Report is 5 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5192   +/-   ##
=======================================
  Coverage   92.32%   92.32%           
=======================================
  Files         197      197           
  Lines        6817     6817           
=======================================
  Hits         6294     6294           
  Misses        523      523           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[gaiksaya]

I had tried something like this https://github.com/gaiksaya/opensearch-build/blob/6d4745692422ba7c7c89a55db110a6789ab97e9c/jenkins/opensearch/test-dist.jenkinsfile#L65-L70

Can you check if this is a better option than that? It will automatically start the build as soon those variables are set. If not the timeout will take care of timing out.

[rishabh6788]

Both are the same, in this we don't need to set additional env variables as we can leverage the ones already available.

[gaiksaya]

Wont that start the build as soon as the variables are set instead of waiting for next 120 seconds? Also wondering, if both or one of the stages failed, we should skip building docker altogether. However,in this case, it will wait for 90 minutes before timing out. Looks like we cannot check stage status (only job status). Can we add some check for that scenario?

[rishabh6788]

Once the variables are set that means the artifacts are available in S3 and can be used immediately, I don't see an issue here. Also as per logic it will only trigger the docker build once both, x64 and arm64 artifacts are available else timeout eventually.
Stage status check is not trivial and there is no straight-forward way to do the same, will need some hack.

[rishabh6788]

We shouldn't be building docker just for x64 or arm64, either build for both else skip.

[gaiksaya]

Right! My point being, the average build time for tarball x64 and arm64 is 20-30min. So if they fail, we would have to wait more 60min (since docker timeout is 90min) for docker stage to timeout eventually.

[rishabh6788]

Sure, may be you can PR it since you have the changes done in your local. I will cancel or you can edit this as well.
I am okay with both.

[rishabh6788]

On a second look, env.TAR_X64_SUCCESS = 'true' would only be set if the groovy lib that executes the actual workflow returns successfully. In case of failure it will exit out of the job without setting this variable.
So it is same as what I'm doing, in case if success/failure both the solutions work same.
Please correct me if I'm wrong.

[gaiksaya]

How about setting env.TAR_ARM64_STATUS = 'somevalue' in post failure stage? Or any value you see fit?

post {
  success {
      script {
          env.TAR_ARM64_STATUS = 'succeeded'
      }
  }
  failure {
    script {
      env.TAR_ARM64_STATUS = 'failed'
    }
  }
}

Check for both values of env.TAR_ARM64_STATUS in the docker build (succeeded and/or failed) and act accordingly. The main motive of this change is to lower the build time and immediately build docker when tarball is ready. Looks like in case of failure we would be waiting 90min for the job to complete.

[gaiksaya]

Synced up offline, looks like this would be too much of hack. We can see how the change in this PR goes and iterate on it later. Thanks!

[peterzhuamazon]

So the idea is to have everything in one parallel stage, then keep waiting until both link available?

[rishabh6788]

So the idea is to have everything in one parallel stage, then keep waiting until both link available?

Yes!

[rishabh6788]

If no more concerns can we approve and merge this? @gaiksaya @peterzhuamazon

[gaiksaya]

Would recommend to use inbuilt feature of waitUntil. We can take up it up as an enhancement if you like. Would avoid us adding code for checking and waiting for 120seconds