fix: Use Amazon Linux 2 provided runtime

[SamStephens]

Breaking change: builds now expect a golang image. This may break set ups that have specified a particular image tag.

Fixes #320.

Supercedes #401.

As part of this change, the compiled Go binary main is renamed to bootstrap, as that is the name that the AL2023 Lambda runtime expects.

We now build using the standard Go Docker image, as the AWS Lambda Go image is deprecated.

It's important to note that the Dockerfile in the lambda/ folder is not used as part of the runtime at all. It's only function is to produce the /asset/bootstrap binary that is then uploaded to Lambda and used with the AL2023 runtime. This was the case before this change, except it used to produce /asset/main to use with the Go runtime.

I've tested this using the Sample application as documented in the README.md, using the steps I documented in my issue about the Sample to make it usable.

My testing steps were:

• Deploy the Sample application using the unchanged package (using the Go runtime);
• Deploy again with this change (prove we can change from the Go runtime to AL2023 successfully);
• Update /test/docker/Dockerfile to point to an older version of Nginx, so we have a change to the deployed asset, and confirm the Sample deploys successfully, and that the asset in the nginx ECR repository the Sample application creates is updated successfully.

If you are going to try these tests yourself, I suggest that before you do the deploy to change the deployed asset, you test that the Lambda function is actually callable. I wasted a lot of time, because if the custom resource Lambda handler does not work, the rollback hangs for an hour before it fails. If you call the Lambda from the console with {} as the input, you should see an error similar to {"errorMessage":"Put \"\": unsupported protocol scheme \"\"","errorType":"Error"}. This indicates execution is getting into the Lambda handler code.

Note that both release packaging and the prebuilt Lambda functionality is affected by this change. I cannot think of a meaningful way to test these changes without actually releasing them.

Release packaging is impacted, because the build Go binary and it's hash are included in the release files. The release is updated to match the new naming, so we have bootstrap and bootstrap.sha256 instead of main and main.sha256.

These release files are then used by the prebuilt Lambda functionality. When using the prebuilt Lambda, the Lambda is downloaded from the Github release files via /lambda/install.js. This logic also had to be changed to reflect the bootstrap naming.

As part of this change we get rid of the _GOPROXY alias for GOPROXY, as this was only needed because of the AWS build image (as per the removed comment).

[fabiozig]

Thanks @SamStephens, I can try to test the changes myself this weekend

[SamStephens]

Something to consider - should this change lead to a major version bump?

I think it should, because it is potentially a backwards incompatible change for anyone specifying a custom buildImage,

[fabiozig]

I wonder if there is any use case where someone has to specify a custom base image other than golang:1. If there is, I think it might be safer to change the major version up.

edit: I think there might be actually... in case you don't want to use docker hub because of image pull rate limits

[SamStephens]

I think the issue is more like if someone is currently specifying a build image other than public.ecr.aws/sam/build-go1.x:latest - say they've fixed to a particular tag of the public.ecr.aws/sam/build-go1.x image because they want repeatable builds. Their builds will break when we switch to a build that expects to be using the golang image.

[SamStephens]

@fabiozig @scanlonp have either of you had a chance to look at this? The Go runtime is deprecated in slightly more than a month. Also, I'm finishing work for the year soon, it'll be difficult for me to continue contributing after that.

[fabiozig]

@SamStephens I tested the changes in my local environment, using the CI flag on a test application (node js). I confirmed the lambda was deployed successfully with the new runtime. I also confirmed my application image was copied from the CDK asset repository to a custom ECR repository. If there are no other issues, I think we can proceed with the deployment.

[fabiozig]

I'm approving the PR, but because I don't know this repository very well, it would be good if someone else checked if we didn't miss anything.

[SamStephens]

Thanks @fabiozig

Before releasing this, someone should also decide if there should be a major version bump, and if so, work out what the procedure for doing a major version bump is.

[scanlonp]

@SamStephens thanks for your work.
@fabiozig appreciate your testing!

I will think about the major version bump and what that would look like for our release. I am good to approve after I take one last thorough look through, but Friday will likely be the earliest that a release will happen.

[SamStephens]

@scanlonp have you had a chance to look at this? We're running out of time to get this released before the end of year deprecation. Especially as we should probably try and give consumers of this package time to pick up this change before the Christmas/New Years period.

[scanlonp]

I have, and I am ready to ship. Just need to configure the major version bump and we are good to go.

[SamStephens]

I have, and I am ready to ship. Just need to configure the major version bump and we are good to go.

Thanks @scanlonp. A thought: should we start with an RC version number or similar, something like 3.0.0rc1, so we can test the changes to the release packaging and the prebuilt Lambda functionality work without the risk of a broken 3.0.0 version.

[iOnline247]

I have, and I am ready to ship. Just need to configure the major version bump and we are good to go.

Thanks @scanlonp. A thought: should we start with an RC version number or similar, something like 3.0.0rc1, so we can test the changes to the release packaging and the prebuilt Lambda functionality work without the risk of a broken 3.0.0 version.

I have some cycles to test this on real world scenarios via CodePipelines.

[SamStephens]

@iOnline247 that'd be amazing if you could help @scanlonp . I can help if needed, but I'd prefer not to because I'm technically on leave for the rest of the year.

[iOnline247]

@iOnline247 that'd be amazing if you could help @scanlonp . I can help if needed, but I'd prefer not to because I'm technically on leave for the rest of the year.

I have a multitude of CDK projects that use this and have been following it closely. I'm not building from source, but a consumer. More than happy to pull in a version and kick the tires for some feedback on projects that are in production.

[fabiozig]

Do you guys have any estimate when it will be deployed? We really need this ASAP as we also have lot of people on leave from next week until mid January. If you guys need any help to get this shipped please let us know!

[iOnline247]

@iOnline247 that'd be amazing if you could help @scanlonp . I can help if needed, but I'd prefer not to because I'm technically on leave for the rest of the year.

I have a multitude of CDK projects that use this and have been following it closely. I'm not building from source, but a consumer. More than happy to pull in a version and kick the tires for some feedback on projects that are in production.

This is a really bad outlook.

#320 (comment)

"Frankly, it appears AWS's custodianship of its open source eco-system appears to be failing; I urge you to raise this with your leadership. AWS needs to either commit more resource, or actually bite the bullet and either shut these packages down or find owners outside AWS for them. Currently these unsupported packages are a trap"

This package is the recommended way to build apps!

[SamStephens]

Do you guys have any estimate when it will be deployed? We really need this ASAP as we also have lot of people on leave from next week until mid January. If you guys need any help to get this shipped please let us know!

@fabiozig it's worth noting that even if you don't manage to pick these changes up until next year, you should not actually have any problems. Looking at https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html#lambda-runtimes, whilst the deprecation date is Jan 8, 2024, function creation is available until Feb 8, 2024 and updates until Mar 12, 2024. According to https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html#runtime-support-policy, "You can continue to invoke your functions indefinitely after the runtime is deprecated.".

So the deprecation date is not actually a hard deadline.

[scanlonp]

@SamStephens thanks for your work on this! I think just the major version bump should be fine and we can roll forward should there be issues.

[SamStephens]

@scanlonp did you see the mutation failure that showed up https://github.com/cdklabs/cdk-ecr-deployment/actions/runs/7215115085/job/19659171565?pr=424

What's happened is that when I ran npx projen, /.projen/tasks.json was changed. Basically those changes are what the mutation failure is complaining about - when the build generates the file, it generates with the original lines, not the lines that generated for me locally.

I could just update /.projen/tasks.json to match what the build expects, what do you think?

[scanlonp]

@SamStephens I did see it, was trying to see why the build is not tracking the new file. Sure lets try that.

Also I can take over the work on this if I am able to have push permission to your branch. If it is a branch other than main I can commit. I do not know if there are ways to give permission to a fork's main branch, but that would also work if you feel comfortable.

[easydonny]

As the end of the year is approaching, I think another alternative is https://github.com/cdklabs/cdk-docker-image-deployment

[scanlonp]

@SamStephens, I got the build working. Running yarn install before npx projen allows the dependencies to be filled. Particularly @latest -> @16 in this case and some other build commands on the tasks.json file.

@fabiozig merging this now, and I will keep an eye on the release. Let me know if it works for you.

@iOnline247 yes, we should update that. We are working on updating expectations of the level of maintenance of many of our cdklabs repositories. I have some ideas on what custodianship should look like, but for now we certainly need to be much clearer with our current level of support. I will make an issue so you can track it.

@easydonny yes, that is another option (that I happen to be the author of 🙃)! It works for its narrow use case, but is not very flexible and has a decent bit of overhead. However, it does not depend on go / the build images that caused this particular issue. I would personally like to fully flesh it out and have it supercede this package, but I do not envision that happening in the near future.

I have said some of the same thoughts & more on some of the issues Sam has kindly opened up to get our attention on these packages.

[yito-dev-tokyo]

@SamStephens @scanlonp
Thank you for your hard work, but the binary file name for prebuilt is incorrect.

• lambda/install.js:48
  const bin = path.join(dir, 'main');

Runtime.InvalidEntrypoint error occurs when running the provided.al2 runtime because the file name is not "bootstrap".

[SamStephens]

Thanks @yito-dev-tokyo

@scanlonp this is exactly why I suggested we release an RC before releasing 3.0.0 - because the functionality in lambda/install.js:48 was only testable once the change was released. I'll create an issue for this.

@yito-dev-tokyo for now you can set the NO_PREBUILT_LAMBDA environment variable as per https://github.com/cdklabs/cdk-ecr-deployment#environment-variables to avoid that broken functionality.

[fabiozig]

We got the same problem here with the pre-built lambda :(

Status: error	Error Type: Runtime.InvalidEntrypoint
Error: Couldn't find valid bootstrap(s): [/var/task/bootstrap /opt/bootstrap]

[scanlonp]

Should be good to go now with v3.0.1.

[fabiozig]

@scanlonp I just want to update you about my tests. Everything seems to be working with only one note. One of my applications had the aws-cdk-lib version locked on the yarn.lock file and the deployment was failing with the following error.

/node_modules/cdk-ecr-deployment/src/index.ts:140:20
/node_modules/aws-cdk-lib/aws-lambda/lib/singleton-lambda.js:1:2886
/node_modules/aws-cdk-lib/aws-lambda/lib/function.js:1:11075

TypeError: Cannot read properties of undefined (reading 'name')

In my case, there was a version conflict with the aws-cdk-lib I was using on my yarn.lock file (2.102.0)... I was able to fix the problem by removing the lock and using the latest version of the library... It seems like deployments using older versions of the aws-cdk-lib could break

Thanks!