Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow warmup of Unqualified Alias #220

Closed
peppeocchi opened this issue Sep 10, 2020 · 5 comments · Fixed by #229
Closed

Allow warmup of Unqualified Alias #220

peppeocchi opened this issue Sep 10, 2020 · 5 comments · Fixed by #229

Comments

@peppeocchi
Copy link

There is currently no way of warming up the Unqualified alias. From what I see in the code there is a way of setting a specific alias via an env variable, otherwise it will default to $LATEST. Which is fine in most of the cases. Up until API Gateway V2 showed up.

The problem arises in this scenario:
Deploy a function and attach API Gateway V2. It will, by default, link the Unqualified alias of the function. There is currently no plugin that allows you to both create aliases and link the correct alias in API Gateway.
There is some kind of CF template you can use to get very close (without having to create a custom plugin) where you can disable versioning from serverless, have a script that automatically generates a unique name for the versioning step, then create the alias and/or switch the version on the alias. The last bit I wasn't able to work out is how to link a specific alias in API Gateway V2, I guess it's so "new" that wasn't really taken seriously, or API Gateway V1 is so deep in the serverless logic that will require some kind of refactoring. Anyway, that's is a different story.

What happens, and I proved it myself (although the AWS support sent me a confused response), is that if you warm up the $LATEST version, the Unqualified alias won't be warmed up, although by default is linked to $LATEST.

To validate what I am saying you can:

  • create a new Lambda function
  • start the warmer on $LATEST - you will see in the monitor of $LATEST the executions of the warmer function
  • switch to the Unqualified alias - the monitor won't show any sign of the warmer running
  • the first execution in Unqualified will have a cold start
  • now create a new alias and point it to $LATEST
  • test the new alias, you will have a cold start

Now, if Lambda supported the setting of env variables as empty, this would be just a matter of creating an empty env variable in the lambda function via serverless, but you can't create empty env variables in Lambda.

It would be great if there was a workaround to this issue and have the warmup (optionally) to run on the Unqualified alias.

I already tried setting Unqualified as SERVERLESS_ALIAS and didn't work

Warm Up Invoke Error: function ResourceNotFoundException: Function not found: arn:aws:lambda:function:Unqualified
@juanjoDiaz
Copy link
Owner

Hi @peppeocchi ,

Sorry for the late response.

This really sounds like some weird behaviour on AWS end since according to the AWS docs When you invoke a function using an unqualified ARN, Lambda implicitly invokes $LATEST. (see https://docs.aws.amazon.com/lambda/latest/dg/configuration-versions.html).

Did you get any resolutions from AWS?

@peppeocchi
Copy link
Author

peppeocchi commented Nov 13, 2020

@juanjoDiaz it is indeed a weird behaviour and we only got a confused response (we have the aws premium support) saying that unqualified points to $LATEST. Which is true, the codebase is the same in both unqualified and $LATEST, but we think that the unqualified is actually its own alias.
If unqualified and $LATEST were the same thing, when calling the unqualified alias I should see the same metrics for both unqualified and $LATEST, which is not true. And I should not get a cold start when calling respectively $LATEST first and then unqualified (or the opposite), which is not true.

I think it's the same behaviour if you have another alias XYZ that points to a specific version 1, then you call XYZ and version 1 separately and you will see different metrics and 2 separate cold starts. But this is something I haven't tried, this is only speculation, as well as thinking that an alias is just a pointer to a specific codebase but in fact it will spin up different containers.

Going back to what I was asking, the problem will be solved if there was a way to specify that you want to use the unqualified alias, so that when it warms up the function it won't add anything in the ARN (I see it's currently adding $LATEST if there no alias is specified).

And I completely understand that I could solve this by using a specific alias, but as I mentioned there is currently no way of setting a specific alias via serverless when building up API Gateway V2.

@juanjoDiaz
Copy link
Owner

I understand the issue.
And I understand that you believe that simply being able to leave the alias unset would fix it.

The easy solutions would be to replace Qualifier: SERVERLESS_ALIAS || '$LATEST', by Qualifier: SERVERLESS_ALIAS. Theoretically, it should make no difference. It didn't make any difference when we added the $LATEST default. However, theoretically, you shouldn't be having this problem either... So now I am concerned about doing such a change...

Btw, when you talk about API Gateway V2, you mean the HTTP Gateway, right? The one that is created when using the httpAPI event instead of http in serverless, right? Or how can I set up a project to replicate this?

Have you checked https://github.com/serverless-heaven/serverless-aws-alias for the alias support?

I know from experience that AWS support is an expert in giving confusing and invalid explanations to avoid fixing things. However, this is clearly a strange case and it should either be fixed or properly documented... So I'd suggest that you use your premium support and push a bit to get more insights on this. Once we really understand what is happening, we can make a decision on how to act on it.

@peppeocchi
Copy link
Author

@juanjoDiaz thanks for your reply, I already checked that plugin and it currently doesn't support Api Gateway V2 (HTTP Gateway).

What I am really asking is that the Qualifier parameter gets removed when invoking the function, if specified by an env variable or some other way.

The AWS premium support has replied (it's not the first time) with a copy&paste from their documentation.

Let me begin by saying that technically speaking the qualified latest version and the unqualified latest version are the same IF no other versions or alias exist. In this case, when you invoke a function using an unqualified ARN, Lambda implicitly invokes $LATEST.[1]

Don't worry if you don't want to make any changes, we have solved it by removing this plugin from the serverless config file after having deployed once. Then we removed the "Qualifier" parameter from the lambda function code and the warmup is working as expected. I'm sure we're not the only one facing this issue and it's a bit annoying having to manually do the above every time we need warmup in a new function.

This plugin works perfectly with API Gateway V1 because the API Gateway link to the lambda ARN includes the alias/version (e.g. $LATEST), with API Gateway V2 the alias/version is not there, not even the UI in the AWS console allows you to pick an alias/version, but you can manually paste in the full ARN and it works.

@juanjoDiaz
Copy link
Owner

This took way to long but v5 is finally released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants