Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lambda warmup not waming up the function properly #66

Closed
nenti opened this issue Jul 2, 2018 · 8 comments
Closed

Lambda warmup not waming up the function properly #66

nenti opened this issue Jul 2, 2018 · 8 comments
Labels

Comments

@nenti
Copy link

nenti commented Jul 2, 2018

I installed the warmup plugin to my project and just realized that it is not working properly.

The warmup-lambda is peroperly executing every 5 minutes:
image

The logs indicate a successful execution:

START RequestId: 3b04bd5d-7de1-11e8-8fdf-7b05371b3361 Version: $LATEST
2018-07-02T10:18:14.729Z	3b04bd5d-7de1-11e8-8fdf-7b05371b3361	Warm Up Start
2018-07-02T10:18:15.187Z	3b04bd5d-7de1-11e8-8fdf-7b05371b3361	Warm Up Invoke Success: glypo-dev-captureTransaction { StatusCode: 200, Payload: '"Lambda is warm!"' }
2018-07-02T10:18:15.187Z	3b04bd5d-7de1-11e8-8fdf-7b05371b3361	Warm Up Finished with 0 invoke errors

But the function monitoring on captureTransaction doesn't log an invocation:
image

Also the end to end test is still showing following load times on sequential execution:

Received: 4846
Received: 1537
Received: 1627
Received: 1580
@goncaloneves
Copy link
Contributor

@nenti WarmUP invokes $LATEST Lambda alias which stores the Invocation metrics under your $LATEST Lambda Resource CloudWatch metric. That's why it doesn't show up on Lambda monitoring view.

You could just check by FunctionName Lambda metric and it will show up fine in CloudWatch.

And you could check your service warmed up functions CloudWatch logs which you can easily cross-check if they are invoked or not.

@nenti
Copy link
Author

nenti commented Jul 3, 2018

@goncaloneves thank you for clarifying this. I found the logs and it is invoked just fine. Problem is, that when I invoke my lambda it hits without the $LATEST and seems to hit an unwarmed instance. Although they hit the same version.

Please see the request times of my test.

@goncaloneves
Copy link
Contributor

Are you running multiple version containers?
WarmUP guarantees one $LATEST version container warmed up. We discussed what would be the best approach to implement it at scale with multiple instances running in parallel in discussion #24.

@nenti
Copy link
Author

nenti commented Jul 3, 2018

No I only run one version of lambda. But still warmup invocations are showing up in different monitoring and the lambda doesn't seem to be warm when I invoke it.

Invocation comes from another service and is called like so:

      const params = {
        FunctionName: arn,
        InvocationType: async ? 'Event' : 'RequestResponse',
        Payload: typeof data === 'object' ? JSON.stringify(data) : data,
      }

      return this.lambda.invoke(params, (error, result) => {
        // do something
      })

@goncaloneves
Copy link
Contributor

You are invoking without Qualifier parameter. It doesn't seem? Or it isn't? Check the logs of the functions you are warming up and confirm.

https://github.com/FidelLimited/serverless-plugin-warmup/blob/e1c6a9b1ef16260d9c7379efd15ea5ebec6b2a8e/src/index.js#L271

@nenti
Copy link
Author

nenti commented Jul 3, 2018

Yes I invoke without Qualifier. But it is not a recommended field in invoke. So shouldn't we be expecting people calling without Qualifier?

UPDATE: I tried hitting with Qualifiers set and now it lists in the same monitoring tab. First hit was still with almost a second delay again while execution time vary within 100ms.

Now first hit was 2400 ms (compared to >3000) only so closer to 2nd hit. Would there be any other layer that needs to be warmed when hitting a lambda from remote? Testsetup is running a local unit test that calls remote lambda. Could be that the unit-test is just running slower on initial execution.

@goncaloneves
Copy link
Contributor

goncaloneves commented Jul 4, 2018

It is fine without Qualifier, by default it sets to $LATEST per AWS documentation. The only difference is the CW metrics are different. Not an issue.

If you are finding issues with cold starts, lower the schedule interval. I set to run every 2 minutes since in most use cases without external invocations most lambdas go cold around 3 minutes.

(Edit) Most containers regardless of activity go down around 15 minutes of uptime. The shorter the interval (1 minute minimum with CW Schedule), less likely your users will feel the cold start pain when there is no container up.

@nenti nenti closed this as completed Jul 5, 2018
@ywang9009gmail
Copy link

Hi, is there a way to keep a specific version of function warm instead of $LATEST? Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants