Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WPA Beanstalk is crashing sometimes with nil pointer error. #88

Open
alok87 opened this issue Jul 2, 2020 · 3 comments · Fixed by #119
Open

WPA Beanstalk is crashing sometimes with nil pointer error. #88

alok87 opened this issue Jul 2, 2020 · 3 comments · Fixed by #119
Labels
bug Something isn't working

Comments

@alok87
Copy link
Contributor

alok87 commented Jul 2, 2020

WPA is crashing and restarting sometimes with the nil pointer error

I0702 11:51:51.353487       1 controller.go:484] dev-call_otp_updater-crstg1 minWorkers=0, maxDisruptableWorkers=0
I0702 11:51:51.353505       1 controller.go:356] dev-call_otp_updater-crstg1: messages: 0, idle: 0, desired: 0
I0702 11:51:51.353513       1 controller.go:593] crstg1/25-29871-otpcallupdater-praccomm-api: WPA status is already up to date
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1202436]

goroutine 45213 [running]:
github.com/beanstalkd/go-beanstalk.(*Conn).cmd(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x16c2531, 0xa, 0xc00d553a38, 0x1, ...)
        /src/vendor/github.com/beanstalkd/go-beanstalk/conn.go:76 +0x66
github.com/beanstalkd/go-beanstalk.(*Tube).Stats(0xc00d553a90, 0x34, 0xc00085c1de, 0x16)
        /src/vendor/github.com/beanstalkd/go-beanstalk/tube.go:93 +0xcf
github.com/practo/k8s-worker-pod-autoscaler/pkg/queue.(*beanstalkClient).executeGetStats(0xc00734c440, 0xc00d553b98, 0x139935a, 0x1541140, 0x15dc740)
        /src/pkg/queue/beanstalk.go:123 +0x90
github.com/practo/k8s-worker-pod-autoscaler/pkg/queue.(*beanstalkClient).getStats(0xc00734c440, 0xc000a2ec00, 0x34, 0x1a0bec0, 0xc00734c440)
        /src/pkg/queue/beanstalk.go:144 +0x2f
github.com/practo/k8s-worker-pod-autoscaler/pkg/queue.(*Beanstalk).getMessages(0xc00017e090, 0xc000a2ec00, 0x34, 0x446235, 0xa1bc22, 0xc00e35a450)
        /src/pkg/queue/beanstalk.go:262 +0x8a
github.com/practo/k8s-worker-pod-autoscaler/pkg/queue.(*Beanstalk).poll(0xc00017e090, 0xc0014c3b00, 0x26, 0xc000a2ec1e, 0x16, 0xc00b599a70, 0x6, 0xc000a2ec00, 0x34, 0xc000a2ec0c, ...)
        /src/pkg/queue/beanstalk.go:351 +0x75
github.com/practo/k8s-worker-pod-autoscaler/pkg/queue.(*Poller).runPollThread(0xc00017e0f0, 0xc0014c3b00, 0x26)
        /src/pkg/queue/poller.go:46 +0x88
created by github.com/practo/k8s-worker-pod-autoscaler/pkg/queue.(*Poller).Run
        /src/pkg/queue/poller.go:93 +0x209
@alok87 alok87 added the bug Something isn't working label Jul 2, 2020
@alok87 alok87 changed the title WPA Beanstalk is crashes with nil pointer error. WPA Beanstalk is crashing sometimes with nil pointer error. Jul 2, 2020
@matkam
Copy link
Contributor

matkam commented Aug 28, 2020 •
edited
Loading

Same thing is happening with SQS, except its causing the pod to go in a CrashLoopBackOff for us:

 panic: runtime error: invalid memory address or nil pointer dereference                                                                                                                                 
 [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x137304a]                                                                                                                                 
 goroutine 79 [running]:                                                                                                                                                                                 
 github.com/aws/aws-sdk-go/service/cloudwatch.(*CloudWatch).newRequest(0x0, 0xc001550080, 0x15f22c0, 0xc001550040, 0x1582580, 0xc0015500c0, 0x0)                                                         
     /src/vendor/github.com/aws/aws-sdk-go/service/cloudwatch/service.go:90 +0x3a                                                                                                                        
 github.com/aws/aws-sdk-go/service/cloudwatch.(*CloudWatch).GetMetricDataRequest(0x0, 0xc001550040, 0x1491d20, 0xc0003f5b01)                                                                             
     /src/vendor/github.com/aws/aws-sdk-go/service/cloudwatch/api.go:1527 +0x1d4                                                                                                                         
 github.com/aws/aws-sdk-go/service/cloudwatch.(*CloudWatch).GetMetricData(0x0, 0xc001550040, 0xc00003e37c, 0x9, 0x24dcf60)                                                                               
     /src/vendor/github.com/aws/aws-sdk-go/service/cloudwatch/api.go:1589 +0x35                                                                                                                          
 github.com/practo/k8s-worker-pod-autoscaler/pkg/queue.(*SQS).getAverageNumberOfMessagesSent(0xc000168360, 0xc00003e370, 0x50, 0x0, 0x0, 0xc000000000)                                                   
     /src/pkg/queue/sqs.go:340 +0x4d6                                                                                                                                                                    
 github.com/practo/k8s-worker-pod-autoscaler/pkg/queue.(*SQS).cachedNumberOfSentMessages(0xc000168360, 0xc00003e370, 0x50, 0x0, 0x1, 0xc000215d08)                                                       
     /src/pkg/queue/sqs.go:260 +0x13f                                                                                                                                                                    
 github.com/practo/k8s-worker-pod-autoscaler/pkg/queue.(*SQS).poll(0xc000168360, 0xc000a6c180, 0x40, 0xc00003e3a1, 0x1f, 0xc00020a040, 0x20, 0xc00003e370, 0x50, 0xc00003e378, ...)                      
     /src/pkg/queue/sqs.go:434 +0xfd1                                                                                                                                                                    
 github.com/practo/k8s-worker-pod-autoscaler/pkg/queue.(*Poller).runPollThread(0xc000606600, 0xc000a6c180, 0x40)                                                                                         
     /src/pkg/queue/poller.go:46 +0x88                                                                                                                                                                   
 created by github.com/practo/k8s-worker-pod-autoscaler/pkg/queue.(*Poller).Run                                                                                                                          
     /src/pkg/queue/poller.go:93 +0x209                                                                                                                                                                  
 stream closed

@matkam
Copy link
Contributor

matkam commented Aug 28, 2020 •
edited
Loading

I added some debug lines, and it turns out the CW client returned by getCWClient(...) is nil when this happens!
The problem is I was using a queueUri that incorrectly referred to a region not included in the --aws-regions flag.

@alok87
Copy link
Contributor Author

alok87 commented Aug 29, 2020

Hey thanks I will check this soon and close.

@alok87 alok87 mentioned this issue Sep 14, 2020
Merged
@alok87 alok87 mentioned this issue Oct 22, 2021
Open
@alok87 alok87 reopened this Oct 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Prevent crash due to connection failures practo/k8s-worker-pod-autoscaler
2 participants
@matkam @alok87