Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error handling is not stack-safe when reconnect is enabled #141

Open
allter opened this issue Jan 20, 2020 · 2 comments
Open

Error handling is not stack-safe when reconnect is enabled #141

allter opened this issue Jan 20, 2020 · 2 comments

Comments

@allter
Copy link

allter commented Jan 20, 2020

In my setup i saw occurence of the following warnings:

Deep recursion on subroutine "Try::Tiny::try" at /path/to/site_perl/5.18.0/Redis.pm line 275.
Deep recursion on subroutine "Redis::connect" at /path/to/site_perl/5.18.0/Redis.pm line 272.
Deep recursion on subroutine "Redis::__build_sock" at /path/to/site_perl/5.18.0/Redis.pm line 610.
Deep recursion on anonymous subroutine at /path/to/site_perl/5.18.0/Redis.pm line 629.
Deep recursion on subroutine "Redis::__std_cmd" at /path/to/site_perl/5.18.0/Redis.pm line 207.
Deep recursion on subroutine "Redis::__with_reconnect" at /path/to/site_perl/5.18.0/Redis.pm line 250.

As well as weird exception texts containing:

	...propagated at /path/to/site_perl/5.18.0/Redis.pm line 613.

How to (possibly) reproduce:

  • Setup haproxy in front of redis
  • Limit redis connections to X
  • Allow more then X connections on haproxy
  • Setup checking of backend liveness on haproxy by authenticating to redis, and drop all existing backend connections on a liveness check failure
  • Connect to redis (make new connections) while occasionally trying to use existing connections

I'm not sure whether this error occurs on connection or when trying to use existing connection after haproxy has dropped the backend connection.

v1.995

@dams
Copy link
Member

dams commented Aug 9, 2020

Can you create a test case ? I think it should be possible with using local spawned redis server and without HAproxy ?

@allter
Copy link
Author

allter commented Aug 9, 2020

Currently i don't have much time for it. :(

I think it should be possible to reproduce locally. To do it one can carefully choose Redis error which can be triggerred by configuration of Redis server (because as I investigated, the problem depends on a particular stage of a Redis connection, like authentification). But it's much reliable to make completely mocked Redis implementation.

In our case, the problem was triggered by HAproxy+monitoring in the middle. But even without HAproxy, weird network error can trigger the same.

P.S. It looks like this issue is a duplicate of #97 . If so, the error might be reproduced by just limiting maxclients on a Redis side

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants