Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StrictRedisCluster client hangs with AWS elasticache - connection pool remains uninitialized #215

Open
adazzi93933 opened this issue Apr 23, 2022 · 1 comment

Comments

@adazzi93933
Copy link

adazzi93933 commented Apr 23, 2022

Checklist

  • Python version
    3.7
  • Using hiredis or just Python parser
    no
  • Using uvloop or just asyncio event loop
    asyncio
  • Does issue exists against the master branch of aredis?
    yes
    git clone [email protected]:NoneGG/aredis.git
    cd aredis/
    python3 setup.py install

Steps to reproduce

using the latest aredis v1.1.8, tried python 3.7,3.8,3.9 and downgraded aredis to lower versions as well ... same problem.

set up load cluster on mac

export REDIS_CLUSTER_IP=0.0.0.0
docker run -e "IP=0.0.0.0" -p 7000-7005:7000-7005 grokzen/redis-cluster:latest
ran the script below and i can connect to it (no ssl, auth, etc) and i can set/get a key
initializing the pool seems optional

output ...

local redis cluster
False
dict_keys([])
init connection pool
True
nodes:
127.0.0.1:7001
127.0.0.1:7004
127.0.0.1:7000
127.0.0.1:7003
127.0.0.1:7002
127.0.0.1:7005
True
True
b'bar' <== worked
True. <== pool got initialized whether i ask for it explicitly or not

###set-up a working cluster on aws elastic cache
My production go service running on istio/k8s works fine and can connect to aws elastic cache!
My production python admin service that used to work and connect to bitnami redis on kubernetes is now unable to work with aws elastic cache???
so i wrote this little script to debug ...

#!/usr/bin/python
# -*- coding: utf-8 -*-

import asyncio
from aredis import StrictRedisCluster
#from rediscluster import RedisCluster

REDIS_MASTER_URL = "clustercfg.mycluster.fmeera.use1.cache.amazonaws.com"
REDIS_PORT = "6379"

def find_node_ip_based_on_port(cluster_client):
    print("nodes:")
    for node_name, node_data in cluster_client.connection_pool.nodes.nodes.items():
        print(node_name)

async def example():
    cl = StrictRedisCluster(host="0.0.0.0", port=7000)
    print("local redis cluster")
    print(cl.connection_pool.initialized)
    print(cl.connection_pool.nodes.slots.keys())
    print("init connection pool")
    await cl.connection_pool.initialize()
    print(cl.connection_pool.initialized)
    #print(cl.connection_pool.nodes.slots.keys())
    find_node_ip_based_on_port(cl)
    #print(await cl.cluster_slots())
    print(cl.connection_pool.initialized)
    await cl.set('foo', 'bar')
    print(cl.connection_pool.initialized)
    result = await cl.get('foo')
    print(result)
    print(cl.connection_pool.initialized)
    #
    print("-----------------------")
    redis = StrictRedisCluster(host=REDIS_MASTER_URL, port=REDIS_PORT, ssl=True, password="some passwd", skip_full_coverage_check=True, decode_responses=True)
    print("aws elastic cache")
    print(redis.connection_pool.initialized)
    #print(redis.connection_pool.nodes.slots.keys())
    print("init connection pool")
    await redis.connection_pool.initialize()
    print(redis.connection_pool.initialized)
    find_node_ip_based_on_port(redis)
    print(await redis.cluster_slots())
    await redis.set('foo', 'bar')
    result = await redis.get('foo')
    print(result)

loop = asyncio.get_event_loop()
loop.run_until_complete(example())

Expected behavior

output is ...


aws elastic cache
False
init connection pool
False
nodes:

it hangs on 
 await redis.connection_pool.initialize()
or 
await redis.cluster_slots()

dumping the connection pool show no configured nodes, slots?

expected behavior is to get a map of nodes and slots.

in this case i am using TLS, and encryption.


connecting to aws elastic cache works using the cli
connecting to aws elastic cache works using the blocking client -> rediscluster.RedisCluster (redis-py-cluster)

Actual behavior

local redis cluster
False
dict_keys([])
init connection pool
True
nodes:
127.0.0.1:7001
127.0.0.1:7004
127.0.0.1:7000
127.0.0.1:7003
127.0.0.1:7002
127.0.0.1:7005
True
True
b'bar'
True
-----------------------
aws elastic cache
False
init connection pool
False
nodes:
hangs ...
@adazzi93933
Copy link
Author

update: i created an aws elasticcache cluster without "Encryption in transit" enabled and auth and it is working now. i will narrow down the problem further. but it seems to be related to encryption. also i had to remove ssl=True to make it work (i was getting an error).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant