Enhancements to MemQ Cluster Sensor

[yisheng-zhou]

This PR addresses an issue where the MemQ Cluster Sensor could fail after running for an extended period by introducing a mechanism to refresh the CuratorFramework client upon encountering ZK call exceptions. Previously, after MemQ orion ran for some time, the CuratorFramework might consistently throw exceptions, requiring engineers to restart orion to restore the system to a normal state. With this update, the client is now automatically recreated when exceptions occur. The newly instantiated client is attached to the MemQ cluster class, ensuring continuous availability.

Key Improvements

• Refreshed Zookeeper Client:

  • The CuratorFramework client is now refreshed automatically on encountering exceptions, enhancing system reliability and reducing manual intervention. This feature can be turned on/off.

• Refactored Zookeeper Logic:

  • Zookeeper-related logic has been consolidated into a new class, MemqZookeeperClient. This refactoring clarifies the MemqClusterSensor code.
  • It's important to note that the MemqZookeeperClient does not share Zookeeper logic with Kafka because the ZK paths and commands differ significantly. Accordingly, it does not extend classes such as OrionZookeeperClient.

• Graceful Handling of Node Broker Clusters:

  • The system now handles node broker clusters more gracefully, contributing to overall stability.

Testing

The changes have been deployed and running within Pinterest for more than one day without any reported sensor errors, demonstrating improved stability and reliability.