Bugs fixed

[mjhubert]

• Memory leak fixed
• ConsumerRegistryDir fixed

[mjhubert]

Hi,

No problem.
What I did was adding Equals and GetHashCode() implementation to your
objects which are used as keys in dictionaries.

http://stackoverflow.com/questions/634826/using-an-object-as-a-generic-dictionary-key

Otherwise each object is an unique entry, which was the reason for the
memory leak.

class ClientIdAndBroker
class ClientIdBrokerTopicPartition

We also experienced problems while using two different consumer groups on
one topic.
I also changed ConsumerRegistryDir [get] in Kafka.Client.Utils.ZKGroupDirs,
but I don't see it the commit differences.

    public string ConsumerRegistryDir
    {
        get
        {
            return this.*ConsumerGroupDir *+ "/ids";
        }
    }

I hope this helps you.

with kind regards,
Michel

2016-02-16 21:59 GMT+01:00 Andrew Perrins [email protected]:

Hey @mjhubert https://github.com/mjhubert - if you have a moment... I
see you contributed a memory leak fix, but it's not immediately clear from
your diff what changed; do you have any pointers as to what, specifically,
to look in your commit
mjhubert@2cacc6d
?

—
Reply to this email directly or view it on GitHub
#5 (comment)
.

[ajperrins]

Thanks, Michael. I see the Equals/GetHashCode diff now; and that
corroborates with the memory dump we took, which showed tens of millions of
items retained in dictionaries.

We ran into the consumer group issue, too, where re-balancing doesn't work
properly if there are multiple consumer groups consuming off the same
topic. Our resolution there was to modify ZkUtils.GetConsmersPerTopic by
changing the iterator to restrict to consumers that begin with the group id:

foreach (var consumer in consumers.Where(c => c.StartsWith(group))) { ... }

Thanks again for taking the time to respond.

Best,
Andrew

On Wed, Feb 17, 2016 at 9:36 AM, M.J. Hubert [email protected]
wrote:

Hi,

No problem.
What I did was adding Equals and GetHashCode() implementation to your
objects which are used as keys in dictionaries.

http://stackoverflow.com/questions/634826/using-an-object-as-a-generic-dictionary-key

Otherwise each object is an unique entry, which was the reason for the
memory leak.

class ClientIdAndBroker
class ClientIdBrokerTopicPartition

We also experienced problems while using two different consumer groups on
one topic.
I also changed ConsumerRegistryDir [get] in Kafka.Client.Utils.ZKGroupDirs,
but I don't see it the commit differences.

public string ConsumerRegistryDir
{
get
{
return this.*ConsumerGroupDir *+ "/ids";
}
}

I hope this helps you.

with kind regards,
Michel

2016-02-16 21:59 GMT+01:00 Andrew Perrins [email protected]:

Hey @mjhubert https://github.com/mjhubert - if you have a moment... I
see you contributed a memory leak fix, but it's not immediately clear
from
your diff what changed; do you have any pointers as to what,
specifically,
to look in your commit
<
mjhubert@2cacc6d

?

—
Reply to this email directly or view it on GitHub
<
https://github.com/ExactTargetDev/kafka-net/pull/5#issuecomment-184872693>
.

—
Reply to this email directly or view it on GitHub
#5 (comment)
.

Andrew J. Perrins
[email protected]