fix and test some memory leaks

[slingamn]

I was working on a new implementation of #75 / #102. I found what I believe are memory leaks, mostly associated with key normalization. For example, _key_normalized_obj was not consistent (across its different conditional branches) as to whether it created a new reference or not. More generally, if a PyObject * key can be overwritten by a normalized or prefixed key, any owned (as opposed to borrowed) references to the original key must be cleaned up.

I've added tests for these issues. The tests pass in this branch, but do not pass in master: https://gist.github.com/slingamn/3b6cec10c1f12f0dbe1b

I think it's possible that there are more leaks:

1. This testing approach can't identify leaks of intermediate values, only in arguments to public functions
2. Most of the error recovery code is not covered

but I think this should cover the leaks that the typical user is most likely to encounter.

Thanks for your time!

[kenrobbins]

I'm just a user of pylibmc, but thanks for this. I discovered the memory leak in _key_normalized_obj as well. Hope this can get merged soon.

[lericson]

Exciting stuff, very nice. I'm a little anxious about the amount of changes, so let's hope your approach is solid. It seems better to test ref counts than not though. But extensive testing will be needed. The way I've isolated memory leaks previously is to just run code in a loop and see if the process starts eating memory.

[aconrad]

Any ETA for a new release containing this fix?

[slingamn]

Thanks!

Yeah, testing this is tricky. I think in order to see a blowup in memory consumption from the issues in this patch, it would be necessary to use distinct keys for each iteration of the loop --- otherwise, only an O(1) amount of objects will become uncollectable.

[jkchin]

ping, any update on when this fix will come out?

[lericson]

Well, the originating branch has been closed so I'm not sure how I can move forward with this.

[kenrobbins]

@lericson Not sure what you mean. It looks like it's merged into master. I believe @jkchin was asking when a build would be created with this change. It appears it is in 1.5.1 released 21 March though I didn't see anything in the changelog.