Freeze (memory/CPU chewed up) when trying to spell long string #18
Comments
dylan-chong
changed the title
|
This seems to happen with long words like Maybe the length of the word screws it up |
dylan-chong
changed the title
filyp
added a commit
to filyp/autocorrect-deprecated
that referenced
this issue
Sep 15, 2019
Optimize by using generators, so the possible typos don't have to be stored in memory.
For long english words:
%timeit spell('disproporttionatelly')
before: 1 s ± 7.81 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
after: 762 ms ± 18.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit spell('indistimguishabble')
before: 821 ms ± 5.22 ms per loop (mean ± std. dev. of 7 runs, 1 loop each
after: 619 ms ± 10 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
More importantly, I'm working on a support for polish, where alphabet is larger, and words tend to be longer, so the change is significant:
%timeit spell('gżegrzółka')
before: 1.51 s ± 67.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
after: 370 ms ± 16.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit spell('anarchokolektuwistycznychh')
before: 3.83 s ± 116 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
after: 2.2 s ± 11.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
It also solves issue phatpiglet#18
Before, running spell('xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx')
would consume all RAM it could. Now it takes only 8KB more than idle, and won't freeze.
When using pypy, difference gets even more dramatic:
%timeit spell('disproporttionatelly')
before: 668 ms ± 19.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
after: 377 ms ± 25.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit spell('indistimguishabble')
before: 585 ms ± 15.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
after: 330 ms ± 29 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit spell('gżegrzółka')
before: Gets killed because it eats up too much RAM
before: 166 ms ± 11.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit spell('anarchokolektuwistycznychh')
before: Gets killed because it eats up too much RAM
after: 994 ms ± 29.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This causes memory to chew up to 6GB+ in a matter of seconds.
Took me all day to figure this out!
Would be good to include some sort of blacklist of weird characters and prevent the mysterious memory hog, eg throw an error (and provide some api to check if a string is spellable)
The text was updated successfully, but these errors were encountered: