Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Time out for function #13

Open
paulaceccon opened this issue Mar 1, 2018 · 3 comments
Open

Time out for function #13

paulaceccon opened this issue Mar 1, 2018 · 3 comments

Comments

@paulaceccon
Copy link

Given a list of words, I'm looping trough them to correct any misspelled words. For now I'm stuck in the 5738 word for more than five minutes, with memory usage up to 12GB of RAM and disk usage of 120MB/s.
It would be nice to have a time out parameter to abort if the search is taking to long. It probably possible to optimize the memory usage either.

@paulaceccon
Copy link
Author

I was able to go around the time issue with this package.

CODE SAMPLE:

from tqdm import tqdm
import stopit

@stopit.threading_timeoutable(default='aborted')
def speel_corrector(word):
    return spell(word)

aborted = []
to_replace = {}
for word in tqdm(results):
    new_word = speel_corrector(word, timeout=20)
    
    if new_word == 'aborted':
        aborted.append(word)
    elif new_word != word:
        to_replace[word] = new_word

@ztmeyer
Copy link

ztmeyer commented Apr 27, 2018

@paulaceccon Thanks for reporting this issue and posting your resolution!

I also encountered this. For example, this spell() invocation took 10 seconds to resolve on my machine:
spell('................................................................')

One workaround is to only run spell() on strings composed of alphanumerics:

import re

if re.match("^[\w]+$", word):
  return spell(word)

@dylan-chong
Copy link

This seems to be related to #18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants