Skip to content
This repository has been archived by the owner on Sep 18, 2024. It is now read-only.

Tokenizer ignores filters if char_level is True #301

Open
2 tasks done
paw-lu opened this issue Jun 9, 2020 · 1 comment · May be fixed by #302
Open
2 tasks done

Tokenizer ignores filters if char_level is True #301

paw-lu opened this issue Jun 9, 2020 · 1 comment · May be fixed by #302
Labels
text Related to text

Comments

@paw-lu
Copy link

paw-lu commented Jun 9, 2020

Please make sure that the boxes below are checked before you submit your issue.
If your issue is an implementation question, please ask your question on StackOverflow or on the Keras Slack channel instead of opening a GitHub issue.

Thank you!

  • Check that you are up-to-date with the master branch of keras-preprocessing. You can update with:
    pip install git+git://github.com/keras-team/keras-preprocessing.git --upgrade --no-deps

  • Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

What I expect

Tokenizer should not tokenize characters listed in filters.

What happens

If char_level=True, Tokenizer will tokenize all characters—even those listed in filters.

Code

text = "ae"tokenizer = keras.preprocessing.text.Tokenizer(filters="e")  # Ignore "e"tokenizer.fit_on_texts(text)
❯ tokenizer.word_index
{'a': 1}  # Ignores "e" as expectedtokenizer = keras.preprocessing.text.Tokenizer(char_level=True, filters="e")
❯ tokenizer.fit_on_texts(text)
❯ tokenizer.word_index
{'a': 1, 'e': 2}  # "e" is tokenized
@paw-lu paw-lu added the text Related to text label Jun 9, 2020
@paw-lu paw-lu linked a pull request Jun 9, 2020 that will close this issue
4 tasks
@paw-lu
Copy link
Author

paw-lu commented Jun 9, 2020

A fix has been attempted in #302

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
text Related to text
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant