You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Sep 18, 2024. It is now read-only.
Check that you are up-to-date with the master branch of keras-preprocessing. You can update with: pip install git+git://github.com/keras-team/keras-preprocessing.git --upgrade --no-deps
Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).
Describe the problem.
from tensorflow.keras.preprocessing.text import Tokenizer
text = ['abc def']
tokenizer = Tokenizer(char_level=True, split='')
tokenizer.fit_on_texts(text)
sequence = tokenizer.texts_to_sequences(text)
text_after = tokenizer.sequences_to_texts(sequence)
print(text_after)
>>> ['a b c d e f']
notice that text_after and text are different, additional SPACEs are inserted
Describe the expected behavior.
text_after should be same as text
I believe this line is where the problem is, replacing:
vect = ' '.join(vect)
with
vect = self.split.join(vect)
will fix the bug in my mini case
The text was updated successfully, but these errors were encountered:
Check that you are up-to-date with the master branch of keras-preprocessing. You can update with:
pip install git+git://github.com/keras-team/keras-preprocessing.git --upgrade --no-deps
Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).
Describe the problem.
notice that
text_after
andtext
are different, additional SPACEs are insertedDescribe the expected behavior.
text_after
should be same astext
I believe this line is where the problem is, replacing:
vect = ' '.join(vect)
with
vect = self.split.join(vect)
will fix the bug in my mini case
The text was updated successfully, but these errors were encountered: