Skip to content
This repository has been archived by the owner on Sep 18, 2024. It is now read-only.

char-level Tokenizer.sequences_to_texts() insert additional SPACEs #346

Open
2 tasks done
XiYuan68 opened this issue Jul 29, 2021 · 0 comments
Open
2 tasks done

char-level Tokenizer.sequences_to_texts() insert additional SPACEs #346

XiYuan68 opened this issue Jul 29, 2021 · 0 comments
Labels
text Related to text

Comments

@XiYuan68
Copy link

  • Check that you are up-to-date with the master branch of keras-preprocessing. You can update with:
    pip install git+git://github.com/keras-team/keras-preprocessing.git --upgrade --no-deps

  • Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

Describe the problem.

from tensorflow.keras.preprocessing.text import Tokenizer

text = ['abc def']
tokenizer = Tokenizer(char_level=True, split='')
tokenizer.fit_on_texts(text)
sequence = tokenizer.texts_to_sequences(text)
text_after = tokenizer.sequences_to_texts(sequence)

print(text_after)
>>> ['a b c   d e f']

notice that text_after and text are different, additional SPACEs are inserted

Describe the expected behavior.

text_after should be same as text

I believe this line is where the problem is, replacing:

vect = ' '.join(vect)

with

vect = self.split.join(vect)

will fix the bug in my mini case

@XiYuan68 XiYuan68 added the text Related to text label Jul 29, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
text Related to text
Projects
None yet
Development

No branches or pull requests

1 participant