Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [Jeff thought you argued in favor of a centrifuge purchase. <eos>] #55

Open
nipengmath opened this issue Jan 23, 2019 · 9 comments
Open

tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [Jeff thought you argued in favor of a centrifuge purchase. <eos>] #55

nipengmath opened this issue Jan 23, 2019 · 9 comments

Comments

@nipengmath
Copy link

Hi Vincent!
I got a problem when i run the recipe LAS/TIMIT, can you give me some help to fix this error? Thanks !

File "/data/nipeng/nabu/nabu/neuralnetworks/trainers/trainer.py", line 819, in train
outputs['training_summaries']])
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 695, in exit
self._close_internal(exception_type)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 732, in _close_internal
self._sess.close()
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 980, in close
self._sess.close()
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1124, in close
ignore_live_threads=True)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/queue_runner_impl.py", line 252, in _run
enqueue_callable()
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1205, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [Jeff thought you argued in favor of a centrifuge purchase. ]
@vrenkens
Copy link
Owner

vrenkens commented Jan 23, 2019 via email

@jtcampbellsoup
Copy link

Hi Vincent,

I'm running into a similar problem to this, I believe. Can you clarify what you mean by setting the target file to contain the phonetic transcriptions?

I'm using the DBLSTM/TIMIT recipe, and the only changes I've made to it are to the database.cfg file, where I set the data files to the proper locations. So the wave files were processed by the feature_processor.cfg and the text was processed by the text_processor.cfg successfully. (I am using a dataset of isolated-one word utterances and their transcriptions from people with dysarthria).

Now, when I run: ./run train --recipe=config/recipes/DBLSTM/TIMIT --expdir=Models/DBLSTM_01 --mode=non_distributed --computing=standard

I get the following traceback error:

Traceback (most recent call last):
File "nabu/scripts/prepare_train.py", line 366, in
main(FLAGS.expdir, FLAGS.recipe, FLAGS.mode, FLAGS.computing)
File "nabu/scripts/prepare_train.py", line 91, in main
expdir=expdir)
File "/home/ubuntu/cs230project/SR-dysarthria/nab/nabu/scripts/train.py", line 90, in train
tr.train(testing)
File "/home/ubuntu/cs230project/SR-dysarthria/nab/nabu/neuralnetworks/trainers/trainer.py", line 787, in train
outputs['increment_step'].run(session=sess)
File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 788, in exit
self._close_internal(exception_type)
File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 826, in _close_internal
self._sess.close()
File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1082, in close
self._sess.close()
File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1244, in close
ignore_live_threads=True)
File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/training/queue_runner_impl.py", line 257, in _run
enqueue_callable()
File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1257, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [behavior]
[[{{node validate/validation/evaluate/input_pipeline/read_data/reader_1/StringReader/assert_equal/Assert/Assert}}]]

Any help by either of you would be greatly appreciated!

@vrenkens
Copy link
Owner

Hey, TIMIT has both textual (written) transcriptions and phonetic transcriptions (sequences of phones). The textual transcriptions are just text, meaning we can read it. The phonetic transcriptions contain phones (symbols like sh aa ae...). The TIMIT recipe for Nabu counts on you using the phonetic transcriptions. It's been a bit too long for me to remember what the path to the phonetic transcriptions typically is in TIMIT... You can probably find some more information by googling it or just exploring the files

@jtcampbellsoup
Copy link

Hmmm, I apologize but I am still a bit confused by this. In the DBLSTM/TIMIT recipe's database.cfg file, it specifies to use the text_processor on the text data. The text_processor.cfg file looks like this:

tp

So it's taking in text data, and processing it as phoenetic transcriptions, right? Which are then used for training. I've done this pre-processing and the files were saved to the location specified in the database.cfg file. So why is the word "behavior" still showing up in the error above? Should that not be the case?

Really appreciate your help on this. The flexibility of this repository is fantastic!

@vrenkens
Copy link
Owner

Hi, the term text_processor is probably a bit confusing. In this case it should actually be processing phones. So it should not be taking in text data, it should be taking in phone data. Could maybe show the data files you have available for limit, what you have defined in your database.cfg file and the content of these files?

@jtcampbellsoup
Copy link

Here is the database.cfg file in that recipe:
audio
text

Here is the train.scp file that it points to, where each line matches a label to a path to a wave file on my local computer:
scp

Here is the train.txt file that it points to, where each line matches a label to the transcript of the corresponding wave file:
txt

It seems perhaps that the the train.txt file is incorrect, and should be listing phones instead? And if so, does the "decoder" part of the LSTM network decode the information back to phones (and therefore output phones) or does it output full words?

@vrenkens
Copy link
Owner

It should indeed be listing phones. The decoder outputs phones, not full words.

@jtcampbellsoup
Copy link

Any recommendations of how I can decompose my words into phones?

@vrenkens
Copy link
Owner

Usually this is done when you do the Kaldi data prep for TIMIT

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nipengmath @vrenkens @jtcampbellsoup