Distributed training

[JRMeyer]

>>> chesterkuo
[February 8, 2018, 10:06am]

Hi there

I had finished the distributed training across multiple PC with multiple
worker , each worker have their own checkpoint folder specific as well,
question i'm not sure is , how to export model for inference with
cluster case ??

Check each worker checkpoint file and see which one have latest
checkpoint file and export model foo any suggestion ?



[Distributed training set up of DeepSpeech code

[This is an archived TTS discussion thread from discourse.mozilla.org/t/distributed-training]

[JRMeyer]

>>> jageshmaharjan
[May 16, 2018, 10:42am]

, How did u add
server ip for distributed training. slash
I did this but i encounter errors, maybe I am not doing wright way. Can
you point what's wong in this script.

python -u DeepSpeech.py slash
--train_files /data/zh_data/data_thchs30/train.csv slash
--dev_files /data/zh_data/data_thchs30/dev.csv slash
--test_files /data/zh_data/data_thchs30/test.csv slash
--train_batch_size 80 slash
--dev_batch_size 80 slash
--test_batch_size 40 slash
--n_hidden 375 slash
--epoch 200 slash
--validation_step 1 slash
--early_stop True slash
--earlystop_nsteps 6 slash
--estop_mean_thresh 0.1 slash
--estop_std_thresh 0.1 slash
--dropout_rate 0.22 slash
--learning_rate 0.00095 slash
--report_count 100 slash
--use_seq_length False slash
--export_dir /data/zh_data/exportDir/ slash
--checkpoint_dir /data/zh_data/checkpoint/ slash
--decoder_library_path /data/jugs/asr/DeepSpeech/native_client/libctc_decoder_with_kenlm.so slash
--alphabet_config_path /data/zh_data/alphabet.txt slash
--lm_binary_path /data/zh_data/zh_lm.binary slash
--lm_trie_path /data/zh_data/trie slash
--ps_hosts '104.211.xx.xx:2222' slash

The error is on --ps_host parameter. If its not that way to assign
parameter server, how should i do. And, my error is:

Traceback (most recent call last):
File 'DeepSpeech.py', line 1838, in
tf.app.run()
File '/home/maybe/anaconda3/envs/asr/lib/python3.6/site-packages/tensorflow/python/platform/app.py', line 126, in run
_sys.exit(main(argv))
File 'DeepSpeech.py', line 1795, in main
train()
File 'DeepSpeech.py', line 1501, in train
results_tuple, gradients, mean_edit_distance, loss = get_tower_results(model_feeder, optimizer)
File 'DeepSpeech.py', line 633, in get_tower_results
device = tf.train.replica_device_setter(worker_device=available_devices[i], cluster=cluster)
File '/home/maybe/anaconda3/envs/asr/lib/python3.6/site-packages/tensorflow/python/training/device_setter.py', line 197, in replica_device_setter
cluster_spec = cluster.as_dict()
File '/home/maybe/anaconda3/envs/asr/lib/python3.6/site-packages/tensorflow/python/training/server_lib.py', line 334, in as_dict
if max(task_indices) + 1 == len(task_indices):
ValueError: max() arg is an empty sequence

[Archived Post]

[JRMeyer]

>>> Tilman_Kamp
[May 22, 2018, 8:36am]

Have you
forgotten to assign the slash --worker_hosts parameter?

[Archived Post]

[JRMeyer]

>>> jageshmaharjan
[May 22, 2018, 8:54am]

I have it in my script , Sorry, i think i didn't paste here. slash
Lemme edit again.
[ slash Tilman_Kamp](

This is the host server (ps) and this is my script:

> python -u DeepSpeech.py slash
> --train_files /data/zh_data/data_thchs30/train.csv slash
> --dev_files /data/zh_data/data_thchs30/dev.csv slash
> --test_files /data/zh_data/data_thchs30/test.csv slash
> --train_batch_size 80 slash
> --dev_batch_size 80 slash
> --test_batch_size 40 slash
> --n_hidden 375 slash
> --epoch 200 slash
> --validation_step 1 slash
> --early_stop True slash
> --earlystop_nsteps 6 slash
> --estop_mean_thresh 0.1 slash
> --estop_std_thresh 0.1 slash
> --dropout_rate 0.22 slash
> --learning_rate 0.00095 slash
> --report_count 100 slash
> --use_seq_length False slash
> --export_dir /data/zh_data/exportDir/distributedTf/ slash
> --checkpoint_dir /data/zh_data/checkpoint/distributedCkp/ slash
> --decoder_library_path
> /data/jugs/asr/DeepSpeech/native_client/libctc_decoder_with_kenlm.so slash
> --alphabet_config_path /data/zh_data/alphabet.txt slash
> --lm_binary_path /data/zh_data/zh_lm.binary slash
> --lm_trie_path /data/zh_data/trie slash
> --ps_hosts localhost:2233 slash
> --worker_hosts localhost:2222 slash
> --task_index 0 slash
> --job_name ps slash slash

And, this is my worker machine, and this is the script:

> python -u DeepSpeech.py slash
> --train_files /data/zh_data/data_thchs30/train.csv slash
> --dev_files /data/zh_data/data_thchs30/dev.csv slash
> --test_files /data/zh_data/data_thchs30/test.csv slash
> --train_batch_size 80 slash
> --dev_batch_size 80 slash
> --test_batch_size 40 slash
> --n_hidden 375 slash
> --epoch 200 slash
> --validation_step 1 slash
> --early_stop True slash
> --earlystop_nsteps 6 slash
> --estop_mean_thresh 0.1 slash
> --estop_std_thresh 0.1 slash
> --dropout_rate 0.22 slash
> --learning_rate 0.00095 slash
> --report_count 100 slash
> --use_seq_length False slash
> --export_dir /data/zh_data/exportDir/distributedTf/ slash
> --checkpoint_dir /data/zh_data/checkpoint/distributedCkp/ slash
> --decoder_library_path
> /data/jugs/asr/DeepSpeech/native_client/libctc_decoder_with_kenlm.so slash
> --alphabet_config_path /data/zh_data/alphabet.txt slash
> --lm_binary_path /data/zh_data/zh_lm.binary slash
> --lm_trie_path /data/zh_data/trie slash
> --ps_hosts localhost:2233 slash
> --worker_hosts localhost:2222 slash
> --task_index 0 slash
> --job_name worker slash
> --coord_host localhost slash
> --coord_port 2222

[Archived Post]

[JRMeyer]

>>> Tilman_Kamp
[May 22, 2018, 9:42am]

Ah OK. Far more complete now. Two things:

parameters (one coodination service all talk to).
used by any worker or ps.

[Archived Post]

[JRMeyer]

>>> jageshmaharjan
[May 23, 2018, 11:55am]

Hi
[ slash Tilman_Kamp](
the rest of the script are same. Just changed the port of the
coordinator. slash
And, I was getting this Error trace.

In worker machine:

> Instructions for updating: slash
> Use the retry module or similar alternatives. slash
> E0523 11:47:02.841798575 86717 server_chttp2.cc:38 slash ]
> {'created': slash '[ slash 1527076022.841733775] slash ','description':'No
> address added out of total 1
> resolved','file':'external/grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc','file_line':302,'referenced_errors': slash [{'created': slash '[ slash 1527076022.841731175] slash ','description':'Failed
> to add any wildcard
> listeners','file':'external/grpc/src/core/lib/iomgr/tcp_server_posix.cc','file_line':337,'referenced_errors': slash [{'created': slash '[ slash 1527076022.841714076] slash ','description':'Unable
> to configure
> socket','fd':53,'file':'external/grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc','file_line':200,'referenced_errors': slash [{'created': slash '[ slash 1527076022.841709676] slash ','description':'OS
> Error','errno':98,'file':'external/grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc','file_line':173,'os_error':'Address
> already in
> use','syscall':'bind'} slash ]},{'created': slash '[ slash 1527076022.841730775] slash ','description':'Unable
> to configure
> socket','fd':53,'file':'external/grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc','file_line':200,'referenced_errors': slash [{'created': slash '[ slash 1527076022.841727075] slash ','description':'OS
> Error','errno':98,'file':'external/grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc','file_line':173,'os_error':'Address
> already in use','syscall':'bind'} slash ]} slash ]} slash ]} slash
> Traceback (most recent call last): slash
> File 'DeepSpeech.py', line 1838, in slash
> tf.app.run() slash
> File
> '/home/maybe/anaconda3/envs/asr/lib/python3.6/site-packages/tensorflow/python/platform/app.py',
> line 126, in run slash
> slash _sys.exit(main(argv)) slash
> File 'DeepSpeech.py', line 1799, in main slash
> server = tf.train.Server(cluster, job_name=FLAGS.job_name,
> task_index=FLAGS.task_index) slash
> File
> '/home/maybe/anaconda3/envs/asr/lib/python3.6/site-packages/tensorflow/python/training/server_lib.py',
> line 147, in init slash
> self. slash _server_def.SerializeToString(), status) slash
> File
> '/home/maybe/anaconda3/envs/asr/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py',
> line 516, in exit slash
> c_api.TF_GetCode(self.status.status)) slash
> tensorflow.python.framework.errors_impl.UnknownError: Could not start
> gRPC server

and in Host machine (PS), it doesn't show anything than this,

> Instructions for updating: slash
> Use the retry module or similar alternatives.

[Archived Post]

[JRMeyer]

>>> jageshmaharjan
[May 23, 2018, 12:05pm]

Oh, Never-mind
[ slash Tilman_Kamp](
It was in the background consuming that port. I killed the process, now,
I think it started to train. slash
Yea, I changed the coordinator port and I think it works fine. Thanks
[ slash Tilman_Kamp](
![:slight_smile:](

[Archived Post]

[JRMeyer]

>>> jageshmaharjan
[May 24, 2018, 3:13am]

Hi
[ slash Tilman_Kamp](
got another error after sometime during training.

> 2018-05-24 03:05:38.015799: F
> tensorflow/core/common_runtime/gpu/gpu_util.cc:343 slash ] CPU- slash >GPU Memcpy
> failed slash
> Aborted (core dumped)

Thinking, unallocated GPU usage by previous process. I re-ran the worker
script. Got another traces.

> E You must feed a value for placeholder tensor 'Placeholder_5' with
> dtype int32 slash
> E slash [ slash [Node: Placeholder_5 = Placeholderdtype=DT_INT32, shape= slash [ slash ],
> slash _device= slash '/job:worker/replica:0/task:0/device:CPU:0 slash ' slash ] slash ] slash
> E slash [ slash [Node: b3/read_S591_G3013 = slash _Recvclient_terminated=false,
> recv_device= slash '/job:worker/replica:0/task:0/device:GPU:1 slash ',
> send_device= slash '/job:worker/replica:0/task:0/device:GPU:3 slash ',
> send_device_incarnation=-8674475802652740309,
> tensor_name='edge_2390_b3/read_S591', tensor_type=DT_FLOAT,
> slash _device= slash '/job:worker/replica:0/task:0/device:GPU:1 slash ' slash ] slash ] slash
> E slash
> E Caused by op 'Placeholder_5', defined at: slash
> E File 'DeepSpeech.py', line 1838, in slash
> E tf.app.run() slash
> E File
> '/home/maybe/anaconda3/envs/asr/lib/python3.6/site-packages/tensorflow/python/platform/app.py',
> line 126, in run slash
> E slash _sys.exit(main(argv)) slash
> E File 'DeepSpeech.py', line 1820, in main slash
> E train(server) slash
> E File 'DeepSpeech.py', line 1489, in train slash
> E tower_feeder_count=len(available_devices)) slash
> E File '/data/jugs/asr/DeepSpeech/util/feeding.py', line 42, in
> init slash
> E self.ph_batch_size = tf.placeholder(tf.int32, slash [ slash ]) slash
> E File
> '/home/maybe/anaconda3/envs/asr/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py',
> line 1777, in placeholder slash
> E return gen_array_ops.placeholder(dtype=dtype, shape=shape,
> name=name) slash
> E File
> '/home/maybe/anaconda3/envs/asr/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py',
> line 4521, in placeholder slash
> E 'Placeholder', dtype=dtype, shape=shape, name=name) slash
> E File
> '/home/maybe/anaconda3/envs/asr/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py',
> line 787, in slash _apply_op_helper slash
> E op_def=op_def) slash
> E File
> '/home/maybe/anaconda3/envs/asr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py',
> line 3290, in create_op slash
> E op_def=op_def) slash
> E File
> '/home/maybe/anaconda3/envs/asr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py',
> line 1654, in init slash
> E self. slash _traceback = self. slash _graph. slash _extract_stack() slash # pylint:
> disable=protected-access slash
> E slash
> E InvalidArgumentError (see above for traceback): You must feed a
> value for placeholder tensor 'Placeholder_5' with dtype int32 slash
> E slash [ slash [Node: Placeholder_5 = Placeholderdtype=DT_INT32, shape= slash [ slash ],
> slash _device= slash '/job:worker/replica:0/task:0/device:CPU:0 slash ' slash ] slash ] slash
> E slash [ slash [Node: b3/read_S591_G3013 = slash _Recvclient_terminated=false,
> recv_device= slash '/job:worker/replica:0/task:0/device:GPU:1 slash ',
> send_device= slash '/job:worker/replica:0/task:0/device:GPU:3 slash ',
> send_device_incarnation=-8674475802652740309,
> tensor_name='edge_2390_b3/read_S591', tensor_type=DT_FLOAT,
> slash _device= slash '/job:worker/replica:0/task:0/device:GPU:1 slash ' slash ] slash ] slash
> E slash
> E The checkpoint in /data/zh_data/checkpoint/distributedCkp/ does not
> match the shapes of the model. Did you change alphabet.txt or the
> slash --n_hidden parameter between train runs using the same checkpoint
> dir? Try moving or removing the contents of
> /data/zh_data/checkpoint/distributedCkp/.

[Archived Post]

[JRMeyer]

>>> gr8nishan
[December 13, 2018, 9:10am]

can you
please share both the scripts that is working for you.

[Archived Post]

[JRMeyer]

>>> jageshmaharjan
[December 13, 2018, 10:47am]

. There could be two
reasons to have that error.

1. Not available GPU memory (either being used by other process or not
enough GPU)
2. The path (directory) for saving checkpoints already have checkpoints
with different hyperparameters.

Also, make sure, GitLarge File system (gfs) is installed.

[Archived Post]

[JRMeyer]

>>> gr8nishan
[December 13, 2018, 12:02pm]

i am trying
to do distributed training across CPUs . I believe that should be
possible.

[Archived Post]

[JRMeyer]

>>> jageshmaharjan
[December 17, 2018, 8:26am]

Yea, off-course that's possible too. However, that will take long
duration depending on your data.

[Archived Post]