Skip to content
This repository has been archived by the owner on Apr 4, 2023. It is now read-only.

ValueError: Cannot execute operation using `run() and CUBLAS_STATUS_EXECUTION_FAILED #60

Open
hppp123 opened this issue Oct 8, 2021 · 0 comments

Comments

@hppp123
Copy link

hppp123 commented Oct 8, 2021

Dear author,
I am a student from Hebei,I'm very interested in your work.While trying to run your code(VoxelMorph-5-liver on eval.py), I have some problems and would like to get your advice.I used the Anacoda virtual environment, and here is my settings:cudatoolkit=10.0.130/cudnn=7.6.5/keras=2.3.1/tensorflow=1.13.1/tensorflow-gpu=1.13.1/tflearn=0.3.2

First,the error ValueError: Cannot execute operation using `run()

File "eval.py", line 148, in
main()
File "eval.py", line 83, in main
tf.global_variables_initializer().run()
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2450, in run
_run_using_default_session(self, feed_dict, self.graph, session)
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 5202, in _run_using_default_session
raise ValueError("Cannot execute operation using run(): No default "
ValueError: Cannot execute operation using run(): No default session is registered. Use with sess.as_default(): or pass an explicit session to run(session=sess)

I tried to change tf.global_variables_initializer().run() to tf.global_variables_initializer().run(session=sess)
I'm not sure that's right.

And then another error

2021-10-07 11:31:54.083602: E tensorflow/stream_executor/cuda/cuda_blas.cc:698] failed to run cuBLAS routine cublasSgemm_v2: CUBLAS_STATUS_EXECUTION_FAILED
0it [14:38, ?it/s]
Traceback (most recent call last):
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(32, 4), b.shape=(4, 588), m=32, n=588, k=4
[[{{node MatMul_3}}]]
[[{{node gaffdfrm/dense3d_spatial_transformer_2/ExpandDims_5}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "eval.py", line 148, in
main()
File "eval.py", line 112, in main
results = framework.validate(sess, gen, keys=keys, summary=False, show_tqdm=True)
File "/home/chen/Gzj/Recursive-Cascaded-Networks/Recursive-Cascaded-Networks-master/network/framework.py", line 160, in validate
*keys), feed_dict=set_tf_keys(fd))
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(32, 4), b.shape=(4, 588), m=32, n=588, k=4
[[node MatMul_3 (defined at /home/chen/Gzj/Recursive-Cascaded-Networks/Recursive-Cascaded-Networks-master/network/transform.py:132) ]]
[[node gaffdfrm/dense3d_spatial_transformer_2/ExpandDims_5 (defined at /home/chen/Gzj/Recursive-Cascaded-Networks/Recursive-Cascaded-Networks-master/network/spatial_transformer.py:102) ]]

Caused by op 'MatMul_3', defined at:
File "eval.py", line 148, in
main()
File "eval.py", line 75, in main
'segmentation_class_value', None), fast_reconstruction=args.fast_reconstruction, validation=True)
File "/home/chen/Gzj/Recursive-Cascaded-Networks/Recursive-Cascaded-Networks-master/network/framework.py", line 54, in init
augFlow = transform.free_form_fields(imgs, control_fields)
File "/home/chen/Gzj/Recursive-Cascaded-Networks/Recursive-Cascaded-Networks-master/network/transform.py", line 132, in free_form_fields
for i in range(0, n - 1)],
File "/home/chen/Gzj/Recursive-Cascaded-Networks/Recursive-Cascaded-Networks-master/network/transform.py", line 132, in
for i in range(0, n - 1)],
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 2455, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5333, in mat_mul
name=name)
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/home/chen/anaconda3/envs/R/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()

InternalError (see above for traceback): Blas GEMM launch failed : a.shape=(32, 4), b.shape=(4, 588), m=32, n=588, k=4
[[node MatMul_3 (defined at /home/chen/Gzj/Recursive-Cascaded-Networks/Recursive-Cascaded-Networks-master/network/transform.py:132) ]]
[[node gaffdfrm/dense3d_spatial_transformer_2/ExpandDims_5 (defined at /home/chen/Gzj/Recursive-Cascaded-Networks/Recursive-Cascaded-Networks-master/network/spatial_transformer.py:102) ]]

Could you tell me how to handle this error? Sincerely hope to get your reply.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant