Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failure while compiling ray's native tensorflow ops. #2

Open
Ji4chenLi opened this issue Aug 11, 2020 · 1 comment
Open

Failure while compiling ray's native tensorflow ops. #2

Ji4chenLi opened this issue Aug 11, 2020 · 1 comment

Comments

@Ji4chenLi
Copy link

TensorFlow version: 1.13.2
Compiling Plasma TensorFlow Op...
Running command ['g++', '-std=c++11', '-g', '-shared', '/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc', '-o', '/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.so', '-DNDEBUG', '-I/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/include', '-L/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow', '-lplasma', '-larrow_python', '-larrow', '-fPIC', '-I/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include', '-D_GLIBCXX_USE_CXX11_ABI=1', '-L/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow', '-ltensorflow_framework', '-O2', '-DGOOGLE_CUDA']
In file included from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:36:0:
/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/include/arrow/util/logging.h:96:0: warning: "DCHECK" redefined
 #define DCHECK(condition)                     \
 
In file included from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/logging.h:25:0,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/lib/core/refcount.h:22,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/tensor_coding.h:21,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/resource_handle.h:19,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/allocator.h:24,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/tensor.h:22,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/device_base.h:24,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:18:
/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/default/logging.h:303:0: note: this is the location of the previous definition
 #define DCHECK(condition) \
 
In file included from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:36:0:
/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/include/arrow/util/logging.h:102:0: warning: "DCHECK_EQ" redefined
 #define DCHECK_EQ(val1, val2)            \
 
In file included from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/logging.h:25:0,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/lib/core/refcount.h:22,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/tensor_coding.h:21,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/resource_handle.h:19,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/allocator.h:24,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/tensor.h:22,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/device_base.h:24,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:18:
/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/default/logging.h:313:0: note: this is the location of the previous definition
 #define DCHECK_EQ(x, y) _TF_DCHECK_NOP(x, y)
 
In file included from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:36:0:
/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/include/arrow/util/logging.h:106:0: warning: "DCHECK_NE" redefined
 #define DCHECK_NE(val1, val2)            \
 
In file included from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/logging.h:25:0,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/lib/core/refcount.h:22,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/tensor_coding.h:21,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/resource_handle.h:19,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/allocator.h:24,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/tensor.h:22,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/device_base.h:24,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:18:
/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/default/logging.h:314:0: note: this is the location of the previous definition
 #define DCHECK_NE(x, y) _TF_DCHECK_NOP(x, y)
 
In file included from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:36:0:
/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/include/arrow/util/logging.h:110:0: warning: "DCHECK_LE" redefined
 #define DCHECK_LE(val1, val2)            \
 
In file included from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/logging.h:25:0,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/lib/core/refcount.h:22,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/tensor_coding.h:21,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/resource_handle.h:19,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/allocator.h:24,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/tensor.h:22,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/device_base.h:24,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:18:
/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/default/logging.h:315:0: note: this is the location of the previous definition
 #define DCHECK_LE(x, y) _TF_DCHECK_NOP(x, y)
 
In file included from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:36:0:
/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/include/arrow/util/logging.h:114:0: warning: "DCHECK_LT" redefined
 #define DCHECK_LT(val1, val2)            \
 
In file included from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/logging.h:25:0,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/lib/core/refcount.h:22,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/tensor_coding.h:21,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/resource_handle.h:19,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/allocator.h:24,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/tensor.h:22,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/device_base.h:24,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:18:
/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/default/logging.h:316:0: note: this is the location of the previous definition
 #define DCHECK_LT(x, y) _TF_DCHECK_NOP(x, y)
 
In file included from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:36:0:
/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/include/arrow/util/logging.h:118:0: warning: "DCHECK_GE" redefined
 #define DCHECK_GE(val1, val2)            \
 
In file included from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/logging.h:25:0,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/lib/core/refcount.h:22,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/tensor_coding.h:21,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/resource_handle.h:19,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/allocator.h:24,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/tensor.h:22,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/device_base.h:24,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:18:
/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/default/logging.h:317:0: note: this is the location of the previous definition
 #define DCHECK_GE(x, y) _TF_DCHECK_NOP(x, y)
 
In file included from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:36:0:
/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/include/arrow/util/logging.h:122:0: warning: "DCHECK_GT" redefined
 #define DCHECK_GT(val1, val2)            \
 
In file included from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/logging.h:25:0,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/lib/core/refcount.h:22,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/tensor_coding.h:21,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/resource_handle.h:19,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/allocator.h:24,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/tensor.h:22,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/framework/device_base.h:24,
                 from /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.cc:18:
/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/include/tensorflow/core/platform/default/logging.h:318:0: note: this is the location of the previous definition
 #define DCHECK_GT(x, y) _TF_DCHECK_NOP(x, y)
 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/plasma.py", line 79, in build_plasma_tensorflow_op
    tf_plasma_op = tf.load_op_library(TF_PLASMA_OP_PATH)
  File "/home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py", line 61, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/jiachen/miniconda3/envs/test/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/tensorflow/plasma_op.so: undefined symbol: _ZN5arrow6StatusC1ENS_10StatusCodeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

Hi,

First thank you so much for the great work, the idea presented in the paper is really inspiring!

Recently I am trying to reproduce the results but came across the error above while compiling ray's native tensorflow ops. I follow the instructions to use

tensorflow-gpu==1.13.2
ray[tune]==0.7.7

Can you help me with this?

@louiskirsch
Copy link
Owner

Hi Jiachen,

thank you for your kind words. Apologies for the kind of outdated software stack (tensorflow 1.x aged quickly).
Can you try running these commands in order?

git clone https://github.com/louiskirsch/metagenrl
cd metagenrl/
python3 -m venv env
. env/bin/activate
pip3 install wheel setuptools psutil
pip3 install ray[tune]==0.7.7 gym[all] mujoco_py==2.0.2.8 tensorflow-gpu==1.13.2 scipy numpy requests
python3 -c 'import ray; from pyarrow import plasma as plasma; plasma.build_plasma_tensorflow_op()'
python3 ray_experiments.py train

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants