[BUG] Memcheck error found in CSV_TEST CsvReaderNumericTypeTest/0.SingleColumn

[davidwendt]

Describe the bug
The nightly builds memcheck found an error in the CSV_TEST CsvReaderNumericTypeTest/0.SingleColumn
No other CSV_TEST gtests are failing.

Steps/Code to reproduce bug

# compute-sanitizer --tool memcheck gtests/CSV_TEST --gtest_filter=CsvReaderNumericTypeTest/0.SingleColumn --rmm_mode=cuda
========= COMPUTE-SANITIZER
Note: Google Test filter = CsvReaderNumericTypeTest/0.SingleColumn
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CsvReaderNumericTypeTest/0, where TypeParam = long
[ RUN      ] CsvReaderNumericTypeTest/0.SingleColumn
========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxPopCurrent_v2.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x2cbe56]
=========                in /usr/lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame: [0xd999f]
=========                in /conda/envs/rapids/lib/libcufile.so.0

[davidwendt]

This is definitely something introduced in the code stack.
I can get the error to produce on various versions of compute-sanitizer including

• 2022.3.0 (11.8)
• 2022.2.1 (11.7)
• 2022.1.1 (11.6)

The stack from includes libcufile.so so perhaps there is a change to that recently?

[davidwendt]

The nightly builds started failing on September 19.
I tried building locally with a commit before that day: 3b691f4
But that fails (as per the description) as well.
I'm not sure the next steps to track this down.

[davidwendt]

Looks like this error has been occurring for awhile but never triggered a memcheck build failure. I checked logs back to the beginning of July and found the error has been occurring at least since then (logs are not kept before that).

[vuule]

Ran the repro locally; it looks like this test fails only because it's first one. Running a different CSV test also leads to the error.
Bit more of the stack:

========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxPopCurrent_v2.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x2aa4f4]
=========                in /lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame: [0x12204f]
=========                in /home/vukasin/anaconda3/envs/cudf_dev/lib/libcufile.so.0
=========     Host Frame: [0x1251a4]
=========                in /home/vukasin/anaconda3/envs/cudf_dev/lib/libcufile.so.0
=========     Host Frame: [0x1256bd]
=========                in /home/vukasin/anaconda3/envs/cudf_dev/lib/libcufile.so.0
=========     Host Frame: [0x12c83e]
=========                in /home/vukasin/anaconda3/envs/cudf_dev/lib/libcufile.so.0
=========     Host Frame:cuFileDriverOpen [0x12ea29]
=========                in /home/vukasin/anaconda3/envs/cudf_dev/lib/libcufile.so.0
=========     Host Frame:kvikio::cuFileAPI::cuFileAPI() [0x150688c]
=========                in /home/vukasin/anaconda3/envs/cudf_dev/lib/libcudf.so
=========     Host Frame:kvikio::defaults::defaults() [0x1509255]
=========                in /home/vukasin/anaconda3/envs/cudf_dev/lib/libcudf.so
=========     Host Frame:kvikio::defaults::instance() [0x150956c]
=========                in /home/vukasin/anaconda3/envs/cudf_dev/lib/libcudf.so
=========     Host Frame:cudf::io::(anonymous namespace)::file_source::file_source(char const*) [0x1517cf7]
=========                in /home/vukasin/anaconda3/envs/cudf_dev/lib/libcudf.so
=========     Host Frame:cudf::io::datasource::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long) [0x1519eb0]

[vuule]

Looks like we forgot to update the issue: we discovered that there is a cuFile bug where cuFileDriverOpen fails if it's the first CUDA call in the thread/process. So, not a libcudf bug. We can add a simple workaround if needed.

[davidwendt]

... So, not a libcudf bug. We can add a simple workaround if needed.

I definitely would appreciate a workaround added for this since this generates a significant amount of noise in the memcheck builds.

[davidwendt]

Looks like this has been fixed for CSV_TEST but I'm still seeing a failure in the first PARQUET_TEST

Running compute-sanitizer on PARQUET_TEST
========= COMPUTE-SANITIZER
[==========] Running 325 tests from 94 test suites.
[----------] Global test environment set-up.
[----------] 11 tests from ParquetChunkedReaderTest
[ RUN      ] ParquetChunkedReaderTest.TestChunkedReadNoData
========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxPopCurrent_v2.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame: [0x2ce616]
=========                in /usr/lib/x86_64-linux-gnu/libcuda.so.1
=========     Host Frame: [0xf2faf]
=========                in /opt/conda/envs/test/bin/gtests/libcudf/../../../lib/./libcufile.so.0