Running hybrid HPX/OMP code

[ddemidov]

a85879b does help in some way. Namely, the following code works:

#include <iostream>
#include <hpx/hpx_main.hpp>
#include <hpx/include/parallel_for_each.hpp>

//---------------------------------------------------------------------------
int main() {
    std::vector<int> x(10);

#pragma omp parallel for
    for(int i = 0; i < 10; ++i)
        x[i] = i;

    hpx::parallel::for_each(
            hpx::parallel::par,
            x.begin(), x.end(),
            [](int v) {
                std::cout << v << "\n";
            });
}

The output is:

$ LD_PRELOAD=/home/demidov/work/stellar/hpxMP/libiomp5.so ./hello1 

0
1
2
3
4
5
6
7
8
9

The following however does not work (the difference is in HPX initialization method):

#include <iostream>
#include <hpx/hpx_init.hpp>
#include <hpx/include/parallel_for_each.hpp>

//---------------------------------------------------------------------------
int hpx_main(boost::program_options::variables_map &vm) {
    std::vector<int> x(10);

#pragma omp parallel for
    for(int i = 0; i < 10; ++i)
        x[i] = i;

    hpx::parallel::for_each(
            hpx::parallel::par,
            x.begin(), x.end(),
            [](int v) {
                std::cout << v << "\n";
            });

    return hpx::finalize();
}
//---------------------------------------------------------------------------
int main(int argc, char *argv[]) {
    boost::program_options::options_description
       options("Usage: " HPX_APPLICATION_STRING " [options]");

    return hpx::init(options, argc, argv);
}

The output is:

$ LD_PRELOAD=/home/demidov/work/stellar/hpxMP/libiomp5.so ./hello2
HPX OpenMP runtime has started
{stack-trace}: 13 frames:
0x7fe33427b6e6  : hpx::termination_handler(int) + 0x86 in /home/demidov/work/hpx/build/lib/libhpx.so.0
0x7fe332a01210  : ??? + 0x7fe332a01210 in /usr/lib/libpthread.so.0
0x7fe33438870d  : ??? + 0x7fe33438870d in /home/demidov/work/hpx/build/lib/libhpx.so.0
0x7fe334372a18  : hpx::threads::threadmanager_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>, hpx::threads::policies::callback_notifier>::run(unsigned long) + 0x558 in /home/demidov/work/hpx/build/lib/libhpx.so.0
0x7fe3342c7ca1  : hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>, hpx::threads::policies::callback_notifier>::start(hpx::util::function<int (), void, void> const&, bool) + 0x391 in /home/demidov/work/hpx/build/lib/libhpx.so.0
0x7fe3342b5f4c  : hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<boost::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_lifo>, hpx::threads::policies::callback_notifier>::run(hpx::util::function<int (), void, void> const&) + 0xc in /home/demidov/work/hpx/build/lib/libhpx.so.0
0x7fe334246daf  : ??? + 0x7fe334246daf in /home/demidov/work/hpx/build/lib/libhpx.so.0
0x7fe33424957d  : ??? + 0x7fe33424957d in /home/demidov/work/hpx/build/lib/libhpx.so.0
0x7fe33424ac1d  : hpx::detail::run_or_start(hpx::util::function<int (boost::program_options::variables_map&), void, void> const&, boost::program_options::options_description const&, int, char**, std::vector<std::string, std::allocator<std::string> > const&, hpx::util::function<void (), void, void> const&, hpx::util::function<void (), void, void> const&, hpx::runtime_mode, bool) + 0x28d in /home/demidov/work/hpx/build/lib/libhpx.so.0
0x40cb33        : ??? + 0x40cb33 in ./hello2
0x7fe330a98040  : __libc_start_main + 0xf0 in /usr/lib/libc.so.6
0x40c8a9        : ??? + 0x40c8a9 in ./hello2
{what}: Segmentation fault
{config}:
  HPX_HAVE_NATIVE_TLS=ON
  HPX_HAVE_STACKTRACES=ON
  HPX_HAVE_COMPRESSION_BZIP2=OFF
  HPX_HAVE_COMPRESSION_SNAPPY=OFF
  HPX_HAVE_COMPRESSION_ZLIB=OFF
  HPX_HAVE_PARCEL_COALESCING=ON
  HPX_PARCELPORT_IPC=OFF
  HPX_PARCELPORT_IBVERBS=OFF
  HPX_HAVE_VERIFY_LOCKS=OFF
  HPX_HAVE_HWLOC=ON
  HPX_HAVE_ITTNOTIFY=OFF
  HPX_RUN_MAIN_EVERYWHERE=OFF
  HPX_LIMIT=5
  HPX_ACTION_ARGUMENT_LIMIT=5
  HPX_COMPONENT_CREATE_ARGUMENT_LIMIT=5
  HPX_FUNCTION_ARGUMENT_LIMIT=8
  HPX_LOCK_LIMIT=5
  HPX_TUPLE_LIMIT=8
  HPX_WAIT_ARGUMENT_LIMIT=5
  HPX_PARCEL_MAX_CONNECTIONS=512
  HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
  HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
  HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
  HPX_PREFIX (configured)=/home/demidov/work/hpx/build
  HPX_PREFIX=/home/demidov/work/hpx/build
{version}: V0.9.10-trunk (AGAS: V3.0), Git: c182b3f6007610102b0b66c81ee73579862fee89
{boost}: V1.57.0
{build-type}: release
{date}: Dec 23 2014 21:33:46
{platform}: linux
{compiler}: GNU C++ version 4.9.2
{stdlib}: GNU libstdc++ version 20141030
Aborted (core dumped)

One thing that is somewhat suspicious is that the line

HPX OpenMP runtime has started

is present in the second output but is missing from the first.

[ddemidov]

Pinging @hkaiser as well.

[ddemidov]

When I switch to hpx initialization with <hpx/hpx_main.hpp> in my real application and use the hpxMP runtime, I get the following error:

$ LD_PRELOAD=/home/demidov/work/stellar/hpxMP/libiomp5.so ./hpx 

./hpx: relocation error: ./hpx: symbol __kmpc_for_static_init_8, version VERSION not defined in file libiomp5.so with link time reference

It happens here: https://github.com/ddemidov/amgcl/blob/master/amgcl/backend/builtin.hpp#L301-L306.

I've tried to reproduce the error with

#include <iostream>
#include <hpx/hpx_main.hpp>
#include <hpx/include/parallel_for_each.hpp>

//---------------------------------------------------------------------------
int main() {
    std::vector<int> x(10);

    int offset = 1;

#pragma omp parallel for
    for(int i = 0; i < 10; ++i)
        x[i] = offset + i;

    hpx::parallel::for_each(
            hpx::parallel::par,
            x.begin(), x.end(),
            [](int v) {
                std::cout << v << "\n";
            });
}

It fails as well, but the error is different:

$ LD_PRELOAD=/home/demidov/work/stellar/hpxMP/libiomp5.so ./hello1
hello1: intel_hpxMP.cpp:70: void __kmpc_fork_call(ident_t*, kmp_int32, kmpc_micro, ...): Assertion `argc < 3' failed.
Aborted (core dumped)

[kempj]

The "__kmpc_for_static_init_8" is simply one of the loop functions I have yet to implement. There is a function call for each different type of the iterator.

The second, I am less familiar with, as icpc uses the runtime differently than clang. The loops are next on my todo list, so it should be done this week. Further compatibility with icc is a bit further down on the list, but I'll take a look at it once I finish the loops.