Run with older libc

[navjotk]

I am trying to run this on a cluster that has libc version 2.12 and on trying to run the mnist example I get the following error:

Exception in thread "main" java.lang.UnsatisfiedLinkError: /home/navjot.kukreja/CaffeOnSpark/caffe-distri/distribute/lib/liblmdbjni.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/navjot.kukreja/CaffeOnSpark/caffe-distri/distribute/lib/liblmdbjni.so)
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1937)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1855)
        at java.lang.Runtime.loadLibrary0(Runtime.java:870)
        at java.lang.System.loadLibrary(System.java:1122)
        at com.yahoo.ml.caffe.LmdbRDD$.com$yahoo$ml$caffe$LmdbRDD$$loadLibrary(LmdbRDD.scala:244)
        at com.yahoo.ml.caffe.LmdbRDD.com$yahoo$ml$caffe$LmdbRDD$$openDB(LmdbRDD.scala:199)
        at com.yahoo.ml.caffe.LmdbRDD.getPartitions(LmdbRDD.scala:46)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
        at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
        at com.yahoo.ml.caffe.CaffeOnSpark.train(CaffeOnSpark.scala:158)
        at com.yahoo.ml.caffe.CaffeOnSpark$.main(CaffeOnSpark.scala:40)
        at com.yahoo.ml.caffe.CaffeOnSpark.main(CaffeOnSpark.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Clearly, liblmdbjni.so was compiled with a higher version of libc. On checking the build logs, I realise that this library was never compiled as part of the build and was actually distributed as a binary.

My choices now are:

• Recompile liblmdbjni with the older version
• Stick to the current binary and make it see a newer version of libc

Option 2 above becomes harder since I don't have root access to the cluster so chroot isn't an option. The path to ld is hardcoded inside the binary and will always be searched inside /lib.

Any ideas here?

[junshi15]

You could try to compile lmdbjni. We use the library from https://github.com/deephacks/lmdbjni

It is specified in the following lines:
https://github.com/yahoo/CaffeOnSpark/blob/master/caffe-grid/pom.xml#L100-L114

After you build your own lmdbjni, you need to change the above pom.xml accordingly so that it does not fetch from maven repo.

You may need to change the top level make file concerning the location of liblmdbjni.so, as well.

[navjotk]

I don't have experience with maven. Could you please elaborate on what I need to do to use a custom liblmdbjni.so?
Apart from deleting the lines you pointed out above and also assuming that LD_LIBRARY_PATH contains liblmdbjni.so somewhere.

Edit: I now have a locally compiled copy of liblmdbjni.so. Attempting to rebuild CaffeOnSpark without any changes to the pom.xml did not change anything. Simply replacing the .so didn't either. It still seems to find the old version somewhere, see error below.

Native Library /home/navjot.kukreja/CaffeOnSpark/caffe-distri/distribute/lib/liblmdbjni.so already loaded in another classloader, /tmp/liblmdbjni-64-3691350142466429244.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /tmp/liblmdbjni-64-3691350142466429244.so)

How do I update the pom.xml to use a locally built version of this library?

[navjotk]

I managed to get past this issue by using the instructions here to install the locally built liblmdbjni to the maven local repo. This was version 0.4.7. When I tried to build CaffeOnSpark again, it downloaded version 0.4.6 of liblmdbjni again from the central repo. To address this, I made the pom look specifically for version 0.4.7 by editing the version lines. This fixed it and the correct version of liblmdbjni.so was picked up this time.

Please note that simply replacing the .so after the build does not do the same for some reason.

Edit: At some point I also tried <scope>system</scope> in the pom.xml but that didn't seem to work either.

[junshi15]

To add you local files to maven build, you can use maven system dependency or install them into your local maven repo. http://stackoverflow.com/questions/4955635/how-to-add-local-jar-files-in-maven-project
Glad you work around this.