xml parsing problem?

[danieljeans]

I think I've run into a strange failure of xml parsing, in which

$k4geo_DIR/ILD/compact/ILD_l5_v02/ILD_l5_v02.xml
is interpreted as
LOCAL_DIRECTORY/$k4geo_DIR/ILD/compact/ILD_l5_v02/ILD_l5_v02.xml

see the line in the output pasted below starting "XercesC FATAL"

[this problem does not occur on a local installation of ilcsoft (ilcsoft/sw/gcc114_64bit/v02-03-03) on the same machine]

[jeans@cw07 test]$ echo $k4geo_DIR
/cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/k4geo/0.20-hapqru/share/k4geo

• OS version: RHEL9
• Compiler version: gcc11.3.1-opt
• Package version: source /cvmfs/sw.hsf.org/key4hep/setup.sh
• Reproduced by:
  on RHEL9 machine (tested on worker node at KEKcc):

source /cvmfs/sw.hsf.org/key4hep/setup.sh
Marlin testinitdd4hep.xml

• Input testinitdd4hep.xml:

<marlin>

  <execute>
    <processor name="InitDD4hep"/>
  </execute>

  <global>
  <parameter name="LCIOInputFiles"> 
anything.slcio
    </parameter>
  </global>

  <processor name="InitDD4hep" type="InitializeDD4hep">
    <parameter name="DD4hepXMLFile" type="string">
$k4geo_DIR/ILD/compact/ILD_l5_v02/ILD_l5_v02.xml
    </parameter>
  </processor>

</marlin>

• Output:

[jeans@cw07 test]$ Marlin testinitdd4hep.xml
<!-- Loading shared library : /gpfs/home/ilc/jeans/myProcessors/simplestProcessor/lib/libsimplest.so.0.0.0 (libsimplest.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/pandoraanalysis/2.0.1-tfotyc/lib/libPandoraAnalysis.so.2.0.1 (libPandoraAnalysis.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/marlinkinfitprocessors/0.5-fisrlr/lib/libMarlinKinfitProcessors.so.0.5.0 (libMarlinKinfitProcessors.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/ildperformance/1.12-7n6fo3/lib/libILDPerformance.so.1.12.0 (libILDPerformance.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/garlic/3.1-33x6y7/lib/libGarlic.so.3.1.0 (libGarlic.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/forwardtracking/1.14.2-b35wc5/lib/libForwardTracking.so.1.14.2 (libForwardTracking.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/clupatra/1.3.1-qmcotk/lib/libClupatra.so.1.3.1 (libClupatra.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-04-12/x86_64-almalinux9-gcc11.3.1-opt/clicperformance/02-05-3ejf3f/lib/libClicPerformance.so.2.5.0 (libClicPerformance.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/overlay/0.23.1-vnmkw7/lib/libOverlay.so.0.23.1 (libOverlay.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/marlintrkprocessors/2.12.5-zz4fiw/lib/libMarlinTrkProcessors.so.2.12.5 (libMarlinTrkProcessors.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/marlinreco/1.35-eidwtk/lib/libMarlinReco.so.1.35.0 (libMarlinReco.so)-->
PersistencyIO    INFO  +++ Set Streamer to dd4hep::OpaqueDataBlock
Info in <TGeoManager::TGeoManager>: Geometry default, Detector Geometry created
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/marlinkinfit/0.6.1-77urex/lib/libMarlinKinfit.so.0.6.1 (libMarlinKinfit.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/marlinfastjet/0.5.3-qvmzkz/lib/libMarlinFastJet.so.0.5.3 (libMarlinFastJet.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/lctuple/1.14-cayexu/lib/libLCTuple.so.1.14.0 (libLCTuple.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/lcfiplus/0.10.1-3e6tzs/lib/libLCFIPlus.so.0.10.1 (libLCFIPlus.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-04-12/x86_64-almalinux9-gcc11.3.1-opt/fcalclusterer/1.1-osfwyn/lib/libFCalClusterer.so (libFCalClusterer.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/marlindd4hep/0.6.2-67sn23/lib/libMarlinDD4hep.so.0.6.2 (libMarlinDD4hep.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/ddmarlinpandora/0.12.01-p742e5/lib/libDDMarlinPandora.so.0.12.1 (libDDMarlinPandora.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/conformaltracking/1.12-2qszcd/lib/libConformalTracking.so.1.12.0 (libConformalTracking.so)-->
<!-- Loading shared library : /cvmfs/sw.hsf.org/key4hep/releases/2024-03-10/x86_64-almalinux9-gcc11.3.1-opt/cedviewer/1.19.1-jxneb2/lib/libCEDViewer.so.1.19.1 (libCEDViewer.so)-->

XMLParser::parse : no <constants/> section found in testinitdd4hep.xml
[ MESSAGE "Marlin"]  ---- no GEAR XML file given  --------- 
[ MESSAGE "InitDD4hep"]  -------------------------------------
[ MESSAGE "InitDD4hep"]  ---- Initializing DD4hep from file  $k4geo_DIR/ILD/compact/ILD_l5_v02/ILD_l5_v02.xml ... 
XercesC          FATAL +++ FATAL Error at file "", Line 0 Column: 0 Message:unable to open primary document entity '/gpfs/home/ilc/jeans/myProcessors/simplestProcessor/test/$k4geo_DIR/ILD/compact/ILD_l5_v02/ILD_l5_v02.xml'
 ***********************************************
 A runtime error occured - (uncaught exception):
      dd4hep: Failed to parse the XML file $k4geo_DIR/ILD/compact/ILD_l5_v02/ILD_l5_v02.xml [Invalid XML ROOT handle]
dd4hep: while parsing $k4geo_DIR/ILD/compact/ILD_l5_v02/ILD_l5_v02.xml
 Marlin will have to be terminated, sorry.
 ***********************************************

any suggestions warmly welcomed!

• Goal: load detector geometry

[tmadlener]

I think I have run into this before, but didn't figure out what the issue was. In the end I resorted to effectively introducing a constant for that value and effectively bypassing the parser, by passing in the value directly, i.e.

<marlin>

  <execute>
    <processor name="InitDD4hep"/>
  </execute>

  <global>
  <parameter name="LCIOInputFiles"> 
anything.slcio
    </parameter>
  </global>

  <constants>
    <constant name="k4geo_DIR" value="/path/to/k4geo_DIR" />
  </constants>

  <processor name="InitDD4hep" type="InitializeDD4hep">
    <parameter name="DD4hepXMLFile" type="string">
${k4geo_DIR}/ILD/compact/ILD_l5_v02/ILD_l5_v02.xml
    </parameter>
  </processor>

</marlin>

and then:

Marlin testinitdd4hep.xml --contant.k4geo_DIR=$k4geo_DIR

This is not really a solution, but it was a workaround for me.

edit: fixed command to run Marlin (see comment below)

[danieljeans]

thanks, this workaround worked for me if I change to

Marlin testinitdd4hep.xml --constant.k4geo_DIR=$k4geo_DIR

(ie contants -> constant )

[dudarboh]

Marlin xml parser does not resolve environment variables.
After parsing it leaves it at string: "$k4geo_DIR"... (Or am I wrong?)

It has something to do with the DD4hep resolver during the geometry loading.
I have tried to trace back how the geometry path is parsed and ended up at DocumentHandler::load.
Funny that it has two definitions of the identical signature :)

This one, which works. It has _clean_fname at the beginning resolving the environment from the string...
https://github.com/AIDASoft/DD4hep/blob/df3700564d330a882b6a392576924f082ef3c94c/DDCore/src/XML/DocumentHandler.cpp#L484-L523

But a few lines above there is also this one. This one I assume fails...

https://github.com/AIDASoft/DD4hep/blob/df3700564d330a882b6a392576924f082ef3c94c/DDCore/src/XML/DocumentHandler.cpp#L330-L372

Which one is used depends on the building flags DD4HEP_USE_TINYXML (I guess??...)

EDIT:
I just found that it is set by DD4HEP_USE_XERCESC.
The doc also says it.

[dudarboh]

https://github.com/AIDASoft/DD4hep/pull/1326/files

This fixes it for me locally...