Add extraLibDirs to runtime lib search paths of library

[alt-romes]

Runtime search paths are hard. Here's the current picture to understand why this patch exists:

• When linking a shared library, GHC will include in the rpath entries of the shared library all the paths listed in the library dirs section of the installed package info of all packages the shared library depends on. * On darwin, GHC has special logic to inject the library dirs listed in the installed dependent packages info into the rpath section instead of passing the dirs as -rpath flags to the linker. However, only the dirs where used libraries are found are actually injected. The others are ignored. This works around limitations of the darwin loader.

• Cabal, in addition, passes directly to the linker (via -optl-Wl,-rpath,...) the library dirs of packages the shared library for the package being built depends on. * In a vanilla cabal installation, this will typically only be the path to the cabal store and the path to the installed GHC's boot libraries store. * When using nix there will a different library dir per installed package. Since these lib dirs are passed directly to the linker as rpaths, we bypass the darwin loader logic and, for very big packages, on darwin, we could end up reaching the load command limit and fail linking. We don't address this situation in this MR.

When we specify extra-lib-dirs in Cabal, these extra-lib-dirs will be added to the library dirs listed in the installed package info of the library they were specified for. Furthermore, when building a shared library, extra-lib-dirs will be passed as -L flags to the linker invocation. However, the same extra-lib-dirs will not be passed as -rpath to the linker.

The end situation is as follows:

1. The shared library `libA` built for a package `A` will be linked
   against some libraries `libExtra` found in extra-lib-dirs
   `extraA`.

2. The RPATH section of `A` will NOT contain `extraA`, because we
   don't pass -rpath extra-lib-dirs when linking the library, but it
   will depend on `libExtra`.

3. The installed package info of that package `A` will contain, in
   the library dirs section, the extra-lib-dirs `extraA` and the
   path to `libA`.

4. When a package `B` depends on package `A`, it will include in the
   RPATH section of the shared library `libB` the lib dirs from the
   installed package info of `A`, i.e. `/path/to/libA` and `extraA`,
   and depends on `libA` and, transitively, on `libExtra`.

The conclusion is:

5. When we load `libB`, we will load `libA`, which is found in
   `/path/to/libA`, and, transitively, load `libExtra` which is
   found in `extraA` -- they are both found because both
   `/path/to/libA` and `extraA` are listed in the RPATH entries.

6. However, if we load `libA` directly we will /NOT/ find
   `libExtra`, because `extraA` is not included in the RPATH
   entries.

So, ultimately, what this commit fixes, is the failure described in (6), caused by the incorrect behaviour of (2), by specifying -rpath extra-lib-dirs when linking the shared library of a package, to include the extra lib dirs in the RPATH entries of that shared library (even though dependents of this library would already get the extra-lib-dirs in their RPATH, the library itself didn't, resulting in cabal#7339 and ghc#19350)

Fixes #7339
Fixes ghc#19350

• Patches conform to the coding conventions.
• Any changes that could be relevant to users have been recorded in the changelog.
• The documentation has been updated, if necessary.
• Manual QA notes have been included.
• Tests have been added. (Ask for help if you don’t know how to write them! Ask for an exemption if tests are too complex for too little coverage!)

[mouse07410]

Yes!!

[alt-romes]

Fixes #7339

[alt-romes]

Should this change also be backported @Kleidukos ? Would require re-applying the change to gbuild and buildOrRepl before these two were split into different modules.

[Kleidukos]

@alt-romes ok, can I let you drive the backport? it's @mergifyIO backport 3.10 in a comment

[alt-romes]

@alt-romes ok, can I let you drive the backport? it's @mergifyIO backport 3.10 in a comment

I'd need some instructions to do so, I haven't worked with mergifyIO. Happy to do so in any case.

[Mikolaj]

@alt-romes: You create a github comment, here, saying "@mergify backport 3.10". That's it. You can do it right now.

[alt-romes]

@mergify backport 3.10

[mergify]

backport 3.10

✅ Backports have been created

• #9639 Add extraLibDirs to runtime lib search paths of library (backport #9554) has been created for branch 3.10 but encountered conflicts

[alt-romes]

I'd be great if you could review this @andreabedini @gbaz

[andreabedini]

I defer to my linking expert @angerman 😂

Edit:

Reading #7339 it seems that @angerman and @bgamari where agreeing that the rpath logic should be left to GHC? From Ben's comment.

Cabal currently uses GHC's -optl-Wl,-rpath and -L flags to precisely specify the compile-time and runtime library search paths needed for a particular install plan. From the perspective of Darwin (the issue fixed in #7094) this status quo is unfortunate since we do not benefit from the clever rpath handling introduced in ghc@4ff93292243888545da452ea4d4c1987f2343591.

However, this is not a fundamental issue: it can be easily avoided by introducing a -rpath flag in GHC's command-line interface, providing a reliable and precise way for the user to influence the runtime library search path. This would allow us to fix the issue pointed out in this ticket (that extra-lib-dirs don't make it into RPATH) while giving us a more robust way to address the Darwin issue addressed in #7094.

[alt-romes]

@andreabedini Yes and no. The rpath logic is not left to GHC, and even if an -rpath flag were to be added, it wouldn't be clear whether extra paths added through it should be handled via the macOS length limit workaround.

In fact, I implemented the suggested -rpath flag, but it wasn't finished because it is unclear what semantics it should have.

I've also posted a comment to @angerman's original MR which attempted something slightly orthogonal to this (#7094 (comment)), to explain how it relates to this one.
I think we indeed all agree (me, @bgamari, @angerman), and I have tried to clarify the confusion regarding this in the commit message.

Replying to what you quoted:

Cabal currently uses GHC's -optl-Wl,-rpath and -L flags to precisely specify the compile-time and runtime library search paths needed for a particular install plan. From the perspective of Darwin (the issue fixed in #7094) this status quo is unfortunate since we do not benefit from the clever rpath handling introduced in ghc@4ff93292243888545da452ea4d4c1987f2343591.

Indeed, Cabal already specifies runtime search paths and compile-time library paths to GHC, with information only cabal is aware of. This path extends the runtime search paths Cabal passes to GHC to include the extra-lib-dirs field which specifies paths where libraries will be found at runtime (the field itself unfortunately conflates compile time and load time paths).

However, this is not a fundamental issue: it can be easily avoided by introducing a -rpath flag in GHC's command-line interface, providing a reliable and precise way for the user to influence the runtime library search path. This would allow us to fix the issue pointed out in this ticket (that extra-lib-dirs don't make it into RPATH) while giving us a more robust way to address the Darwin issue addressed in #7094.

It turns out that the Darwin workaround will only include in the RPATH entries of a dynlib the paths where libraries are found at runtime, in order to not include RPATHs to libraries which are never used by the library. However, sometimes, we really do want to specify the path to a directory where a library can be found at runtime, even if no library is found there at compile time. extra-lib-dirs should make it into RPATH, because they are extra directories where the user manually specifies they want to search. Arguably, there could be an extra-runtime-lib-dirs field to be more precise about the diference between runtime and compile time search paths, but, at present time, extra-lib-dirs serves that purpose.

So, even if there were an -rpath flag, it is unclear if such a flag should go through Darwin's special rpath logic, or bypass it completely (again see !11801).

[andreabedini]

@alt-romes Thank you for your very detailed reply!