AppImages that bundle *everything*

[aferrero2707]

As suggested in issue #224 I am trying to create a GIMP recipe that bundles all libraries, without blacklisting.
The standard recipe is presently not working on some distributions (Kubuntu 16.04 LiveCD for example), and I hope that the full bundling will solve such incompatibilities.

I have followed the example for sigil (https://github.com/probonopd/AppImages/blob/master/recipes/sigil/Recipe), but somehow I still cannot get the appimage to run correctly. In particular, the gimp executable crashes with a

free(): invalid next size (normal)

error that I cannot really interpret.

The full-bundling recipe can be found here: https://gist.github.com/aferrero2707/aea9aa96648078b306d6034daaf0372d

Does anyone have an idea how to fix that?

Thanks in advance!

[probonopd]

Don't bundle everything, it is neither recommended not supported. I also have issues with it.

[darealshinji]

If you really need newer versions of libc6, libstdc++6 or similar base libraries, running an AppImage inside a chroot might be an option.

[probonopd]

Doesn't a chroot need root?

[darealshinji]

schroot doesn't. But you need to be root to set it up. So... yeah.
Isn't it possible to use fuse to setup a chroot without root rights?

[probonopd]

...or use something like fakechroot

[probonopd]

or proot as is used by https://github.com/fsquillace/junest

[probonopd]

Also check https://github.com/lvml/makeaoi, a tool that bundles "everything"

[probonopd]

Another option may be to edit everything with patchelf so that absolutely no libraries get loaded from the system anymore (check with LD_DEBUG=libs!), and then in AppRun invoke the application like so:

"$APPDIR"/lib/x86_64-linux-gnu/ld-2.23.so "$APPDIR"/usr/bin/appstreamcli "$@"

Seems to have worked for me where all other options have failed me.

[probonopd]

Another option that automates this:

https://github.com/Intoli/exodus#how-it-works

[probonopd]

https://intoli.com/blog/exodus-2/

[TheAssassin]

Such a feature has been requested as a plugin for linuxdeploy, see linuxdeploy/linuxdeploy#5.

[darealshinji]

Building a snap package may be an option.

[TheAssassin]

@darealshinji we want to be able to provide users with a solution to build AppImages this way, too, though.

[azubieta]

This should be addressed at linux deploy. Closing this issue as duplicated.
See linuxdeploy/linuxdeploy#5

[probonopd]

Some application authors think it is not necessary to be able to build on the oldest still-supported LTS release. E.g., maoschanz/drawing#82 (comment). For those applications, bundling everything might be a last resort.

[probonopd]

Another candidate for bundling everything: VioletGiraffe/file-commander#246 (comment)

[probonopd]

Reopening this until we have a generic (fallback) solution for those cases, too. AppImage can totally do it, but we (so far) generally haven't recommended it.

[azubieta]

Just for the record the https://github.com/AppImageCrafters/AppImageCraft/ is an experiment to allow building this kind of AppImages

[probonopd]

Another candidate for bundling everything: probonopd/linuxdeployqt#405 (comment)

[probonopd]

Yet another candidate for bundling everything: https://github.com/tim-janik/beast. According to timj on IRC,

we use C++17 and Electronjs, so packaging on dists before 18.04 is too much effort, especially given our current priorities
April 2024, isn't soon by any measure though ;-)

[probonopd]

We need a better name for "AppImages that bundle everything".

How about "Standalone AppImages"?

[TheAssassin]

"Fully self-contained" makes more sense, technically and semantically.

[probonopd]

Self-contained yes, "fully" not so much. We are still not bundling kernel, X.org/Wayland, NVIDIA driver libraries and such, after all.

[TheAssassin]

Let's keep the kernel out of this for now.

If x.org, graphics drivers etc. aren't bundled you can hardly say they bundle everything.

The only AppImages that'd match your CLI AppImages that don't have graphical deps. That's the ones I'm thinking of. So let's focus on those.

Such AppImages are also standalone in the way that they don't require any external runtime, not even FUSE strictly speaking. As they can be described as standalone and have been described so in the past, this adjective shouldn't be used to differentiate between AppImages which bundle libc, the linker etc.

"Fully self-contained" refers to the fact these AppImages don't have any dependencies other than the kernel any more. That's the big difference.

Think of whirlpool and <insert famous whirlpool brand name here>. Not every whirlpool is a ***, but every *** is a whirlpool. Every AppImage is standalone, but not every standalone AppImage is fully self-contained.

[probonopd]

Would it make sense to define "self-contained AppDir" as an AppDir with the following behavior:

LD_LIBRARY_PATH='' find ./AppDir -type f -exec ldd {} 2>&1 \; | grep '=>' | grep -v AppDir
(returns nothing)

In other words, an AppDir that does not depend on shared libraries outside of the AppDir.

[probonopd]

Some time ago I had asked on Twitter how much overhead would be acceptable for an AppImage that does not depend on shared libraries outside of the AppImage. This pretty much summarizes it:

More responses can be seen in the thread.

[probonopd]

Looking at Mumble, a Qt-based application, it seems like we can produce such AppImages with very little, if any, overhead (especially if we base them on a musl libc based, size-optimized system such as Alpine Linux):

mumble-voip/mumble#3959 (comment)

[azubieta]

Apline Linux support is planned to be added on appimage-builder 0.5. If you have any idea or suggestion please feel free to add them in the following issue: AppImageCrafters/appimage-builder#10

[probonopd]

Cool @azubieta maybe you want to re-use something from my (currently still experimental) go-based implementation; see the example script at mumble-voip/mumble#3959 (comment).

[TheAssassin]

@probonopd your definition still doesn't work. Please call it a "fully self-contained AppDir", if any. Lingo is extremely important. I've explained more than once why the property "self-contained" applies to the existing AppDir already. I don't know why I have to keep repeating myself on that.

Regarding the overhead, I don't think anyone wants to ship 30 MiB of runtime for an application of 2 MiB. That poll was way too imprecise, as it doesn't put the AppImage size in relation with the size of the overhead.

There is also no need to enforce this kind of behavior. I think our established way has been working absolutely amazing, given its simplicity.

Also your approach still ignores the heterogenity of hardware. As soon as you depend on some special hardware, e.g., graphics accelerators, being "fully self-contained" is nearly impossible unless you make an AppImage for every possible situation. That usually means one per graphics card driver for a game, for instance. You will never get away with full isolation between AppImage resources and host system, there'll always be situations in which there is a need to share some resources. And I don't think that's an issue at all.

P.S.: I think it's absolutely possible and also necessary to keep < 1 MiB for the runtime. That should be our goal. Even if we had to bundle a (subset of a) libc implementation, e.g., musl, as well as some FUSE code.
If glibc, ld-linux & Co. became hard dependencies that have to be bundled as the contents require that, they become part of the runtime IMO. And in that case, the size of the runtime is going to go through the roof immediately.

[TheAssassin]

Also, the following questions haven't been covered in this issue yet:

How to implement security maintenance? Reusing some critical bits like libc or libssl from the system means they're usually up to date (and if not, that's really the problem of the system owner).

If you rely on updating the AppImages instead of reusing system libs, how can you ensure long term maintenance? Check appimage.github.io, there's tons of AppImages that haven't been updated in years. Anything that contains a browser (which is like, half of them, as they're using that damn Electron) is a security hazard.

Please don't yell "sandboxing!" now. Founding security on sandboxing is a horrible idea. Not to mention we don't have any serious sandboxing available for AppImages!

[azubieta]

I think that people will be totally fine with an extra 30 (or even 50 MB) of AppImage size if that's the only way the app will work. Consider the cases where the user wants to keep running very old and well-known software or his favorite retro-game.

Developers, on the other hand, will love the idea of being able to build their packages in the latest and really up-to-date distribution where they can find all the app resources instead of dealing with several hairy backports and for a nearly to dead distribution (yes I'm talking about Centos 6).

AppImage have been always a gamble between portability and size. You want to produce a more portable binary then you have to embed more stuff. Why it is not enough to bundle just the middleware? We all know the answer to that, GNU/Linux distributions are a mess (a reference to the @probonopd platform issues goes here).

It's not about enforcing this kind of AppImages is about to give an alternative. Developers will have the final word on whether they want their binaries to be more or less portable. We must give options.

You will never get away with full isolation between AppImage resources and the host system, there'll always be situations in which there is a need to share some resources. And I don't think that's an issue at all

Agreed, yet the AppImages produced using this approach haven't had any significant issue related to this topic. It will be always a tradeoff.

On the security topic. Security starts on the code if the application is not maintained you cannot expect it to be up to date with the latest security standards. The 1.1 version of libssl provides better encryption mechanisms that version 1.0. The software must change to use such an improved mechanism. So if a given piece of software is being maintained properly spinning a new bundle with up to date binaries is not a big deal, they can even set a programmed task on their build service. And the delta updates mechanism used by AppImageUpdate makes the process simple for the final user. Also being able to build in the very latest distributions makes it easier getting such updates.

Those were my 50c on the topic

[TheAssassin]

@azubieta you seem to agree to the point that there are cases in which this might be useful, but it shouldn't be required nor enforced. It's IMO still an edge case, and we shouldn't replace the current "build on old systems" approach.

On the security topic. Security starts on the code if the application is not maintained you cannot expect it to be up to date with the latest security standards. [...]

I've been following the same approach distributions have been using for decades. Old software can still be around, but its dependencies are at least kept up to date. For crypto libraries for instance this works pretty well, as devs are instructed to use them instead of shipping their own. If the calls are in the right order, the stability of the crypto libs' APIs usually guarantees that it'll be safe at the time of development as well as the future. (If it ain't securely developed in the first place it will never be safe in the future, sure, but that's not the point).

IMO bundling everything should be the ever last resort after everything else has failed. One should IMO reference at least some LTS system (e.g., the latest(!) Ubuntu LTS) for a project's dependencies, not the cutting edge, but ideally you target the oldest still-supported LTS you can find (CentOS usually).
Regarding keeping things up to date, any kind of "bundles everything" AppImage should be built on OBS, as that will make sure it'll receive updates in the future.

[azubieta]

we shouldn't replace the current "build on old systems" approach

I found it easier a "build on newer systems" approach, but it's not up to us to decide that. The decision should be made by the developers. We must give them accurate information about the different existent approaches, their strengths, and weakness. And of course, we should provide the tools.

[probonopd]

From a cursory glance it seems like some of the patches in https://github.com/flatpak/freedesktop-sdk-images modify upstream software so that it works in non-standard locations.

ALSA, GL, GLX, fontconfig, libva-vdpau, mesa, mesa-dri, Nvidia stuff, SDL,...

Who understands what is going on there? Is this helpful for AppImages that bundle everything?

[azubieta]

Interesting, but that forces us to build the whole dependencies stack to create an AppImage. Or use the binaries they already have.

[probonopd]

Thanks @azubieta for pointing out this link in IRC:
https://blogs.igalia.com/itoral/2014/07/29/a-brief-introduction-to-the-linux-graphics-stack/

[probonopd]

Seems like a real downside of AppImages that bundle everything is that we cannot get Nvidia GPU acceleration to work in them, unless we introduce some concept of runtimes/dependencies (which we don't want).

At least I get this impression when I look at what Flatpak is doing:

On a system that has this kernel driver version (340.107):

me@host:~$ dmesg | grep 340.107
[   19.879987] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  340.107  Thu May 24 21:54:01 PDT 2018

Flatpak installed this runtime automatically (340.107):

me@host:~$ ls .local/share/flatpak/runtime/org.freedesktop.Platform.GL.nvidia-340-107/x86_64/1.4/b0e864d2e4465f9a554d45a62d353b2f768815a3cebeee1ed9354ce3b6a8b47d/files/extra/
libcuda.so               libnvcuvid.so.1                libnvidia-ml.so.340.107
libcuda.so.1             libnvcuvid.so.340.107          libnvidia-opencl.so
libcuda.so.340.107       libnvidia-cfg.so.340.107       libnvidia-opencl.so.340.107
libEGL.so.1              libnvidia-compiler.so.340.107  libnvidia-tls.so.340.107
libEGL.so.340.107        libnvidia-eglcore.so.340.107   libnvidia-wfb.so.340.107
libGLESv1_CM.so.340.107  libnvidia-encode.so.1          libvdpau_nvidia.so
libGLESv2.so.1           libnvidia-encode.so.340.107    libvdpau_nvidia.so.340.107
libGLESv2.so.340.107     libnvidia-fbc.so.340.107       libvdpau.so.340.107
libGL.so.1               libnvidia-glcore.so.340.107    libvdpau_trace.so.340.107
libGL.so.340.107         libnvidia-glsi.so.340.107      OpenCL
libglx.so.1              libnvidia-ifr.so.340.107       tls
libglx.so.340.107        libnvidia-ml.so.1

So it seems like it matches the runtime to the version of the installed kernel module. Which means that if we wanted to ship Nvidia drivers, we would have to bundle all possible versions of Nvidia drivers. Which is clearly impossible for future versions of the driver...

So it seems that Nvidia acceleration cannot be supported without some concept of dependencies/runtimes.

Thanks @azubieta for helping me learn this.

barthalion on IRC confirmed that this is not the case for Intel and AMD.

[Kirtai]

Doesn't Flatpak require this due to the sandbox, rather than the bundling?

[probonopd]

According to @azubieta, in order to get Nvidia GPU acceleration to work it is required to use some libraries from the target operating system, especially the Nvidia libraries. This means that if we want to use Nvidia GPU acceleration, then we cannot have a 100% standalone AppImage since whenever the glibc, libz, and libstdc++ on the system are newer than what comes with the AppImage, then the Nvidia libraries on the system may require those newer versions rather than what comes with the AppImage. @azubieta hence uses a logic to determine whether the version of those libraries (and auxiliary files that go along with them) is newer in the AppImage or on the system, and uses that in https://github.com/AppImageCrafters/AppRun.

While this solution is not as elegant as I'd like it to be, it's probably the best that we can technically do at the moment, given how all of this works... Flatpak seems to be the Nvidia stuff in a Flatpak runtime as well (rather than using what comes with the system). I would assume they have many different variations/versions of org.freedesktop.Platform.GL.nvidia which is infeasible for the AppImage "one app = one file" concept, but I haven't checked it in detail.

[probonopd]

One thing that still puzzles me is why this appears to be an issue for Nvidia but not for Intel nor AMD graphics.

Or is it?

[azubieta]

Intal and AMD drivers are compatible between different versions, but still you may not have access to the latest driver/device features if you're using an old client library.