Skip to content

Commit

Permalink
Basic spring clean to get started again
Browse files Browse the repository at this point in the history
  • Loading branch information
drbenmorgan committed Mar 21, 2018
1 parent 5d18013 commit 524a386
Show file tree
Hide file tree
Showing 3 changed files with 86 additions and 14 deletions.
82 changes: 77 additions & 5 deletions RelocatableSoftware/HSFReloc/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,13 @@ Example program/library code to demonstrate package relocatability concepts.
The following software is required to build the base set of programs:

- [CMake](https://www.cmake.org) version 3.3 or newer
- C++11 compatible compiler (GNU, Clang, Intel, MSVC)
- C++11 compatible compiler (GNU, Clang, Intel)
- macOS or Linux OS (Windows not yet supported)

The following software is optional:

- [Qt5](https://www.qt.io) for the [qt](qt) subproject
- [Poco](http://pocoproject.org) for the [poco](poco) subproject
- [Qt5](https://www.qt.io) for the [qt](programs/hsfreloc-qt.cpp) example application
- [Poco](http://pocoproject.org) for the [poco](hsfreloc-poco.cpp) example application

To build the software, simply create a build directory and run `cmake` inside it,
pointing it to the top level source directory (i.e. the directory holding this
Expand All @@ -25,7 +26,7 @@ $ cmake --build .
```

If CMake has issues finding any dependencies, ensure their root install
prefixes are listed in `CMAKE_PREFIX_PATH` and passing this list to
prefixes are listed in `CMAKE_PREFIX_PATH` and pass this list to
`cmake` on the command line or setting it in your environment.

Build Outputs
Expand All @@ -38,8 +39,79 @@ If you configure the project for an IDE like Xcode,
the `BuildProducts` directory will contain an extra level of directories
to support each build type (`Release`, `Debug` etc)


Compiled Languages
==================
The [programs](programs) directory focuses on C/C++ implementations, but PRs are
welcome to demonstrate application/library self-location in other languages.

The `hsfreloc` program
----------------------

This basic C/C++ example demonstrates the use of the [`binreloc`](programs/binreloc)
library for simple application self-location. It also comes with a basic resource
file "resource.txt" to show how these can be located. Simply running the application
will print its location and the contents of the resource file:

``` console
$ ./BuildProducts/bin/hsfreloc
[application in]: /AbsPathToWhereYouRanCMake/./BuildProducts/bin
[resource]: 'hello from builtin hsfreloc resource file!
'
$
```

Relocatability can be tested by copying the `BuildProducts` directory to any other location
you like on the local machine. Rerun `hsfreloc` and it should print its new location and
the resource contents. You can prove that it's not using build time paths by removing
the original build directory.


The `hsfreloc-poco` program
---------------------------

Demonstrates the self-location interfaces supplied by the [Poco](http://pocoproject.org) libraries.


The `hsfreloc-qt` program
-------------------------

Demonstrates the self-location interfaces supplied by the [Qt5](https://www.qt.io) libraries.


Scripting Languages
===================
The directories under [scripting](scripting) contain (very) basic
examples of relocatable Python and Ruby programs.
examples of relocatable Python and Ruby programs. If your language du jour
is not in here, feel free to add a PR showing how it implements self-location.


Dealing with Packaging Issues
=============================

When implementing new or patching existing HEP packages, the above techniques
should ensure relocatablity. Packaging involves integrating many pieces of software,
many of which may not be fully relocatable. This section aims to provide demonstrations
of the typical cases and fixes where possible (and "this is broken" where not...).
Examples are:

1. A program/library links to others - use of `patchelf` and `install_name_tool`, RPATHS,
RUNPATHS, and linker tricks.
2. A program/library hard codes a path into binary. For example

``` c
/* bad.cc */
#define RESOURCEPATH /usr/share/badpackage
```
Likely to be the most awkward case, but conda/spack may have the tools to repoint these
paths.
3. A package has text based files that hard code paths, e.g.
``` sh
#!/some/build/time/path/to/python
```

Simple search-and-replace should work here.


4 changes: 2 additions & 2 deletions RelocatableSoftware/HSFReloc/programs/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ set(MANGLE_BINRELOC "hsf_reloc_binreloc${MANGLE_BINRELOC}")
configure_file(binreloc/hsf_binreloc.h.in binreloc/hsf_binreloc.h @ONLY)

# - Configure "front end" to binreloc, adding known *relative path*
# from exe dire to resource dir
# from exe dir to resource dir
file(RELATIVE_PATH HSF_BINDIR_TO_RESOURCEDIR
"${CMAKE_INSTALL_FULL_BINDIR}"
"${CMAKE_INSTALL_FULL_DATADIR}/${PROJECT_NAME}/resources"
Expand Down Expand Up @@ -43,7 +43,7 @@ else()
endif()

#-----------------------------------------------------------------------
# - Optional Qt5 deom for its builtin self-location
# - Optional Qt5 demo for its builtin self-location
# - Only need core lib
find_package(Qt5Core QUIET)

Expand Down
14 changes: 7 additions & 7 deletions RelocatableSoftware/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,7 @@ their own interpreters, whose paths may also end up hard coded into scripts:
print("hello world")
```

The resulting stack is then not relocatable as the interpreter path will not exist after relocation.
The resulting stack is then not relocatable as the interpreter path will not exist after relocation.

Rather than hard coding system or custom interpreter paths, script authors should prefer the use of the
[`env`](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/env.html) program as the shebang, e.g.
Expand All @@ -298,12 +298,12 @@ print("hello world")

Use of `env` makes the program relocatble, but defers location of the interpreter to the `PATH` environment variable,
and consequently the configuration management system for the software stack. Whilst package authors should prefer
usage of the `env` pattern, software stack managers can also consider rewriting the shebang line during install
usage of the `env` pattern, software stack managers can also consider rewriting the shebang line during install
and on any relocation to the absolute path of the required interpreter. As it is plain text, simple regular expression
replacement can be used, but the chosen packaging system must support this, and care must be taken
if the resultant stack is to be deployed over network file systems (and hence unknown mount points).

**TODO?** Binaries *also* have an interpreter (on Linux, `ld-linux.so`, On macOS, `dyld`). These are also hardcoded,
**TODO?** Binaries *also* have an interpreter (on Linux, `ld-linux.so`, On macOS, `dyld`). These are also hardcoded,
though can be changed with, e.g., `patchelf` for ELF binaries.


Expand Down Expand Up @@ -370,7 +370,7 @@ Development Tools
CMake
-----
To support use of a Project by a CMake based client project, scripts for
use with CMake's [`find_package`](http://www.cmake.org/cmake/help/v3.2/command/find_package.html) command in "config"
use with CMake's [`find_package`](http://www.cmake.org/cmake/help/v3.2/command/find_package.html) command in "config"
mode should be provided. A `FindPACKAGENAME.cmake` should *not* be implemented, including the use of CMake commands
like `find_path`, `find_library` as these are intended to locate packages not supplying any CMake support files. CMake
"ProjectConfig.cmake" files are installed alongside the project and can self-locate the project's headers/libraries/executables
Expand Down Expand Up @@ -467,7 +467,7 @@ additional arguments.
For libraries using environment variables, it may be possible to wrap
these with a small facade library. This would do nothing more that
self-locate, set the needed environment variables and expose the rest
of the library symbols. However, this has implications for usability
of the library symbols. However, this has implications for usability
and runtime manipulation of the environment by clients.

When absolute paths are hardcoded into binaries, then only intrusive
Expand Down Expand Up @@ -497,8 +497,8 @@ Relocation and Packaging
========================
**TODO** Topics involving relocatability when it comes to the packaging level. Define "packaging" as the things we need
to do/write to allow `spack|brew|whatever install mypackage` to work, whether building `mypackage` from source locally
or downloading and unpacking a binary. The "package manager" needs to include tooling to manage relocation, and this
may include things like changing RPATHs in binaries, to shebangs in interpreted programs. Other topics like deployment
or downloading and unpacking a binary. The "package manager" needs to include tooling to manage relocation, and this
may include things like changing RPATHs in binaries, to shebangs in interpreted programs. Other topics like deployment
to CVMFS etc.

Take simple example of two packages and a "typical" versioned tree plus "views"? e.g.
Expand Down

0 comments on commit 524a386

Please sign in to comment.