Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

make3: Toward a friendlier build system #12

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

make3: Toward a friendlier build system #12

wants to merge 6 commits into from

Conversation

xelxebar
Copy link

Overview

This PR is part of a larger goal to make J easier to package and share across OSes and Linux distributions.

Currently, the make and make2 build systems provide a particularly non-standard build process. They work well under their intended environments but provide significant munging to work as part of other build systems. This is due to the builds being largely managed by an ad-hoc collection of scripts rather than relying on standard tooling.

My hope is that, at the very least, this initiates discussion toward making J buildable and distributable in a more robust and friendly way.

Project Goal

Anecdotally, the current build system(s) provide significant challenges to users simply wishing to try out J on their system. The overall goal is to provide a build that uses the familiar and build-system-friendly make and, if needed, autotools:

$ ./configure
$ make
$ make install

The collection of commits in this PR shares my initial progress in this regard.

Specific Pain Points

In particular, the following is a collection of issues I have had to work around in order to package J for the Void Linux distribution:

  1. Sparse documentation

The build process is specific to J and highly non-standard. However, the documentation introducing said process is limited to overview.txt and make{,2}/make.txt which are both quite sparse. The existing documentation works well under a limited set of build conditions, but there is little information, either within the source tree or online, regarding how to make the build fit your requirements. This is mostly a case of "the code is the documentation."

  1. Fragile logic for setting CFLAGS,

Currently, CFLAGS is set with -Werror and a collection of -W flags that depends on whether the compiler is thought to be gcc or not. The gcc-detection scheme relies on ad hoc path parsing and symlink resolution which still breaks when using tools such as ccache. It is possible to short-circuit the detection to your preferred compiler; however, doing so is very non-trivial for someone not familiar with the build scripts. Furthermore, assuming the compiler to be only gcc or clang imposes an unexpected and artificial restriction that is decoupled from where such requirements may actually originate.

  1. Fails to respect CFLAGS, LDFLAGS etc. of environment,

When compiling as part of a larger build chain, it is standard to have CFLAGS populated with options you wish all your compiles to pick up. However, the build scripts in make and make2 clobber these environment variables. This is unexpected bevahivour and requires careful modification of the scripts to rectify.

  1. Non-friendly "install",

By default, the build system simply builds. There is no equivalent of a make install. Implicitly, the assumption is that the user will either run jconsole directly from the build output directory or copy these to some location, preserving the directory structure. This violates the expectations of the Filesystem Hierarchy Standard (FHS) for installed software on Unices and Linux distributions.

It is possible to manually pull apart the pieces and install J in a way that follows the FHS; however, this requires editing profile.ijs and providing (undocumented) flags to jconsole. This seems like an unreasonably high barrier to entry for someone simply wishing to offer J as a package for their favorite Linux distribution.

  1. Intractible use of shell scripting, and

The build scripts, while functional, are overall in need of a lot of TLC. There are deeply-nested if statements, giant case blocks with many repeated elements, and a lot of custom path-munging via cd and pwd. These techniques make the inherent logic within the scripts quite difficult to follow and fail to leverage standard shell facilities that, ostensibly, would perform the same job more concisely and robustly.

  1. Undocumented build options.

As far as I can tell, apart from the standard build environment variables the following allow the user to toggle various J-specific build options:

  • USE_OPENMP,
  • USE_LINENOISE,
  • USE_THREAD,
  • VERBOSELOG, and
  • jolecom (?).

However, the only place these are documented is within thes scripts themselves. Are these inteneded as undocumented/experimental options? If they are inteded to be user-facing, it would make sense to keep track of this kind of flag within make.txt at the very least.

What this PR does

I hope the above makes a case that the build system can be improved. I am willing to lead the charge in that endeavor if upstream is on board.

At the moment, the collection of build scripts here do little to change the above pain points, instead opting for a gradual and incremental evolution of make2 into the inteded target. The HEAD of this PR contains a new make3 directory, which is an exact copy of make2 at the start of this branch (commit cb4d852).

Within make3 I have only refactored build_jconsole.sh and build_libj.sh into code that I hope is "better". I personally feel it's more grokkable, and it reduces the SLOC count to about half of the originals.

I made an effort to ensure the changes are completely transparent by confirming identical hashes of (a tar of) the build products. That said, I have only confirmed this for linux and raspberry (under all j64x targets). My plan is to do the same for android, windows and darwin if this idea gains traction.

Notes

The make3 build should Just Work™ in exactly the same way as make2; however, as per the comments regarding scripting style, I opted to make one change to how build options are passed. Previously, each option was only enabled if the associated variable was set to "1". The scripts now enable each option if they are set to anything at all (other than the empty string). USE_LINENOISE is the only exception, since it defaults to enabled. You can disable as previously it by setting said variable to anything other than "1".

Any other "API" differences are unintended.

Prologue

If you made it this far, thank you for reading through this long PR explanation! If any of the above comes across as overly brash, please accept my apologies. My head has been deep in the code and associated frustrations for a while.

I look forward to hearing your feedback.

@xelxebar xelxebar changed the title Devel/make3 make3: Toward a friendlier build system Feb 15, 2020
@lemenkov
Copy link

It seems that meson build-system is getting momentum. It is faster, much easy to read than autotools/cmake, allows cross-platform builds, offers configurability with build options, etc.

Maybe better to switch to it instead?

@xelxebar
Copy link
Author

xelxebar commented Feb 18, 2020 via email

@zhihaoy
Copy link

zhihaoy commented Apr 29, 2020 •
edited
Loading

I think CMake is the only reasonable option. As a cross-platform build system generator, it has builtin support in Visual Studio, which proves its acceptance.

See also https://www.rojtberg.net/1481/do-not-use-meson/

@JRaspass JRaspass mentioned this pull request Jul 18, 2020
Closed
@moon-chilled
Copy link
Contributor

CMake is an abomination, and should be avoided at all costs.

Makefiles and scripts aren't inherently bad; they just need to be made friendlier.

@zhihaoy
Copy link

zhihaoy commented Aug 21, 2020

Makefiles and scripts are an abomination, and should be avoided at all costs.

CMake isn't inherently bad; it just needs to be made friendlier.

@eiverson
Copy link
Contributor

eiverson commented Jul 29, 2021 via email

@xelxebar
Copy link
Author

xelxebar commented Aug 2, 2021 via email

@xelxebar xelxebar force-pushed the devel/make3 branch 2 times, most recently from 813e45e to c3204a6 Compare January 12, 2022 11:03
@xelxebar xelxebar force-pushed the devel/make3 branch 2 times, most recently from 31a893e to c42aa42 Compare April 24, 2022 08:00
@moon-chilled
Copy link
Contributor

moon-chilled commented Apr 24, 2022 •
edited
Loading

@xelxebar you may find my build scripts (modifications to make2) helpful in constructing make3. I broke arm, and never got around to fixing it; but the core logic is largely factored, and hence much easier to change and understand.

@xelxebar
Copy link
Author

@moon-chilled Cool. It looks like some of your factoring went in a similar direction as my original (now clobbered) commits. Thanks for sharing. I might have a closer look later.

@cdburke cdburke force-pushed the master branch 10 times, most recently from c6377af to 1528769 Compare June 3, 2022 22:42
@cdburke cdburke force-pushed the master branch 2 times, most recently from 5b6e7be to 830c1dc Compare October 16, 2022 08:26
@cdburke cdburke force-pushed the master branch 2 times, most recently from c695434 to f379f4c Compare March 29, 2023 03:54
B. Wilson added 6 commits March 30, 2023 14:39
Reboot project to overhaul the make2 build system.
242f764 
The goal is to provide a build process that is friendly to package
maintainers for arbitrary Linux distributions, while maintaining the
current status quo for non-Linux targets.

In particular, the project aims to improved the following perceived
deficiences of current make2:

1. Ad-hoc, idiosyncratic shell scripts;
2. Fragile custom code for compiler detection;
3. Ignores host's CFLAGS, LDFLAGS etc;
4. Non-obvious availability of other build flags, e.g. `USE_OPENMP` etc;
5. Non-error exit code on failure;
6. Unsavory reputation amongst existing package maintainers.

The above issues are adressable by updates to the build scripts alone.
However, there are further issues that require supporting changes
elsewhere:

7. Build non-determinism (due to `__DATE__` and `__TIME__` macro usage);
8. Hostile to FHS-conforming installs.

Importantly, in the course of working on the above, we impose some
important constraints on ourself:

1. Build system changes must not introduce unintended changes in build
   products;
2. The current, officially supported build environments must not be
   broken;
3. Changes should be as conservative as possible to achieve goals.

This commit introduces a temporary meta-build tool, `jbuild.scm`, to
help maintain the above mentioned constraints. See that file for
details.
jbuild.scm: Instrument in automatic build verification
8cc6265 
Executing jbuild.scm as a script will build J from source, hash the
output, and display the result, with a comparison against a known good
target hash.

Note that, unfortunately, the diff in this commit got mixed with a large
re-indentation fix, but the bulk of semantic changes take place below
the `;;; Execution' section.
make3: Initialize make3 as a copy of make2
8484555 
Our starting point for make3 is make2.

We anticipate that trialing make3 will necessitate it sitting parallel
to make3, at least temporarily, and so have opted to copy make2 instead
of munging it directly.
make3:fix: Disable parallel builds
b9d3f68 
Unfortunately, we ran into build non-determinism, causing random
target-hash mismatches. In particular, we noticed this when a hash
mismatch fired after removing some dead code from build_jconsole.sh.

By diffing the xtrace, i.e. `set +x`, results with and without the
change, we confirmed that the executed commands were exactly the same;
however, a quick inspection using diffoscope on the two output
directories revealed differences, not in jconsole, but in libj.so.

The issue seems to be that build outputs are colocated with source
files, meaning that parallel builds end up clobbering each other's
object files.
make3: Explicitly define guix channel revision
5cdde63 
For 100% build reproducibility we also need to ensure that we build
against a stable set of dependencies. This commit defines an explicit
guix channel revision against which the build script runs.

In particular, without an explicit channel revision and when run on
different machines, the build script is likely to produce different
outputs, causing a mismatch with the target hash. In fact, the same may
happen on a single machine, if invoked across `guix pull` updates.
make3: Switch to env-style shebang
0f3038a 
Much ink has been spilled over the "correct" shebang. Here, opting for
env-style is making a declaration of *intent*: These scripts prioritize
environment flexiblity over execution precision.

In particular, using a `#!/bin/sh` shebang declares that this script
shall be run by the interpreter sitting at the absolute path `/bin/sh`.
This style of shebang is useful when needing precise control over the
executing interpreter.

However, the use case for these scripts means that we have little a
priori control over the executing interpreter, and there are legitimate
cases where users may wish to use some other shell in preference to
`/bin/sh`, e.g. containerized environments. The `#!/usr/bin/env sh`
shebang caters to these needs.

Of course, there are some cases (quite rare, now) where the env-shebang
in this commit will fail [0]; however, we make the decision to
de-prioritize these cases over the aforementioned ones.

[0]:https://www.in-ulm.de/~mascheck/various/shebang/#env
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@xelxebar @lemenkov @zhihaoy @moon-chilled @eiverson