custom-build-runtimes.md

This spec describes how we might improve Gradle's support for customized build runtimes, also known as 'custom Gradle distributions'.

Use cases

The following uses the term "build runtime" to mean a logical set of Java runtime, Gradle runtime and Gradle plugins. It may or may not be packaged in a ZIP or set of JARs or whatever.

• Make a set of custom plugins available by plugin id for each build.
• Apply some implicit configuration (possibly implemented as a plugin) for each build.
• Logic to determine where to download the runtime and plugins from.
• Use the daemon with custom runtime.
• Dynamic and changing runtimes.
• Integration testing of a runtime.
• Build and publish a runtime.
• Bootstrap a new project to use a custom runtime.
• Locate or install a compatible Gradle runtime.
• Locate or install compatible JVM.
• Wrapper works well when added to the command-line PATH.
• Wrapper cannot be used because 3rd-party libraries (i.e. gradle-wrapper.jar) cannot be checked into source control.
• Dependency resolution should treat all libs that are part of the build runtime as local resource candidates, to avoid additional downloads.

Implementation plan

The Gradle wrapper and core will be split up into several pieces:

1. A bootstrapper that initiates the startup process.
2. A meta-data provider that inspects the local environment and meta-data files to determine where to find the other peices.
3. An artifact download and cache provider.
4. The Gradle launcher that completes the startup process and invokes the Gradle runtime.
5. The Gradle runtime.

Gradle startup becomes:

• Boostrapper looks for a bootstrap meta-data provider:
  • A bootstrap meta-data provider implements a public API so that a custom implementation can be used.
  • Located by looking in a number of places:
    • A JAR or JARs in a well-known location in the source tree.
    • Bundled with the bootstrapper.
    • Present in the download cache.
    • Downloaded from some hard-coded fallback URL at https://services.gradle.org
  • Default implementation generated by the wrapper task uses gradle-wrapper.properties and/or gradle.properties to decide where to locate the launcher ZIP.
  • If the bootstrapper cannot find an appropriate meta-data provider, it points the user at some download location at the Gradle website for manual installation.
• Bootstrapper downloads the launcher ZIP, if required, and then calls some well-known, but internal, entry point in it.
• The tooling API and the gradlew and gradle commands are all boostrappers, as per above.
  • The tooling API falls back to use gradle-wrapper.properties if no provider found (for backwards compatibility), and a hard-coded URL if no gradle-wrapper.properties found.
  • The gradle command falls back to use gradle-wrapper.properties if no provider found (to work with older builds), and uses the distribution it is running from to provide the launcher installation if none other has been specified.
  • Each bootstrapper may use its own download and cache implementation, so that gradlew may use java.net.URL whereas gradle and the tooling API may use the http-client and artifact cache backed resource implementations.
• Launcher ZIP contains:
  • Dependency resolution infrastructure.
  • Logging and console integration.
  • Launcher implementation.
• Launcher looks for a build environment meta-data provider:
  • Build environment meta-data provider uses whatever local meta-data and environment information it needs to determine:
    • The build runtime that the build requires (expressed as dependency notations) + the definitions of the repositories to locate these runtimes.
    • A build runtime implies a set of plugins to be implicitly applied to the build. These might be implemented as init scripts or as compiled Plugin<Gradle> implementations, and packaged in a number of JAR files.
    • A build runtime implies a set of compatible Gradle runtimes. These might be packaged in a single ZIP or as a number of JAR files.
    • A build runtime implies a set of compatible Java runtimes.
  • Uses a public API so that a custom implementation can be used.
  • Located in the same way as the bootstrap meta-data provider.
  • Default implementation uses gradle-wrapper.properties and/or gradle.properties to provide this meta-data.
• Launcher resolves the core modules it needs. Gradle runtime version and Java runtime version are inferred from the runtime meta-data.
  • Asserts that the runtime will-work with the Gradle version the launcher provides.
• Launcher starts the build in a compatible JVM (as it does now - either in a long-lived daemon or a single use daemon).
• Launcher resolves and applies the plugins for the specified runtime. These in turn may apply further plugins.
• Build proceeds.

The Gradle artifacts described above would be published by Gradleware to a global repository and be visible via https://services.gradle.org.

Here's how you might use this:

Firstly, make one or more build runtimes available. You can simply use the vanilla runtimes hosted at https://services.gradle.org. Alternatively you could:

1. Add the Gradle artifacts to your binary artifact repository. This would be made up of a launcher ZIP, plus some other artifacts - maybe a few ZIPs or JARs + meta-data. You would normally do this by simply proxying https://services.gradle.org.
2. Publish one or more build runtime components. You would usually do this by setting up a build that uses a new build-runtime plugin to publishe the appropriate stuff, along with a new gradle-plugin plugin to publish a set of Gradle plugin components.
3. Optionally, you might implement and publish a custom launcher or build environment meta-data provider. These would simply be packaged as JARs.

To setup a build to use this:

1. Configure the Wrapper task (or some model object):
   • Define one or more repositories where build infrastructure can be located.
   • Declare the build runtime dependencies, in dependency notation.
   • Optionally declare a custom launcher or build environment meta-data provider, in dependency notation.
2. Run the wrapper task:
   • If a launcher meta-data provider has been specified, this task embeds the wrapper JAR, meta-data files and the provider JAR into the source tree.
   • If not, this task resolves the build runtime to determine the launcher ZIP to use, then embeds the wrapper JAR, meta-data files that include the launcher ZIP URL, and the default providers, into the source tree.

To use the build, you would run the gradlew or gradle commands, or point the tooling API at it.

Note that the above only supports a static Gradle version - out of the box at least. In order to change the Gradle version used by the build, you need to run the wrapper task again. The plugins are resolved as per normal and support all of the dependency resolution capabilities.

Some options for supporting a dynamic Gradle version:

1. You implement a custom launcher meta-data provider that calculates the appropriate launcher to use.
2. The launcher ZIP specified in the wrapper meta-data is used only for the first build. This build resolves the build runtime dependencies, and the result is cached in the $rootDir/.gradle directory. A bootstrapper would use this cached value for any subsequent builds. The cached value would be refreshed periodically.
3. There are two built-in meta-data providers - a small implementation that handles static versions only, and a larger implementation that can resolve the launcher ZIP to use. The wrapper task would choose the appropriate provider to embed in the source tree based on the declared dependencies.
4. Some kind of multi-stage startup, where the launcher resolves the runtime and then invokes it through some cross-version internal API. This is complicated by the fact that logging and console integration plus dependency resolution is required by both the launcher and runtime.

From the command-line:

• Provide some way to define the default Gradle version of your machine via a Gradle command. This version is used if no project specific metadata is provided.
• Provide a command line switch to ignore project metadata and use the default metadata (i.e. a build master wants to try the build with the default version of Gradle).
• Provide some way to enable/disable the daemon from the command line.