Single Process

The idea of the "single process" project is to run all Druid services within a single Java executable. Most useful during debugging, and may also be useful in other contexts.

Analysis

This section explains how Druid's service configuration works today.

Main

Druid code resides in a single Java jar file. Command line options choose which service to run. The entry point is org.apache.druid.cli.Main. Main is essentially a wrapper around the io.airlift.airline.Cli abstraction (marked as deprecated by the Airlift project). The work of each available option occurs in the "command" (runnable) selected by the CLI.

The CLI configuration offers, as an option, "Run one of the Druid server types." So, our first challenge is to alter the CLI to run multiple server types.

Today, we launch a service using server historical, say. The CLI works by building a mapping from commands to Runnables. Here we focus on the list of servers:

    builder.withGroup("server")
           .withDescription("Run one of the Druid server types.")
           .withDefaultCommand(Help.class)
           .withCommands(serverCommands);

    List<Class<? extends Runnable>> toolCommands = Arrays.asList(
        DruidJsonValidator.class,
        PullDependencies.class,
        CreateTables.class,
        DumpSegment.class,
        ResetCluster.class,
        ValidateSegments.class,
        ExportMetadata.class

The first part of the command, server appears above as the name of the group. We include a list of classes. The CLI gets the second name, historical from an annotation:

@Command(
    name = "historical",
    description = "Runs a Historical node, see https://druid.apache.org/docs/latest/Historical.html for a description"
)
public class CliHistorical extends ServerRunnable

Hence, at this level, we want to specify multiple services, instead of just one.

Note that we list the command classes above: the CLI mechanism creates an instance for us of the selected command. This pattern conflicts with Guice, which wants to be the one to create instances and thus do dependency injection. This conflict is handled below.

Basic Guice Configuration

Druid uses Google Guice to manage dependencies. (Tutorial). Key points:

• Configuration is done via an Injector. The "universe" of bindings available to an application is defined by that Injector. Fortunately, it appears Guice allows the creation of multiple, independent Injectors.
• A Module is a collection of bindings.

Configuration starts with a "startup injector":

    final Injector injector = GuiceInjectors.makeStartupInjector();

The startup injector provides a set of service-independent modules defined in GuiceInjectors. Since these are independent of services, we can safely ignore these modules for this project.

Jerseys appears to be a global collection of JSR 311 implementations used to (define the REST API?)

Next, the code connects the CLI mechanism with the startup Guice configuration:

      final Runnable command = cli.parse(args);
      injector.injectMembers(command);
      command.run();

Note that the code here and elsewhere is simplified: error handling and other non-essential cruft is removed.

The injectMembers method works around the fact that the CLI created the instance without dependency injection: we ask Guice to inject dependencies into member variables after creation. At this point, all we can inject are the startup modules mentioned above. Hence, dependency injection here is still independent of the service. For example, for the base ServerRunnable class:

    @Inject
    @Self
    private DruidNode druidNode;

    @Inject
    private DruidNodeAnnouncer announcer;

    @Inject
    private ServiceAnnouncer legacyAnnouncer;

    @Inject
    private Lifecycle lifecycle;

    @Inject
    private Injector injector;

And in the GuiceRunnable[https://github.com/apache/druid/blob/master/services/src/main/java/org/apache/druid/cli/GuiceRunnable.java] class:

  @Inject
  public void configure(Injector injector)
  {
    this.baseInjector = injector;
  }

Note that we inject the startup injector twice. Not sure if this is a bug or a feature.

At this point the service command runs which is where the service-specific configuration happens.

Service Configuration

Service configuration happens in ServerRunnable.run():

  public void run()
  {
    final Injector injector = makeInjector();
    final Lifecycle lifecycle = initLifecycle(injector);
    lifecycle.join();
  }

The above basically boils down to three steps:

• Service configuration: makeInjector()
• Start the service: initLifecycle(injector)
• Wait for the service to complete: lifecycle.join()

Of these, the last is a bit tricky because each service will manage a set of threads, and all must be launched and waited upon rather than just one today.

Server Module Configuration

In GuiceRunnable.makeInjector():

  public Injector makeInjector()
  {
    return Initialization.makeInjectorWithModules(baseInjector, getModules());
  }

The baseInjector is the startup injector we saw injected above. The called method is:

  public static Injector makeInjectorWithModules(final Injector baseInjector, Iterable<? extends Module> modules)
  {
    final ModuleList defaultModules = new ModuleList(baseInjector);
    ...
    return Guice.createInjector(Modules.override(intermediateModules).with(extensionModules.getModules()));
  }

The second parameter is a list of modules specific to the service we want to run.

The first line creates a module list with all the modules from the base (startup) injector. Copying modules results in recreating any singletons already defined. To work around this issue, the ModuleList constructor copies over some instances:

    public ModuleList(Injector baseInjector)
    {
      this.baseInjector = baseInjector;
      this.modulesConfig = baseInjector.getInstance(ModulesConfig.class);
      this.jsonMapper = baseInjector.getInstance(Key.get(ObjectMapper.class, Json.class));
      this.smileMapper = baseInjector.getInstance(Key.get(ObjectMapper.class, Smile.class));
      this.modules = new ArrayList<>();
    }

Open question: is there a reason for the above pattern rather than just creating a child injector? Maybe the code above was written before Guice offered child injectors?

The bulk of the method defines the default modules, asks the service to add its own modules, and includes any available extension modules.

The last line creates a new injector which essentially copies all modules from the startup injector, and adds a large set shared by all services. The good news is that each service gets its own injector, and so its own set of mappings. The bad news is that, if we allow multiple services to run, the common services will be duplicated across services, possibly resulting in multiple copies of objects which should be singletons.

The getModules() defines modules unique to each service. For example, for the Historical:

  protected List<? extends Module> getModules()
  {
    return ImmutableList.of(
        new DruidProcessingModule(),
        new QueryableModule(),
        new QueryRunnerFactoryModule(),
        new JoinableFactoryModule(),
        binder -> {
          binder.bindConstant().annotatedWith(Names.named("serviceName")).to("druid/historical");
          binder.bindConstant().annotatedWith(Names.named("servicePort")).to(8083);
          binder.bindConstant().annotatedWith(Names.named("tlsServicePort")).to(8283);
          ...
          binder.bind(ServerManager.class).in(LazySingleton.class);
          binder.bind(SegmentManager.class).in(LazySingleton.class);
          binder.bind(ZkCoordinator.class).in(ManageLifecycle.class);
          ...
          Jerseys.addResource(binder, QueryResource.class);
          Jerseys.addResource(binder, SegmentListerResource.class);
          Jerseys.addResource(binder, HistoricalResource.class);
          LifecycleModule.register(binder, QueryResource.class);
          ...
          },
        ...
    );

There is quite a bit going on: only a few samples are shown that show the issues we must consider.

First, we are advised to "think of Guice as being a map". We see some examples of this basic concept where we bind constants to keys:

          binder.bindConstant().annotatedWith(Names.named("serviceName")).to("druid/historical");

If we run multiple services, then clearly we need a distinct map for each service to manage such bindings.

In other cases, we rely on some indirection. For example, for the segment walker:

          binder.bind(ServerManager.class).in(LazySingleton.class);

Which is implicitly bound to the QuerySegmentWalker interface in:

public class QueryLifecycleFactory
{
  ...

  @Inject
  public QueryLifecycleFactory(
      final QueryToolChestWarehouse warehouse,
      final QuerySegmentWalker texasRanger,

As a result, in the Historical, we bind QuerySegmentWalker to ServerManager. But, in the Broker:

          binder.bind(CachingClusteredClient.class).in(LazySingleton.class);

Note that there is no explicit knowledge that we're binding to the interface: Guice evidently figures it out on the fly. This seems to use Guice linked bindings. Again, this is not an issue if each service has its own injector (Guice map).

Druid Guice Extensions

Druid adds several extensions to the basic Guice story. Most of these live in the ModuleList class. ModuleList works with two injectors, the explicit baseInjector, and the to-be-created service injector. Essentially, each module added to ModuleList is injected with dependencies defined in the baseInjector, giving a module four stages of configuration:

• Empty constructor (no configuration done here)
• Member dependency injection from the base (startup) injector.
• Guice-called configure() method in which the module adds its configuration to the target service injector.
• Guice injects dependencies into other objects using the bindings defined in the module.

In addition, the module list allows modules to be excluded via a configuration setting in ModulesConfig.

The "default" (cross-service) modules are added to a ModuleList directly. The per-service modules are added first to a List, then copied into a ModuleList, ensuring that all modules go through the above process.

Lifecycle

Property Configuration

Druid allows configuration via a set of Java properties-style configuration files.

(How are these mixed into Guice?)

Jetty Configuration

Druid uses the Jetty server to manage REST requests. The JettyServerModule(https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/server/initialization/jetty/JettyServerModule.java) class configures and manages the Jetty server. It is one of the modules added during pre-command initialization, and is thus shared by all services.

The Jetty server itself is started via the Lifecycle class, as a Handler:

    lifecycle.addHandler(
        new Lifecycle.Handler()
        {
          @Override
          public void start() throws Exception
          {
            log.debug("Starting Jetty Server...");

As a result, it would seem that if we run multiple services, we would (perhaps) have multiple lifecycles, and thus multiple Jetty servers. This suggests that, to run multiple services, we must have an overall lifecycle that manages Jetty (among other things), with a "child" lifecycle to manage each service.

Service Lifecycle

The Lifecycle class manages the set of objects for a service.

The GuiceRunnable.initLifecycle() method, implemented on each service CLI object, assumes it is managing the only service in a JVM. For example:

• It logs information about memory and CPU resources.
• It lists all the configuration properties.
• It configures the (global) logger.

To run multiple services, this concept must be split to have one process manager (which does the above) along with a separate service manager (which handles many other tasks.) This is more than just a "wiring" task.

The lifecycle itself is created through Guice, which maintains a singlton instance:

      final Lifecycle lifecycle = injector.getInstance(Lifecycle.class);

The call happens early enough that all services must use the same class, which is good. Unfortunately, if multiple services were run in the same process, they would also use the same Lifecycle instance, which is not so good. Thus, perhaps we'd want to register a "lifecycle factory" to be able to manage multiple instances.

Further, the LifecycleModule class provides static methods that register classes with both the lifecycle and Guice. This class assumes that Guice holds our one and only Lifecycle instance.

Approach

The analysis above suggests we must tackle three problems:

• CLI revision to allow multiple services to be requested
• Separate the per-server configuration from per-service configuration
• Separate the per-server lifecycle from the per-service lifecycle

CLI Changes

The command line changes are the simplest. The current service command is:

<druid exec> service historical

The desired new behavior is to allow multiple services:

<druid exec> service broker historical

However, it seems that the library Druid uses can't quite be made to work as above. An alternative is to introduce a new command:

<druid exec> bundle broker historical

Where the command is bundle and the services are Boolean options.

((TODO: Implementation details.))

A new command is necessary which could be a composite command: a "run multiple services" command that has, as children, the services to run. The composite command would handle the shared-server setup as distinct from the single-service setup.

Per-Server and Per-Service Configuration

In a nutshell, the Initialization.makeInjectorWithModules() method must be split. The modules listed in that method are common to all services, while the modules defined in the modules parameter (created by GuiceRunnable.getModules()) must be defined per service.

Fortunately, Guice 2.0 and later defines the idea of a child injector. The rough design is then to use three injectors:

           initialization injector
                     ^
                     |
               server injector 
               ^            ^
               |            |
  broker injector          historical injector

In this way, objects which are different for each service go into the service injector; those which are for the server (and shared across multiple services) go into the server injector. The initialization injector exists. The server injector would be created as a child to hold the modules from makeInjectorWithModules() while the service injector is a child of the server injector which holds the modules created in getModules(). (The server injector is not absolutely necessary, presenting it just simplifies the discussion.)

The trick is that some seemingly-shared resources (such as the LifecycleModule) hold state for a single service and so must be adjusted so that it can live in the service injector. (Which other modules are similar?)

Per-Server and Per-Service Configuration

The Lifecycle explained above appears to make assumptions that it a single instance exists per Java process, and that it manages a single service. We must split this concept into two:

• ServerLifecycle which manages a Druid process, and
• ServiceLifecycle which manages a Druid service.

(It is not clear if we need two classes, or just different uses of the current class.)

Further, the ServerRunnable, despite its name, is really a Service Runnable. This code is at the server level:

  public void run()
  {
    ...
    final Lifecycle lifecycle = initLifecycle(injector);
    lifecycle.join();

A service command should have two parts. The first just runs the service. The second runs a server with the service. When launching a single service, the same command handles both tasks. When launching multiple services, a new "composite" command handles the server tasks, while the service commands handle just the service tasks.

Development Steps

A plan to implement the above might be:

Verify that the child injector idea works as desired:

• Child injector for a service (initially, everything in makeInjectorWithModules().)
• Two injectors: server (those modules listed in makeInjectorWithModules()) and service (those modules passed in as arguments.)

Multiple service arguments.