Skip to content

Single Process

Jump to bottom
Paul Rogers edited this page Dec 2, 2021 · 17 revisions

The idea of the "single process" project is to run all Druid services within a single Java executable. Most useful during debugging, and may also be useful in other contexts.

Below is a list of areas of Druid that are affected by the multi-service concept. Each section discuses the current structures, challenges for multi-service, and presents possible solutions. The sections are inter-dependent. Solutions presented in one section must assume solutions for other areas defined in another section.

Open Issues

Additional topics to investigate:

  • Service announcements
  • Zookeeper
  • DruidNode

Approach

The analysis above suggests we must tackle several problems.

  • Allow access to multiple configuration files
  • Separate the per-server configuration from per-service configuration
  • Separate the per-server lifecycle from the per-service lifecycle

Development Steps

  • Add the bundle CLI command. (Done)
  • Add the modified properties file handling. (Done)
  • Child injector experiment. (See below)
  • Mechanism to manage per-service properties in per-service injector

Combined Injector

For multiple reasons, it seems not practical to use a two-level injector system. Eventually, if might be good to get there, but doing so is likely to be invasive. The alternative is to throw everything into a single injector. Some challenges to resolve:

  • Binding conflicts: keys bound one way in one service, another way in another. First have to identify such conflicts. Then, resolve them, perhaps with binding annotatins.
  • Jetty web apps. There can't be one pool of web apps, resources must be grouped by app, which is bound to a port.

Child injector experiment - Not practical

Goal is to have two-levels of properties, three levels of injectors, and to use injector inheritance.

Partial results showed that this approach would be difficult: Druid makes many assumptions about Guice configuration which are violated when we move to multiple injectors. See the Guice Configuration page for details.

  • Create startup injector as two levels: a (for now) empty root and a (for now) startup with the usual stuff (Done)
  • Mechanism to pass the root startup injector to the command (just ask for the parent)
  • Command creates a child injector based on the (for now) empty root. (Done)
  • Test and resolve issues (Done)
  • Move properties from startup to child injector. (Done)
  • Remove property copying from DruidSecondaryModule (Done)
  • Test to see if inheritance works (Done)
  • Do the same for the other "copied" modules.
  • Test
  • Revise properties to shift back to startup injector. Add a properties provider to secondary. Secondary takes name and gets service props.
  • Test
  • Split service injector into shared, specific bits. Handle inheritance.
  • Test

As expected, things are not quite as easy as it looks:

  • The Jackson stuff is built twice: once in the startup injector (based on defaults?) and a second time in the secondary injector (based on extensions?)
  • The Config thing doesn't seem to want to work properly
  • The real challenge is with module overrides

Verify that the child injector idea works as desired:

  • Child injector for a service (initially, everything in makeInjectorWithModules().)
  • Two injectors: server (those modules listed in makeInjectorWithModules()) and service (those modules passed in as arguments.)
Clone this wiki locally