Update to Druid 0.10.1, incomplete #233

leventov · 2017-06-16T04:39:26Z

No description provided.

leventov · 2017-06-30T07:01:00Z

This PR is incomplete but I probably not going to work on it soon, so you are welcome to continue this work if you need it, please announce this with a comment to this PR if you do.

Igosuki · 2017-07-31T12:16:39Z

Is this the official PR to update to druid 0.10.0 ?

leventov · 2017-07-31T12:54:38Z

@Igosuki what do you mean by "official"? It's just a PR that I started to support Druid 0.10.1 as a dependency of Tranquility

Igosuki · 2017-07-31T13:45:04Z

@leventov ok I meant it as, if I help you finish it, it's not gonna get overwritten by any core committer who already was working on it.

leventov · 2017-07-31T13:48:36Z

@Igosuki I don't plan to continue work on this PR anytime soon so go ahead.

Igosuki · 2017-08-01T09:47:00Z

@leventov Alright, thanks !

Igosuki · 2017-08-03T14:50:58Z

After updating to 0.10.1-rc2, it seems the default Guice bindings in druid force me to add a lot of modules to the Tranquility Guicer, have you come across these problems ? Do I have to implement some noops for these modules for tranquility ?
Here are the modules I'm forced to add after updating :

+      new DruidProcessingConfigModule,
+      new DruidProcessingModule,
        new QueryableModule,
+      new QueryRunnerFactoryModule,
+      classOf[EmitterModule],

Igosuki · 2017-08-25T09:40:19Z

Yeah, the module structure changed in 0.10 so you need to pull other modules in tranquility for it to work

…

On Fri, Aug 25, 2017, 10:39 AM Gaurav M Shah ***@***.***> wrote: tried with druid-0.10.1 but stuck with: [info] - Default ExtensionsConfig *** FAILED *** [info] com.google.inject.CreationException: Unable to create injector, see the following errors: [info] [info] 1) No implementation for java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory> was bound. [info] while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory> [info] for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.<init>(DefaultQueryRunnerFactoryConglomerate.java:36) [info] at io.druid.guice.QueryableModule.configure(QueryableModule.java:49) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.guice.QueryableModule) [info] [info] 1 error [info] at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:466) [info] at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155) [info] at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107) [info] at com.google.inject.Guice.createInjector(Guice.java:96) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#233 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAYyhhzIkwy5_O4PzMYU3fLThkjMqoGwks5sbohTgaJpZM4N8BBv> .

Gauravshah · 2017-08-27T01:08:48Z

@Igosuki Thanks. can you help me get started with test cases I would like to contribute.
I am running 'sbt test' but getting disco error with null as requested host. I have local druid 0.10.1 with zookeeper. Should I add some configuration for test case for it to discover my local zookeeper ?

Gauravshah · 2017-08-29T04:09:05Z

backtrace:

[info]   com.metamx.common.ISE: Failed test
[info]   at com.metamx.tranquility.test.DirectDruidTest$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(DirectDruidTest.scala:142)
[info]   at com.metamx.tranquility.test.DirectDruidTest$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(DirectDruidTest.scala:122)
[info]   at com.metamx.tranquility.test.common.DruidIntegrationSuite$$anonfun$withDruidStack$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3.apply(DruidIntegrationSuite.scala:213)
[info]   at com.metamx.tranquility.test.common.DruidIntegrationSuite$$anonfun$withDruidStack$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3.apply(DruidIntegrationSuite.scala:212)
[info]   at com.metamx.tranquility.test.common.DruidIntegrationSuite$$anonfun$withOverlord$1.apply(DruidIntegrationSuite.scala:195)
[info]   at com.metamx.common.scala.control$$anonfun$retryOnErrors$1.apply(control.scala:42)
[info]   at com.metamx.common.scala.control$$anonfun$retryOnErrors$1.apply(control.scala:41)
[info]   at com.metamx.common.scala.control$$anonfun$withBackoff$1.apply(control.scala:53)
[info]   at com.metamx.common.scala.control$$anonfun$withBackoff$1.apply(control.scala:53)
[info]   at com.metamx.common.scala.control$.untilSome(control.scala:28)
[info]   ...
[info]   Cause: java.lang.IllegalStateException: Failed to create merged beam: druid:tranquility:indexer/xxx
[info]   at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$sendAll$2$$anonfun$26.apply(ClusteredBeam.scala:379)
[info]   at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$sendAll$2$$anonfun$26.apply(ClusteredBeam.scala:375)
[info]   at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:107)
[info]   at com.twitter.util.Promise$Transformer.k(Promise.scala:107)
[info]   at com.twitter.util.Promise$Transformer.apply(Promise.scala:117)
[info]   at com.twitter.util.Promise$Transformer.apply(Promise.scala:98)
[info]   at com.twitter.util.Promise$$anon$1.run(Promise.scala:421)
[info]   at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:200)
[info]   at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:158)
[info]   at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:272)
[info]   ...
[info]   Cause: java.lang.IllegalStateException: Failed to save new beam for identifier[druid:tranquility:indexer/xxx] timestamp[2017-08-29T04:00:00.000Z]
[info]   at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:285)
[info]   at com.metamx.tranquility.beam.ClusteredBeam$$anonfun$2.applyOrElse(ClusteredBeam.scala:282)
[info]   at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:1119)
[info]   at com.twitter.util.Future$$anonfun$rescue$1.apply(Future.scala:1117)
[info]   at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:107)
[info]   at com.twitter.util.Promise$Transformer.k(Promise.scala:107)
[info]   at com.twitter.util.Promise$Transformer.apply(Promise.scala:117)
[info]   at com.twitter.util.Promise$Transformer.apply(Promise.scala:98)
[info]   at com.twitter.util.Promise$$anon$1.run(Promise.scala:421)
[info]   at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:200)
[info]   ...
[info]   Cause: com.twitter.finagle.NoBrokersAvailableException: No hosts are available for disco!druid:tranquility:indexer, Dtab.base=[], Dtab.local=[]. Remote Info: Not Available
[info]   ...
Reporter completed abruptly with an exception after receiving event: TestFailed(Ordinal(0, 105),Failed test,DirectDruidTest,com.metamx.tranquility.test.DirectDruidTest,Some(com.metamx.tranquility.test.DirectDruidTest),Druid standalone,Druid standalone,Vector(),Some(com.metamx.common.ISE: Failed test),Some(108631),Some(IndentedText(- Druid standalone,Druid standalone,0)),Some(SeeStackDepthException),Some(com.metamx.tranquility.test.DirectDruidTest),None,pool-1-thread-1-ScalaTest-running-DirectDruidTest,1503979593710).
java.io.NotSerializableException: com.twitter.finagle.context.RemoteInfo$NotAvailable$
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
	at java.lang.Throwable.writeObject(Throwable.java:985)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
	at java.lang.Throwable.writeObject(Throwable.java:985)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
	at java.lang.Throwable.writeObject(Throwable.java:985)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
	at org.scalatest.tools.SocketReporter.apply(SocketReporter.scala:31)
	at org.scalatest.DispatchReporter$Propagator$$anonfun$run$1.apply(DispatchReporter.scala:240)
	at org.scalatest.DispatchReporter$Propagator$$anonfun$run$1.apply(DispatchReporter.scala:239)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at org.scalatest.DispatchReporter$Propagator.run(DispatchReporter.scala:239)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "Thread-5" java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: com.twitter.finagle.context.RemoteInfo$NotAvailable$
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1539)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
	at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:552)
	at java.lang.Throwable.readObject(Throwable.java:914)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

gianm · 2017-08-29T04:11:35Z

@Gauravshah the tests should be completely self-contained; this might be an issue with the tests. In particular this line seems suspicious:

java.io.NotSerializableException: com.twitter.finagle.context.RemoteInfo$NotAvailable$

Maybe it's related to a finagle version change?

Gauravshah · 2017-10-09T10:27:31Z

@Igosuki were you able to move forward ?
I had spent sometime but was not able to figure out the java.io.NotSerializableException

Igosuki · 2018-01-18T11:18:18Z

@Gauravshah I ended up pausing that work, because 0.12 is coming out soon, and the main signature types and required modules completely changed.

Plus it seems the strategy is to move away from tranquility towards kafka indexing service, will come back to that later.

I am currently running the latest Tranquility with avro extensions against 0.10.x and it works fine because the index specs haven't changed.

Gauravshah · 2018-01-18T11:30:17Z

thanks @Igosuki , we are using kinesis so wondering how would kafka indexing service would work in that world.

Druid 0.10.1, work in progress

c5fcf3c

leventov changed the title ~~Druid 0.10.1, work in progress~~ Druid 0.10.1 Jun 30, 2017

leventov changed the title ~~Druid 0.10.1~~ Update to Druid 0.10.1, incomplete Jun 30, 2017

leventov mentioned this pull request Jun 30, 2017

Tranquility fails with latest druid-histogram plugin #235

Open

leventov mentioned this pull request Jul 20, 2017

Support Druid 0.10.0 #240

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to Druid 0.10.1, incomplete #233

Update to Druid 0.10.1, incomplete #233

leventov commented Jun 16, 2017

leventov commented Jun 30, 2017 •

edited

Loading

Igosuki commented Jul 31, 2017

leventov commented Jul 31, 2017

Igosuki commented Jul 31, 2017

leventov commented Jul 31, 2017

Igosuki commented Aug 1, 2017

Igosuki commented Aug 3, 2017

Igosuki commented Aug 25, 2017 via email

Gauravshah commented Aug 27, 2017 •

edited

Loading

Gauravshah commented Aug 29, 2017

gianm commented Aug 29, 2017

Gauravshah commented Oct 9, 2017

Igosuki commented Jan 18, 2018

Gauravshah commented Jan 18, 2018

Update to Druid 0.10.1, incomplete #233

Are you sure you want to change the base?

Update to Druid 0.10.1, incomplete #233

Conversation

leventov commented Jun 16, 2017

leventov commented Jun 30, 2017 • edited Loading

Igosuki commented Jul 31, 2017

leventov commented Jul 31, 2017

Igosuki commented Jul 31, 2017

leventov commented Jul 31, 2017

Igosuki commented Aug 1, 2017

Igosuki commented Aug 3, 2017

Igosuki commented Aug 25, 2017 via email

Gauravshah commented Aug 27, 2017 • edited Loading

Gauravshah commented Aug 29, 2017

gianm commented Aug 29, 2017

Gauravshah commented Oct 9, 2017

Igosuki commented Jan 18, 2018

Gauravshah commented Jan 18, 2018

leventov commented Jun 30, 2017 •

edited

Loading

Gauravshah commented Aug 27, 2017 •

edited

Loading