Timeout or Race condition resulting in Cannot run all required containers

[sumdog]

In our continuous integration environment, I've been seeing errors like the following a lot:

com.example.iw.MigrationSpecs *** ABORTED *** (21 seconds, 591 milliseconds)
  java.lang.RuntimeException: Cannot run all required containers
  at com.whisk.docker.DockerKit.startAllOrFail(DockerKit.scala:59)
  at com.whisk.docker.DockerKit.startAllOrFail$(DockerKit.scala:43)
  at com.example.iw.MigrationSpecs.startAllOrFail(MigrationSpecs.scala:9)
  at com.whisk.docker.scalatest.DockerTestKit.beforeAll(DockerTestKit.scala:21)
  at com.whisk.docker.scalatest.DockerTestKit.beforeAll$(DockerTestKit.scala:19)
  at com.example.iw.MigrationSpecs.beforeAll(MigrationSpecs.scala:9)
  at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)

The test kit I'm using is pretty simple too:

import com.whisk.docker.{DockerContainer, DockerKit, DockerReadyChecker}

import scala.concurrent.Await
import scala.concurrent.duration._

trait DatabaseDockerContainers extends DockerKit {

  val mariaDBContainer = DockerContainer("mariadb:5.5").
    withEnv("MYSQL_DATABASE=dev","MYSQL_USER=dev", "MYSQL_PASSWORD=dev", "MYSQL_ROOT_PASSWORD=dev").
    withPorts(3306 -> None).
    withReadyChecker(DockerReadyChecker.LogLineContains("ready for connections"))

  val cassandraContainer = DockerContainer("cassandra:3").
    withPorts(9042 -> None).
    withReadyChecker(DockerReadyChecker.LogLineContains("Starting listening for CQL clients on"))

  def containerIP(c : DockerContainer) = Await.result(c.getIpAddresses(), 20 second).head

  abstract override def dockerContainers : List[DockerContainer] =
    mariaDBContainer :: cassandraContainer :: super.dockerContainers

}

And my Spces2 is built like so:

import com.whisk.docker.impl.dockerjava.DockerKitDockerJava
import com.whisk.docker.scalatest.DockerTestKit
import org.flywaydb.core.internal.exception.FlywaySqlException
import org.scalatest.{FreeSpec, Matchers}

class MigrationSpecs extends FreeSpec with Matchers with DockerTestKit with DatabaseDockerContainers with DockerKitDockerJava {

  "MySQL migrations" - {

I suspect that DockerKit is failing out too early when waiting for the containers to become alive. I'm going to turn logging up and see if I can reproduce/extract more information. Is there any way to expose the internal timeout and make it configurable?

[sumdog]

I think I found a solution. There are hard coded timeouts in DockerKit, but they can be overridden pretty easily:

trait MyContainers extends DockerKit {
  //override DockerKit timeouts
  override val PullImagesTimeout = 120.minutes
  override val StartContainersTimeout = 120.seconds
  override val StopContainersTimeout = 120.seconds

I think this should be mentioned in the Readme. I'll try to add it and make a pull request this weekend.