unexpected ordering of tests?

[coreyoconnor]

Not entirely sure. Trying to work through some oddness I'm encountering in a library that uses distage-test when upgrading the distage dependency from 1.1.0-M23 to 1.2.15. This library also replaces the test registration mechanism which, no doubt, complicates all this. Anyways:

1. create a test that depends on a specific order of test cases.
2. Using Sequential to request sequential ordering of test cases: https://izumi.7mind.io/distage/distage-testkit.html#execution-order

The expectation is that for a given group, the expected sequential order is guaranteed. from docs: "This test requires the effect of the first test case to occur prior to the second test case. As discussed Execution Order section: Without configuring test cases for sequential execution this order would not be guaranteed."

However, I have a test structured like below. Which executes out of order. Specifically, the order ends up 1 and then 4. Which is unexpected.

      memoizationRoots = superConfig.memoizationRoots ++ Set(
        DIKey[Delegate],
        DIKey[Instance],
        DIKey[PreludeInit.Ref]
      ),
      forcedRoots = superConfig.forcedRoots ++ Set(
        DIKey[Delegate],
        DIKey[Instance],
        DIKey[PreludeInit.Ref]
      )
    )
  }

  "ServiceFragmentRouter" should {

(1)
    "init" in {
      (spawner: ActorSpawner, system: ActorSystem[Nothing], preludeInit: PreludeInit.Ref) =>
...
    }

(2)
    "initially responds to /_ops/groups.json with an empty set" in {
...
    }

(3)
    "adding a group adds to the groups set" in {
...
    }

(4)
    "re-initializes with added group" in {
      (
        config: Config,
        spawner: ActorSpawner,
        registry: ControllerRegistry.Delegate,
        system: ActorSystem[Nothing]
      ) =>
...
    }

(5)
    "responds to /_ops/services.json with prelude enabled services" in {
...
    }

(6)
    "added group defaults to no instantiated services" in {
...
    }
  }

The test logs indicate all 6 tests are in the same enviornment:

I 2024-11-25T17:47:08.525          (DistageTestRunner.scala:136)  …ner.logEnvironmentsInfo.133 [30:pool-1-thread-1     ] phase=tes
tRunner Memoization environment with suites=1 tests=6 suitesMemoizationTree=                                                      
╗ [Level 0; 0 current tests + 6 nested tests] roots: [ {type.QuasiIORunner[=λ %0 → ZIO[-Any,+Throwable,+0]]}, {type.TestTreeRunner
[=λ %0 → ZIO[-Any,+Throwable,+0]]} ] transitive: [ {id.Executor@cpu}, {type.UnsafeRun2[=λ %1,%2 → ZIO[-Any,+1,+2]]}, {type.PlanCir
cularDependencyCheck}, {type.IndividualTestRunner[=λ %0 → ZIO[-Any,+Throwable,+0]]}, {id.Executor@io}, {type.ExtParTraverse[=λ %0 
→ ZIO[-Any,+Throwable,+0]]}, {type.TestReporter}, {type.TimedActionF[=λ %0 → ZIO[-Any,+Throwable,+0]]}, {type.IO2[=λ %0,%1 → ZIO[-
Any,+0,+1]]}, {type.Async2[=λ %1:0,%1:1 → ZIO[-Any,+1:0,+1:1]]}, {type.UnsafeRun2::ZIORunner[=Any]}, {type.IzLogger}, {type.TestSt
atusConverter}, {type.PlanningOptions}, {type.UnsafeRun2::FailureHandler}, {type.QuasiIO[=λ %0 → ZIO[-Any,+Throwable,+0]]}, {id.Li
st[+ZLayer[-Any,+Nothing,+Any]]@zio-runtime-configuration}, {type.TestEnvironment::EnvExecutionParams}, {id.ZEnvironment[+Any]@zio
-initial-env}, {type.QuasiAsync[=λ %0 → ZIO[-Any,+Throwable,+0]]}, {type.HKTag[=(Object {type Arg = λ %0 → ZIO[-Any,+Throwable,+0]
})]}, {id.IzLogger@distage-testkit}, {type.TestkitLogging} ]                                                                      
║                                                                                                                                 
╚════╗ [Level 1; 6 current tests + 0 nested tests] roots: [ {type.ActorSpawner}, {type.ServiceFragmentRouter::Instance}, {type.Act
orRef[-Unit]}, {type.ActorRef[-ServiceFragmentRouter::Envelope]}, {type.ActorRoot}, {type.Cluster} ] transitive: [ {type.ActorSyst
em[-Nothing]}, {resource.{type.ActorSpawner}}, {type.ServerConfig}, {type.Pekko}, {type.ActorSystem[-SpawnProtocol::Command]}, {re
source.{type.Pekko}}, {type.Logging}, {effect.{type.ActorRef[-ServicesManager::Proto]}}, {type.ActorRef[-ServicesManager::Proto]},
 {resource.{type.ActorSystem[-SpawnProtocol::Command]}}, {type.Config}, {resource.{type.ServerConfig}}, {proxyinit.{type.ActorRef[
-ServiceFragmentRouter::Envelope]}}, {type.ActorRef[-ControllerRegistry::Envelope]} ]                                             
     ╠══* ServiceFragmentRouterSpec                                                                                               
I 2024-11-25T17:47:08.586          (DistageTestRunner.scala:111)  i.d.t.r.i.D.p.93.108.110 [30:pool-1-thread-1     ] Processing te
sts=6 using monad=λ %0 → ZIO[-Any,+Throwable,+0]                                                                                  

Of course, perhaps my understanding or the docs are wrong. Which, iirc, I wrote that doc so that would just be the worst T_T

Looking into the distage code groupTests seems to determine the (unintented?) ordering:

    val out = distageTests
      .groupBy(_.environment.getExecParams)
      .view
      .mapValues(_.groupBy(_.environment))
      .toSeq
      .map {
        case (envExec, testsByEnv) =>
          val configLoadLogger = IzLogger(envExec.logLevel).withCustomContext("phase" -> "testRunner")

          val memoizationEnvs =
            QuasiAsync[Identity].parTraverse(testsByEnv) {

Which eventually goes to ExtParTraverse:

      val sorted = l.groupBy(getParallelismGroup).toList.sortBy {
...
        case (Parallelism.Sequential, _) => 3
      }
      F.traverse(sorted) {
...
        case (_, l) => F.traverse(l)(f)
 

The _.groupBys stick out to me. IIRC those do not guarantee order stability.

But why does this only show up in my Case tests? I mentioned I override the test registration. Internally I use a var registeredTest = Seq.empty[...] instead of an mutable.ArrayBuffer. Which, if I change from one to the other: A different test order.

Am I incorrect in my expectations?

[neko-kai]

@coreyoconnor
Hmm, we don't have tests for test ordering right now. We may have lost the guarantees for test ordering at some point when adding new features – if we had them in the first place!

If you could use the debugger to find the location in testkit where the ordering gets mangled or write a standalone reproduction that would be greatly appreciated. Then we could fix the issue and add regression tests to make sure we actually do keep the ordering.

[coreyoconnor]

As for standalone reproduction. I'm in the process of opensourcing the whole project I'm working on. So there will be that shortly. But a more focused tests seems easy. I'll see what I can do.

[pshirshov]

I think you are right about .groupBy, it builds a hashmap thus breaks the order. We need to make an order-safe version of it.

[coreyoconnor]

I haven't had time to look at this again. The draft PR I put I think changes too much. Looking at a narrower change.