Potential to remove OFF_SCRIPT case.

[pefribeiro]

Currently the ROS test driver will exit with OFF_SCRIPT in case it sees an event (output?) which is not expected.

rtcg/internal/gen/cpp/embed/templates/base/out_callback.cpp.tmpl

Lines 25 to 29 in a10b169

	else
	{
	{{ template "log" "info" }}"Unexpected value " << value << " for {{ $.Channel }} in state {{ $s }}."{{ template "end_log" }}
	end(rtcg::Status::OFF_SCRIPT);
	}

It may be unecessary to distinguish this in the implementation. Effectively for (SUT |[ αSP ]| T )\αSP we have a deadlock when SUT and T can't progress further.

[MattWindsor91]

Would there be any possibilities where the SUT can produce an unwanted output A initially, then produce the wanted output B immediately after, without any intervening input from T? In that case, the OFF_SCRIPT case would end the test, but I think removing it would cause T to progress as if it had never seen A, only B.

That said, it might be that either the message passing semantics of ROS1 or the structure of the tests make that possibility moot or impossible to test against.

[pefribeiro]

Would there be any possibilities where the SUT can produce an unwanted output A initially, then produce the wanted output B immediately after, without any intervening input from T? In that case, the OFF_SCRIPT case would end the test, but I think removing it would cause T to progress as if it had never seen A, only B.

Ok, I think I see your point. I was just expecting the machine for T to deadlock. Is this early termination an optimisation?

[MattWindsor91]

It's a bit difficult because the mental model I had for this generator has turned out to not match up with ROS1 particularly well, and it's been a while since I wrote the templates, but I think the effect if the off-script cases here were eliminated would be that the tester would stay in the same state. That isn't quite the same as deadlock, because it's still able to receive and act upon subsequent outputs.

So it'd be something like

Test: [out A, out B] fails if we see out C

This generates the STM:

• State 1 accepts A (goes to state 2)
• State 2 accepts B (goes to state 3)
• State 3 accepts C (fails)

In the current situation, if the SUT outputs B in state 1, the whole test goes off-script, as we're assuming that the SUT is outputting topics in-order (which may be an invalid assumption). If the off-script tests were removed, then the SUT would stay in state 1 after the SUT outputs B. But if we then receive A from the SUT, it moves to state 2, and so the trace

[out B, out A, out B]

becomes passing instead of inconclusive and the trace

[out B, out A, out B, out C]

becomes failing instead of inconclusive.

However... if I understood Lenka correctly, there might not be any guarantee that if the SUT sends A then B, A will propagate through to the tester before B. They're separate topics. I think the tester runs in a sequential manner, so there shouldn't be any parallelism induced ordering strangeness at its end, but I'm not sure how we establish the guarantees we'd need to be able to reason about the order of outputs.

[pefribeiro]

Ok, thanks for your input Matt. No need to consider this further, I'll take it from here.