Watch closer what SLIM is doing #1500

six42 · 2024-04-01T20:37:02Z

This is an alternative implementation to #1469 from @pvbemmelen62
As it is fully implemented in the Slim Server it will work for all Slim ports and not only for the Java Slim Client as the other mentioned PR.

It adds a new feature to see the instructions SLIM is creating step by step and also see how a SLIM table changes with each execution.
To achieve this result the Slim Test System needed a change to suport processing of SINGLE instructions.
The existing SLIM implementation always sends a full table to the SLIM client for execution, which I call BULK mode. In the past there may have been performence concernces for the network communication to do it in this bulk way. Nowdays this is not an issue.
Modern protocols like the Language Server Protocol or Debug Adapter Protocol work also in a step by step approach to get the best interactive experience. Nevertheless this new SINGLE mode is not active by default. Default is still the BULK mode. Only when more interaction is requested the SLIM test system will switch to this mode.

Usage Guide - From the updated Wiki page

When a SLIM test is run the the webpage will update each time a table has been fully processed.
If you want to get quicker an undate you have to set the property: ''slim.show''

At the top of the test page you will then see 2 additional outputs

A list of the last 10 instructions the SLIM server is sending to the SLIM Client and the results of the same.
The current table and how its cells are updated after each instruction executed.

In addition you can use the following two properties to customize this further:

slim.show.size - default 10 - Number of instructions to show
slim.show.sleep - default 0 - Waits for X milliseconds between each instruction call. This is for making nice videos of a test run. Otherwise I don't think there is a usecase for slowing down the execution of your test

Those properties can be either provided by a wiki page, on the command line (e.g. ''-Dslim.port=9000'') or in the plugins.properties file.

Watch the video:

Fitnesse_StepByStep_final.mp4

Sample Invocation:
http://localhost:80/FitNesse.UserGuide.TwoMinuteExample?test&slim.show&slim.show.sleep=90&slim.show.size=15

…ts after each instruction execution Two new parameters can be given on the command line to see what fitnesse is currently doing 1) A) slim.sbys=SHOWTABLE show the html table after each executed instruction B) slim.sbys=SHOWINSTRUCTIONS show the instruction after each executed instruction C) slim.sbys=TRUE instead of sending a full tabke to the slim client send just one instruction each time 2) only if 1 is given, will delay the execution by 0.5 seconds just useful for showing the beaviour slim.sbys.sleep=SLEEP Examples: http://localhost/FitNesse.SuiteAcceptanceTests.SuiteSlimTests.TestComparators?test&slim.sbys=SHOWINSTRUCTIONS&slim.sbys.sleep=SLEEP http://localhost/FitNesse.SuiteAcceptanceTests.SuiteSlimTests.TestComparators?test&slim.sbys=SHOWTABLE&slim.sbys.sleep=SLEEP

…e slim.sbys =SHOWINSTRUCTIONS

Added documentation all code for the display is now in the SuiteHtmlFormatter and not in the SlimTestSystem

pvbemmelen62 · 2024-04-04T16:55:17Z

Hi six42 ,

I see a couple of drawbacks in having the slim instructions shown in the run log page itself:

the java code is complex, compared to when showing it in a separate window.
the slim instructions are shown at the top, but the run log page grows at the bottom: you can't see both at the same time.

And you don't see the currently running instruction , which is a big minus.

Ad 1) The complexity is added in the current code. In #1469 , which uses a separate window, any added complexity is in the code for the extra window. It doesn't add complexity to existing code.

Okay... #1469 uses a Swing window. @fhoeben likes to have a browser window showing the slim instructions. Maybe I will release the "browser version" that I currently have, even if it is not quite usable because it hardcodes the browser port. That way others can take a look at it, and maybe help in adding "a separate option" for specifying the browser port .

six42 · 2024-04-17T19:32:20Z

Hi @pvbemmelen62 Paul,
thanks for taking the time and looking at the PR.
Following the fitnesse architecture might look complex. Having the reporting happening in the reporting engine I see as an advantage. It should not be done by the SUT. It has the advantage that this PR will work with any language SLIM supports.
And it add the feature of the updating tables in real time.

Just showing the finished instructions is a limitation of the current interface design and could be changed by addding one more method to the Interface: TestSystemListener.

If a test is bigger than one page length my browser is not scrolling to the bottom automatically. Is yours doing this?
So I anyway never see the current test output with Fitnesse.
Showing the new data at the top was a choise to ensure the instructions are always visible. For me it feels better than what was available so far.
It could also be appended to the bottom but as said than you are busy with scrolling all the time and can less focus on what the test is doing.

Removed the limitation that the instructions are only show after the execution. Now they are first shown with as "in progress" and after execution the actual result is added.

six42 · 2024-07-01T00:00:07Z

Removed the limitation that the instructions are only show after the execution.
Now they are first shown as "in progress" and after execution the actual result is added.

FitNesse.sbys.inProgress.2024-07-01.015051.mp4

fhoeben · 2024-08-16T16:35:10Z

@six42 I like the idea of seeing progress as its happening very much. In the past it always bugged me to have to split up a long script table in multiple to have some overview of where the execution was.

Some minor notes. I think we should require a true value for the slim.show property. That makes it easier to configure it always and just changing the value to enable/disable.

Have you checked whether the HTML output generated to file via a jUnit run is changed. I don't expect it, but just wondering.

Furthermore some unit tests and an update of the release notes would have been nice...

six42 · 2024-08-16T20:42:54Z

Hi Fried, Thanks for looking at the PR. Good remarks. The XML output stays the same with this design. Additional output is only created in the SuiteHtmlFormatter.java which is only used for interactive usage and independent from the XML Formatter. slim.show: I kept it in the tradition of other flags like 'debug', where just the presence enables the feature. I also didn't wanted to make it boolean. A use case I could imagine is that you just want to show the instruction list and not the HTML table or vice versa. Maybe leave it like this for the moment and get more feedback from more users to understand the right design. Test cases: When the feature is disabled all existing test cases pass. This is already a good coverage and result :) Switching from BULK to Single, I run all test cases once and they passed. To not double testing time I don't want to make this part of the standard regression set. In a future release we can make SINGLE rather the default mode. But better to first get feedback from more users which actually use it. For the additional output I also think it is better to get more user feedback before carving it in stone with test cases which check presence of specific HTML tags. In general I agree with you that each PR should have test cases. In this case I see the status as a disabled BETA feature which will finalise with more users providing feedback. If you have a test case in mind which you see material at this stage I will add it. Release notes: I will add if you are good to merge :)

fhoeben

I general take a look at the formatting/whitespace of the new code. I believe there are a lot of places where there is no space between if and the opening parenthesis or between ){.

fhoeben · 2024-08-16T21:20:05Z

src/fitnesse/reporting/SuiteHtmlFormatter.java


 import static fitnesse.testsystems.ExecutionResult.getExecutionResult;

-public class SuiteHtmlFormatter extends InteractiveFormatter implements Closeable {
+public class SuiteHtmlFormatter extends InteractiveFormatter implements Closeable  {


Suggested change

public class SuiteHtmlFormatter extends InteractiveFormatter implements Closeable {

public class SuiteHtmlFormatter extends InteractiveFormatter implements Closeable {

fhoeben · 2024-08-16T21:20:37Z

src/fitnesse/reporting/SuiteHtmlFormatter.java

+    String sbys = getPage().getVariable(SLIM_SHOW);
+    testSystemListenerShowHtmlTable = (sbys != null);
+    testSystemListenerShowInstructions = 0;
+    if(testSystemListenerShowHtmlTable){


Suggested change

if(testSystemListenerShowHtmlTable){

if (testSystemListenerShowHtmlTable) {

Missing whitespaces like this in many places

fhoeben · 2024-08-16T21:20:59Z

src/fitnesse/reporting/SuiteHtmlFormatter.java

+      }
+    } 
+    //Fill the list with empty lines to reserve the space on the screen
+    this.instructionsHtmlText = new LinkedList<String>(Collections.nCopies(testSystemListenerShowInstructions,""));


Suggested change

this.instructionsHtmlText = new LinkedList<String>(Collections.nCopies(testSystemListenerShowInstructions,""));

this.instructionsHtmlText = new LinkedList<String>(Collections.nCopies(testSystemListenerShowInstructions, ""));

fhoeben · 2024-08-16T21:22:54Z

src/fitnesse/reporting/SuiteHtmlFormatter.java

+  @Override
+  public void testAssertionVerified(Assertion assertion, TestResult testResult) {
+    String resultString = testResult != null ? testResult.toString() : "VOID";
+    showAssertionResult(assertion,resultString, true);


Suggested change

showAssertionResult(assertion,resultString, true);

showAssertionResult(assertion, resultString, true);

fhoeben · 2024-08-16T21:25:43Z

src/fitnesse/reporting/SuiteHtmlFormatter.java

+  }
+
+  @Override
+  public void testAssertionVerified(Assertion assertion, TestResult testResult) {


I think a unit test checking that testAssertionVerified, testExceptionOccurred and testAssertionWillBeExecuted all call showAssertionResult() (with the right parameters) would be good already. The actual HTML and JavaScript created don't have to be tested yet

fhoeben · 2024-08-16T21:42:27Z

src/fitnesse/testsystems/slim/SlimTestSystem.java

+    }
+  }
+
+  private boolean evaluateAssertion(Object InstructionResult, boolean IgnoreTestTable, SlimAssertion a, String key)


Great improvement to have this in a separate method. I think by making it protected the code using it should be a bit easier to test, and this might aid future changes.

Suggested change

private boolean evaluateAssertion(Object InstructionResult, boolean IgnoreTestTable, SlimAssertion a, String key)

protected boolean evaluateAssertion(Object instructionResult, boolean ignoreTestTable, SlimAssertion assertion, String key)

fhoeben · 2024-08-16T21:47:10Z

src/fitnesse/testsystems/slim/SlimTestSystem.java

+        stopSuiteCalled = PageData.SUITE_SETUP_NAME.equals(testContext.getPageToTest().getName());
+      }
+      if (exceptionResult.isStopSuiteException()) {
+        //IgnoreTestTable = stopTestCalled = stopSuiteCalled = true;


Suggested change

//IgnoreTestTable = stopTestCalled = stopSuiteCalled = true;

fhoeben · 2024-08-16T21:48:06Z

src/fitnesse/testsystems/slim/SlimTestSystem.java

+    if (InstructionResult != null && InstructionResult instanceof String && ((String) InstructionResult).startsWith(EXCEPTION_TAG)) {
+      SlimExceptionResult exceptionResult = new SlimExceptionResult(key, (String) InstructionResult);
+      if (exceptionResult.isStopTestException()) {
+        //IgnoreTestTable = stopTestCalled = true;


Suggested change

//IgnoreTestTable = stopTestCalled = true;

fhoeben · 2024-08-16T21:51:33Z

src/fitnesse/reporting/SuiteHtmlFormatter.java

+        if (testSystemListenerSleep > 0)
+          Thread.sleep(testSystemListenerSleep);
+      } catch (Exception e){
+        String em = e.getMessage();


This exception is ignored. I assume it is never expected to happen. Shouldn't it be logged (so it shows up in the console where the wiki was started)?

fhoeben · 2024-08-16T21:51:39Z

src/fitnesse/reporting/SuiteHtmlFormatter.java

+        String insertScript = JavascriptUtil.makeReplaceElementScript("step-by-step-Id2", html).html();
+        writeData(insertScript);
+      } catch (Exception e){
+        String html = e.getMessage();


This exception is ignored. I assume it is never expected to happen. Shouldn't it be logged (so it shows up in the console where the wiki was started)?

six42 added 4 commits March 4, 2024 18:30

Current table is expanded to see all updates

5366fe6

List of Instructions is now shown instead of just the last one in mod…

133bc09

…e slim.sbys =SHOWINSTRUCTIONS

finalized slim.show and slim.mode properties

69d61e7

Added documentation all code for the display is now in the SuiteHtmlFormatter and not in the SlimTestSystem

Show Current Running Assertion

92bc457

Removed the limitation that the instructions are only show after the execution. Now they are first shown with as "in progress" and after execution the actual result is added.

fhoeben requested changes Aug 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Watch closer what SLIM is doing #1500

Watch closer what SLIM is doing #1500

six42 commented Apr 1, 2024 •

edited

Loading

pvbemmelen62 commented Apr 4, 2024

six42 commented Apr 17, 2024

six42 commented Jul 1, 2024

fhoeben commented Aug 16, 2024

six42 commented Aug 16, 2024 via email

fhoeben left a comment

fhoeben Aug 16, 2024

fhoeben Aug 16, 2024

fhoeben Aug 16, 2024

fhoeben Aug 16, 2024

fhoeben Aug 16, 2024

fhoeben Aug 16, 2024

fhoeben Aug 16, 2024

fhoeben Aug 16, 2024

fhoeben Aug 16, 2024

fhoeben Aug 16, 2024

	public class SuiteHtmlFormatter extends InteractiveFormatter implements Closeable {
	public class SuiteHtmlFormatter extends InteractiveFormatter implements Closeable {

	if(testSystemListenerShowHtmlTable){
	if (testSystemListenerShowHtmlTable) {

	this.instructionsHtmlText = new LinkedList<String>(Collections.nCopies(testSystemListenerShowInstructions,""));
	this.instructionsHtmlText = new LinkedList<String>(Collections.nCopies(testSystemListenerShowInstructions, ""));

	showAssertionResult(assertion,resultString, true);
	showAssertionResult(assertion, resultString, true);

	private boolean evaluateAssertion(Object InstructionResult, boolean IgnoreTestTable, SlimAssertion a, String key)
	protected boolean evaluateAssertion(Object instructionResult, boolean ignoreTestTable, SlimAssertion assertion, String key)

Watch closer what SLIM is doing #1500

Are you sure you want to change the base?

Watch closer what SLIM is doing #1500

Conversation

six42 commented Apr 1, 2024 • edited Loading

Usage Guide - From the updated Wiki page

pvbemmelen62 commented Apr 4, 2024

six42 commented Apr 17, 2024

six42 commented Jul 1, 2024

fhoeben commented Aug 16, 2024

six42 commented Aug 16, 2024 via email

fhoeben left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

six42 commented Apr 1, 2024 •

edited

Loading