using MOxUnit to run tests

[Remi-Gau]

Another issue about infra aspects:

How about switching to using MOxUnit and MoCov to run our tests and get some idea of our code coverage?

https://github.com/MOxUnit/MOxUnit
https://github.com/marketplace/actions/moxunit-action

[gllmflndn]

I might be in a minority here but I tend to use the MATLAB Testing Framework because I think it's the most future-proof option (MOxUnit is based on xUnit that precedes the 'official' tool).
It is not yet available in Octave but I would prefer investing time in implementing it there rather than using an incompatible variant. By doing so, it benefits the entire Octave ecosystem, see some starting points here, here and here for example.
That said, at the moment, only MOxUnit is compatible with MATLAB and Octave so I would understand a pragmatic argument if using Octave on Travis to run the unit tests.

[Remi-Gau]

Definitely in favor of long term solutions too.

Will have a look at the matlab testing framework but I am wondering if the cost of switching later will be THAT huge.

I think that to be cautious we might want to minimize that later cost (i.e try to not rely on some of the MOxunit functions like assertEqual).

[gllmflndn]

That might be the most pragmatic option: using MOxunit in its most similarly looking syntax to MATLAB Testing Framework so that we can easily switch later on. They have the same ancestor so they must share a lot of similarities (see e.g. assertEqual).
I do have a beginning of an implementation of the 'new' framework that works with Octave for simple unit tests. I could try and dig it.

[Remi-Gau]

Might be worth keeping an eye on that Octave implementation. :-)

[apjanke]

Hi! I'm the author of Testify, one of the Octave implementations-in-progress that gllmflndn linked to. Now that I know there's some actual potential users out there for this, I'll bump up the priority on it.

Note that the port of the Matlab Unit Testing framework won't be fully Matlab-compatible, due to language-level limitations of Octave: Octave doesn't support method attributes/annotations, so the Testify version of it will rely on naming conventions or something similar instead for the detection of test methods and properties.

[gllmflndn]

Thanks @apjanke. As mentioned above, I think it would be great to have an Octave implementation of the Matlab Unit Testing framework - if I remember correctly, this is something we discussed last year with other Octave devs on IRC.

I think having the Function-Based Unit Tests would be a good start and perhaps enough here so we wouldn't yet come across the current limitation in Octave to implement Tag Unit Tests with Class-Based Unit Tests. When reaching this limitation, the aim would then be to implement the missing feature in Octave.

[apjanke]

I think having the Function-Based Unit Tests would be a good start and perhaps enough here

For what it's worth, I plan on implementing Script, Function, and Class-Based Unit Tests all at the same time. The back end (that actually executes the tests and records results) will IMHO be a lot easier to implement using OOP, so may as well go right to Class-Based Unit Tests (to the extent that can be done in Octave) and then provide Function-Based Unit Tests as a convenience wrapper around that layer.

[nno]

Greetings, I'm the author of MOxUnit.

I'm aware that MOxUnit does not support all testing functionality provided by the Matlab testing framework. For example, currently assertEqual does not support the absTol or relTol options (though I'm looking into adding these), and not all testing functions are present. At the same time, Matlab test cases are supported through the MOxUnitMatlabUnitWrapperTestCase class.

Out of curiosity, which features do the people in this thread find lacking in MOxUnit? Which concerns are there concerning compatibility, and could these be resolved?

[Remi-Gau]

hey @nno

thanks for checking on the thread.

@gllmflndn might be able to correct me but I don't think there was any specific feature in mind that MOxUnit is lacking but more a general idea to try to stay close to the "official" tools (and help to bring the Octave version "up to speed").

What would be nice though is to see if/how tests "should" be written to maximize compatibility across frameworks. I suspect this is more a documentation issue though (Note: I have not checked if this already exists in the MOxUnit repo).

Does that make sense?

[Remi-Gau]

From a quick browse, it seems that we could minimize the cost of switching between frameworks if we write our tests for functions by:

• avoiding calling the "assert" functions that are actually specific to MOxUnit
• make sure that our test functions and sub-functions start or end with test (case insensitive)

The main "cost" I can see so far would be of switching the content of the main function for each test. See below for more details.

I will open a PR just to test a switch to MOxUnit and make sure that the bids_runtests.m that @gllmflndn is still happy with this.

For functions with MOxUnit

https://github.com/MOxUnit/MOxUnit

It is crucial that the output of the main function is a variable named test_suite, and that the output of local functions is assigned to a variable named test_functions.
Define subfunctions whose name start with test or end with test (case-insensitive).

function test_suite = exampleTest 
    try 
        test_functions = localfunctions();
    catch 
    end
    initTestSuite;
end

For functions for the matlab framework

https://fr.mathworks.com/help/matlab/matlab_prog/write-function-based-unit-tests-.html

The name of the main function corresponds to the name of your test file and should start or end with the word 'test', which is case-insensitive.

function tests = exampleTest
  tests = functiontests(localfunctions);
end

[gllmflndn]

I will be fine with whatever you decide - my concern is being future-proof and minimising time wasted refactoring unit tests in the future.
This is how the tests for jsonread look like using the MATLAB framework (well, only using verifyTrue):
https://github.com/spm/spm12/blob/master/tests/test_spm_jsonread.m
localfunctions is a good example of what I was mentioning in https://github.com/bids-standard/bids-matlab/issues/44#issuecomment-699938602https://github.com/bids-standard/bids-matlab/issues/44#issuecomment-699938602: this function was kindly added after making a request for it when trying to implement the MATLAB testing framework in Octave.

[Remi-Gau]

Thanks for the jsonread example makes it more concrete !

Okidoki. I will try to see how things compatible I can make things for now and hope that future-me won't hate me (too much). 🤞

[apjanke]

Octave Testify author here again.

MOxUnit has the distinct advantage of actually existing and working right now. I'm looking at my calendar for the next quarter, and thinking it's unlikely that Octave Testify will have a working Matlab-compatible Unit Test Framework before, say, March 2021. Even then it'll be an un-battle-tested version 1.0. I don't know who all's using MOxUnit, but it has been around a few years, long enough to have some bugs shaken out.

I would recommend that if you want Octave-compatible unit tests for BIDS-MATLAB, you just go with MOxUnit now and not worry about future-proofing. If and when an Octave Unit Test Framework becomes available and you decide to switch to it, porting your tests to it would be an easy mechanical exercise – a bit of drudgery, but no big challenge.