JOSS Review: No test suite

[trappitsch]

Testing is an integral part of software and a requirements for JOSS submission, see here. While I saw that pytest is listed as a requirements, it can't find any tests and neither did I looking manually. I might have missed it, could you clarify?

Details on the test suite that needs to pass should also go into the documentation.

[silkemaes]

It is true that we haven't written automated tests for MACE. I know JOSS strongly encourages to write tests, however since I am not experienced with automated testing, currently we have not provided these tests.

Do you want me to provide some tests? Then I will look into it.

[trappitsch]

Theses don't have to be automated, from the guidelines:

Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?

I saw that you have a test.py, which is a test case, however, I'm unclear if you'd consider this a test for the user to verify things work and what I would compare too. Would I compare to the annotated notebook? I think a bit of elaboration on this in the docs would be great such that the user can double check and run an example, compare it to what you have provided, and make sure it runs as intended. Hope this makes sense.

[silkemaes]

Hi @trappitsch, sorry I completely missed your reply on this issue.

First my best wishes to you for 2025!

The test.py script is not a testing script to verify whether things work or not, but it is part of the training. This is stated at the beginning of the script as follows:

This script contains functions to apply a trained MACE model on a test dataset, 
i.e., test the trained model.

    - Test the model on 1 consecutive time step or on the evolution of the system.
    - Test the model on a full evolution of the system.

There are no tests (yet) to verify everything works correctly. As I understood correctly from the documentation online for JOSS submissions, tests are advised to include, but not obligatory.

Moreover, the code MACE provides is rather a method than a tool, and hence a test suite is less applicable here in our opinion.

For now, is it possible to proceed with the review without the tests?

If not, If you do require us to provide a test suite, we will try to do so.