Suggested test guidelines

[SandPod]

Suggestion for test guidelines.

There doesn't seem to be any general test guidelines for dart provided by google. So I created this based on common best practices and applied them to the dart test framework.

Does this seem reasonable?
Anything missing?

[Isakdl]

Over all pretty solid :)

Some thoughts to add to the discussion:

Do we favor comments over explicitly putting the given / when / then in the test description? In the example you have here we are doing both.

This example: Cat when belly is scratched then scratches back we have both when and then but a "silent" given. How explicit should we be here? Any reason for not writing given?

What about the word should? I have had this discussion before and not 100% what I think but there is an argument for that a test expects something. I.E. then something happens, not that it should happen. Very philosophical and subtle difference 😄. But given that we advocate for the Given, When, Then, syntax it might make it more clear if we simply replaced should with then.

Example: when added multiplied should return product vs when added multiplied then return product

What do you think?

[Isakdl]

Some other things we could think about adding:

Encourage the usage of test builders to construct the objects we want to test. I have started using these for the tests I'm making for the code generator currently.

Avoid mocking as much as possible and push side effects as far up the stack as possible. As this makes the tests much less brittle and independent from the implementation.

Test file structure, where do we put the test files and what names do we give them? For example in the case of the analyzer I have many test files for a single method as the tests are several thousands lines of code. Do we want some guideline for how to organize our files?

Test scope, how much code are we testing? Personally I think unit tests that cover large amount of (pure) code is to be preferred, meaning potentially multiple classes are tested indirectly. Perhaps what @SandPod would call module tests. Unit tests that only test a single method deep inside the logic are in my experience pretty useless and gets in the way. There are of course exceptions such as smaller methods that are used in many different places or are critical, I wouldn't mind a test that explicitly verifies a regex for example.

Abstraction level of tests. I think encouraging a direct and explicit test code is in most cases a benefit. I.E. not hiding expects deep in some hidden method or obscure test setups. As a developer you should be able to more or less understand the test by only looking at a specific test in isolation.

Thoughts? :)

[SandPod]

Over all pretty solid :)

Some thoughts to add to the discussion:

Do we favor comments over explicitly putting the given / when / then in the test description? In the example you have here we are doing both.

This example: Cat when belly is scratched then scratches back we have both when and then but a "silent" given. How explicit should we be here? Any reason for not writing given?

What about the word should? I have had this discussion before and not 100% what I think but there is an argument for that a test expects something. I.E. then something happens, not that it should happen. Very philosophical and subtle difference 😄. But given that we advocate for the Given, When, Then, syntax it might make it more clear if we simply replaced should with then.

Example: when added multiplied should return product vs when added multiplied then return product

What do you think?

Good thought!

I don't think it is so important that we assert a specific phrasing, if some other word better communicates the expected outcome so that it is helpful for the future developer that reads it, then that should be fine to use. What I think is most important is that the pattern is there so that readers don't have to adjust the way they read the description. And more importantly don't have the parts either switching position or missing.

In the example with the Cat (even if that is probably not the best example) I think the parts are still clearly communicated.

But might still be good to update the examples so that they all use the same formatting, even if we don't want to lock down the phrasing.

[SandPod]

I think all of these with the exception of the last one are worth adding and with the following comment.

Test scope -I think this boils down to how many people are modifying the code at a given time. If we are few people with good knowledge, interfaces might not need as much guarding. But if we become a larger organization the behavior of interfaces might need to be properly documented using tests. Fixing what seems like a bug for one application of the code could be expected behavior in another, or worse, it might become impossible to tell. Having tests that clearly communicate the intent of the code can help there. But with that said I don't think there is a reson to encourage super strict unit tests for us right now.

Abstraction level of tests For the last one I definitely think that expects should be clear, but I wouldn't mind allowing tests to have their own helper expect method when working with either complex objects or asserting large states. I believe this is very situational and locking it down could creates bad code patterns with a lot of repetition, that then needs to be parsed by the future test reader.