Taplint should exit with a non-zero exit code if errors are detected

[esabol]

We would like to use taplint in a CI/CD context to ensure there are no regressions in our web application's TAP implementation. As such, it would be helpful if taplint exited with a non-zero exit code if any errors or failures are encountered. It would make implementing taplint as a CI/CD test a lot easier. Would that be possible? Otherwise, I think we have to egrep the output for something like "^Totals: Errors(:.*; *Failures)?: *[1-9]" (Or what do you suggest?)

[mbtaylor]

Hi @esabol. I don't think I'm going to make that change; the range of issues that taplint can detect is quite wide, in some cases it's debatable what Errors are tolerable or fixable and what Warnings should be avoided, so I'd say the primary output from taplint is as a human-readable document rather than a simple yes/no answer. Of course that doesn't stop you using it that way in a CI/CD environment. If you want to do that then yes I'd suggest wrapping it in grep.

The stdout reports are specifically designed to be grep-friendly, so if you want to detect Errors you can grep "^E-". That approach will also give you the opportunity to exclude won't-fix errors or include specific Warnings etc if you want to (I suggest that you get a human to read the output in detail at least once before you plumb it into the CI/CD sausage machine). If you want to reduce the output for feeding to grep you can supply parameters like report=E maxrepeat=1 truncate=80. If you prefer to parse JSON than grep plain text that's also an option (format=json).

One other note: Failures are not necessarily a bad thing (possibly Failure wasn't the best term to use here); they mean that the validator couldn't carry out a particular test for some reason or other, but that reason may not be something wrong with the service; e.g. it's trying to validate the ObsCore, but the service is not an ObsCore service.

If you have any other questions please feel free to raise other issues or contact me by email.

[esabol]

I feel like you need to have a human read the output in order to get to a zero errors state in the first place, so that's a given.

I really don't see the downside to exiting with a non-zero exit code if errors are detected, but, if you didn't find our use-case persuasive, I'll close the issue and we'll just resort to using grep. I just thought it would be an easy enhancement and one that others would also find useful, but OK.

Thanks for the suggestions to use report=E, to grep for "^E-", and to not worry about failures. All solid advice. Cheers.