Converting documentation examples to tests

The documentation contains lots of code examples embedded in the text. Some of them have bugs or are incorrect in various ways, either because they were never correct (having been typed in for the docs themselves and never directly tested), or were once correct but now the APIs have changed and the example was never updated to match.

To combat this, we would like to go through all the documentation, and to the greatest extent possible, convert every significant code example in the docs into a test -- that is, into actual code that will compile and run every time we run the testsuite -- and then the documentation will merely include the test code by reference, so they can never get out of sync again.

We prototyped this approach in this PR, which you should read, but the quick summary if you want to try a conversion yourself is:

1. Find a code example in the documentation that you wish to convert.

   PR 3977 used the first C++ and first Python code examples in the ImageOutput chapter. You can find the documentation source corresponding to that chapter in src/doc/imageoutput.rst.

2. You'll see that the testsuite has docs-examples-cpp and docs-examples-python directories to house the documentation code examples for C++ and Python, respectively.

   For C++, within that directory you'll see src/docs-examples-imageoutput.cpp file containing the C++ itself. And for Python, src//docs-examples-imageoutput.py.

   This is where you will add the code that was previously in the example. There will be additional .cpp and .py files for each chapter, in addition to the imageoutput one. (If there are not, I guess you'll have to add them!)

3. In the right .cpp or .py file for the chapter you're taking an example from, paste the code. The code in the examples currently are just "code fragments", but you'll need to turn it into a function in the C++ or Python test code.

   Since many examples live in each .cpp or .py file, you can mark the beginning and ending of the part you wish to clip with these special marker comments:

   // BEGIN-imageoutput-simple
   // END-imageoutput-simple
   

   That's just an example of one already used. Pick your own name, but make of the form BEGIN-<chapter>-<example_name>. Use the appropriate comment markers for the language, // for C++ and # for Python. If the example is best illustrated in the docs with only a few lines instead of a whole function, you can have those begin/end marker comments in the middle of a function, too.

   Don't forget to also add a call to the function in the "main", so it's actually executed when the test runs.

4. Back in the docs file (such as src/doc/imageoutput.rst), replace the code section you removed with a reference to the code in the testsuite. For example,

   .. literalinclude:: ../../testsuite/docs-examples-cpp/src/docs-examples-imageoutput.cpp
       :language: c++
       :start-after: BEGIN-imageoutput-simple
       :end-before: END-imageoutput-simple
   

   Notice how we use the Sphinx literalinclude and the modifiers :start-after: and :end-before: in conjunction with the bracketing comments we mentioned above, to clip only the desired lines from the example.

5. That should be enough to cause the test to run. Additionally, in the testsuite entry, you'll want to modify the test script to verify that your new test behaved correctly. If you are producing an output image, you'll want to make sure writes a uniquely named image for your test, and add that name to the list of outputs it's checking, for example in the line at the end of docs-examples-cpp/run.py,

   outputs = [ "simple.tif", "scanlines.tif",
               "out.txt" ]
   

   Here, simple.tif and scanlines.tif are outputs from examples. You'll want to add another (after scanlines.tif).

   You'll also need to add a reference image of the correct output to the testsuite/docs-examples-cpp/ref/ subdirectory.

6. Now run the testsuite. On Linux or Max, the makefile wrapper is really handy here. From the top level,

   make all test TEST=docs-examples
   

7. The test will probably fail, and that's ok, because you haven't finished updating the test yet. Try this:

   cd build/testsuite/docs-examples-cpp
   xxdiff out.txt ref            # or the comparison program of your choice
   

   Make sure it all looks ok, then you can copy out.txt to ref to update the correct text output, which includes the report of how the image comparisons did.

If you get stuck, just ask on the mail list or Slack! Once you've run through this once, you'll get the hang of it and it'll be easy.