Updated sim mode docs (#641)

add a section on how to simulate failure

---------

Co-authored-by: Oliver Sanders <[email protected]>

src/user-guide/running-workflows/simulation-modes.rst

@@ -41,18 +41,58 @@ If :cylc:conf:`[runtime][<namespace>]execution time limit` and
 run length is computed by dividing the time limit by the speedup factor.
 
 
+Simulated Failure
+^^^^^^^^^^^^^^^^^
+
+Tasks always complete custom outputs, by default they succeed.
+
+.. warning::
+
+   In simulation mode a succeeded output may not imply that
+   submitted and/or started outputs are generated - so you will not
+   be able to test graph pathways such as ``task:started => do_something``.
+
+You can set some or all instances of a task to fail using
+:cylc:conf:`[runtime][<namespace>][simulation]fail cycle points`.
+``fail cycle points`` takes either a list of cycle point strings or "all".
+
+Tasks set to fail will succeed on their second or following simulated
+submission. If you want all submissions to fail, set
+:cylc:conf:`[runtime][<namespace>][simulation]fail try 1 only = False`.
+
+For example, to simulate a task you know to be flaky on the half
+hour but not on the hour:
+
+.. code-block:: cylc
+
+   [[get_observations]]
+      execution retry delays = PT30S
+      [[[simulation]]]
+         fail cycle points = 2022-01-01T00:30Z,  2022-01-01T01:30Z
+
+In another case you might not expect the retry to work, and want to test
+whether your failure handling works correctly:
+
+.. code-block:: cylc
+
+   [[get_data]]
+       execution retry delays = PT30S
+       [[[simulation]]]
+          fail try 1 only = false
+          fail cycle points = 2022-01-01T03:00Z
+
 Limitations
 ^^^^^^^^^^^
 
 Dummy tasks run locally, so dummy mode does not test communication with remote
 job platforms. However, it is easy to write a live-mode test workflow with
-simple ``sleep 10`` tasks that submit to a remote platform. 
+simple ``sleep 10`` tasks that submit to a remote platform.
 
 Alternate path branching is difficult to simulate effectively. You can
 configure certain tasks to fail via
 :cylc:conf:`[runtime][<namespace>][simulation]`, but all branches based
 on mutually exclusive custom outputs will run because all custom outputs get
-artificially completed in dummy mode and in simulation mode. 
+artificially completed in dummy mode and in simulation mode.
 
 .. note::