extend documentation of testing for examples (#812)

* reformat

* add more context, document monitoring

internal/readme.md

@@ -1,20 +1,63 @@
 ## `Internal/`
 
-This is internal repository and documentation management code. It does not contain examples.
+This is internal repository and documentation management code. It does not
+contain examples.
 
-### Frontmatter
+### Continuous Integration and Continuous Deployment
 
-Examples have frontmatter that control testing and deployment behavior. Fields:
+Modal cares deeply about the correctness of our examples -- we have also
+suffered from janky, poorly-maintained documentation and we do our best to
+ensure that our examples don't pay that forward.
 
-- `deploy`: If `true`, the example is deployed to the website with `modal deploy`. If `false`, it is not. Default is `false`.
-- `cmd`: The command to run the example for testing. Default is `["modal", "run", "<filename>"]`.
+This document explains the CI/CD process we use. It is primarily intended for
+Modal engineers, but if you're contributing an example and have the bandwidth to
+set up the testing as well, we appreciate it!
+
+#### Frontmatter
+
+Examples can include a small frontmatter block in YAML format that controls
+testing and deployment behavior.
+
+Fields:
+
+- `deploy`: If `true`, the example is deployed as a Modal application with
+  `modal deploy`. If `false`, it is not. Default is `false`.
+- `cmd`: The command to run the example for testing. Default is
+  `["modal", "run", "<filename>"]`.
 - `args`: Arguments to pass to the command. Default is `[]`.
-- `lambda-test`: If `true`, the example is tested with the cli command provided in `cmd`. If `false`, it is not. Default is `true`.
-- `runtimes`: Control which runtimes the example is executed on in synthetic monitoring. Default is `["runc", "gvisor"]`.
+- `lambda-test`: If `true`, the example is tested with the cli command provided
+  in `cmd`. If `false`, it is not. Default is `true`. Note that this controls
+  execution in the CI/CD of this repo and in the mointor-based testing.
+- `runtimes`: Control which runtimes the example is executed on in synthetic
+  monitoring. Default is `["runc", "gvisor"]`.
 
-Example for a web app. Note that here we `modal serve` in the test so as to not deploy to prod when testing.
+Below is an example frontmatter for deploying a web app. Note that here we
+`modal serve` in the test so as to not deploy to prod when testing. Note that in
+testing environments, the `MODAL_SERVE_TIMEOUT` environment variable is set so
+that the command terminates.
 
 ```yaml
+---
 deploy: true
 cmd: ["modal", "serve", "10_integrations/pushgateway.py"]
+---
+# example prose and code begins here
 ```
+
+#### Testing in GitHub Actions
+
+When a PR is opened, any changed examples are run via GitHub Actions.
+
+This workflow is intended to catch errors at the time a PR is made -- incuding
+both errors in the example and issues with the execution of the example in the
+monitoring system.
+
+This component is new as of early summer 2024, so there may be rough edges.
+Issue reports and fixes especially welcome!
+
+#### Continual Monitoring
+
+Examples are executed regularly and at random to check for regressions. The
+results are monitored.
+
+Modal engineers, see `synthetic_monitoring` in the `modal` repo for details.