From 6ae9e790c2817e66b65f07fb2ff291b260d89503 Mon Sep 17 00:00:00 2001 From: Charles Frye Date: Thu, 11 Jul 2024 13:20:24 -0700 Subject: [PATCH] extend documentation of testing for examples (#812) * reformat * add more context, document monitoring --- internal/readme.md | 59 +++++++++++++++++++++++++++++++++++++++------- 1 file changed, 51 insertions(+), 8 deletions(-) diff --git a/internal/readme.md b/internal/readme.md index 3f690da0d..fb03c9cde 100644 --- a/internal/readme.md +++ b/internal/readme.md @@ -1,20 +1,63 @@ ## `Internal/` -This is internal repository and documentation management code. It does not contain examples. +This is internal repository and documentation management code. It does not +contain examples. -### Frontmatter +### Continuous Integration and Continuous Deployment -Examples have frontmatter that control testing and deployment behavior. Fields: +Modal cares deeply about the correctness of our examples -- we have also +suffered from janky, poorly-maintained documentation and we do our best to +ensure that our examples don't pay that forward. -- `deploy`: If `true`, the example is deployed to the website with `modal deploy`. If `false`, it is not. Default is `false`. -- `cmd`: The command to run the example for testing. Default is `["modal", "run", ""]`. +This document explains the CI/CD process we use. It is primarily intended for +Modal engineers, but if you're contributing an example and have the bandwidth to +set up the testing as well, we appreciate it! + +#### Frontmatter + +Examples can include a small frontmatter block in YAML format that controls +testing and deployment behavior. + +Fields: + +- `deploy`: If `true`, the example is deployed as a Modal application with + `modal deploy`. If `false`, it is not. Default is `false`. +- `cmd`: The command to run the example for testing. Default is + `["modal", "run", ""]`. - `args`: Arguments to pass to the command. Default is `[]`. -- `lambda-test`: If `true`, the example is tested with the cli command provided in `cmd`. If `false`, it is not. Default is `true`. -- `runtimes`: Control which runtimes the example is executed on in synthetic monitoring. Default is `["runc", "gvisor"]`. +- `lambda-test`: If `true`, the example is tested with the cli command provided + in `cmd`. If `false`, it is not. Default is `true`. Note that this controls + execution in the CI/CD of this repo and in the mointor-based testing. +- `runtimes`: Control which runtimes the example is executed on in synthetic + monitoring. Default is `["runc", "gvisor"]`. -Example for a web app. Note that here we `modal serve` in the test so as to not deploy to prod when testing. +Below is an example frontmatter for deploying a web app. Note that here we +`modal serve` in the test so as to not deploy to prod when testing. Note that in +testing environments, the `MODAL_SERVE_TIMEOUT` environment variable is set so +that the command terminates. ```yaml +--- deploy: true cmd: ["modal", "serve", "10_integrations/pushgateway.py"] +--- +# example prose and code begins here ``` + +#### Testing in GitHub Actions + +When a PR is opened, any changed examples are run via GitHub Actions. + +This workflow is intended to catch errors at the time a PR is made -- incuding +both errors in the example and issues with the execution of the example in the +monitoring system. + +This component is new as of early summer 2024, so there may be rough edges. +Issue reports and fixes especially welcome! + +#### Continual Monitoring + +Examples are executed regularly and at random to check for regressions. The +results are monitored. + +Modal engineers, see `synthetic_monitoring` in the `modal` repo for details.