Test: add reef to squid upgrade test (#461)

# Description

Add upgrade test


## Type of change

Please delete options that are not relevant.

- [x] CleanCode (Code refactor, test updates, does not introduce
functional changes)

## How Has This Been Tested?

> **_NOTE:_** All functional changes should accompany corresponding
tests (unit tests, functional tests etc).

Please describe the addition/modification of tests done to verify this
change. Please also list any relevant details for your test
configuration.

## Contributor's Checklist

Please check that you have:

- [ ] self-reviewed the code in this PR.
- [ ] added code comments, particularly in hard-to-understand areas.
- [ ] updated the user documentation with corresponding changes.
- [ ] added tests to verify effectiveness of this change.

Signed-off-by: Peter Sabaini <[email protected]>

.github/workflows/r2s-edge-upgrade.yaml

@@ -0,0 +1,64 @@
+name: Upgrade a r/stable cluster to squid/edge
+on:
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch: null
+
+jobs:
+  r2s-upgrade-test:
+    name: Test reef/stable to squid/edge upgrades
+    runs-on: ubuntu-22.04
+    steps:
+
+    - name: Checkout code
+      uses: actions/checkout@v3
+      with:
+        fetch-depth: 0
+
+    - name: Copy utils
+      run: cp tests/scripts/actionutils.sh $HOME
+
+    - name: Clear FORWARD firewall rules
+      run: ~/actionutils.sh cleaript
+
+    - name: Free disk
+      run: ~/actionutils.sh free_runner_disk
+
+    - name: Install dependencies
+      run: ~/actionutils.sh setup_lxd
+
+    - name: Create containers with loopback devices
+      run: ~/actionutils.sh create_containers public
+
+    - name: Install reef stable from store
+      run: ~/actionutils.sh install_store reef/stable
+
+    - name: Bootstrap
+      run: ~/actionutils.sh bootstrap_head
+
+    - name: Setup cluster
+      run: ~/actionutils.sh cluster_nodes
+
+    - name: Add 3 OSDs
+      run: |
+        for c in node-wrk0 node-wrk1 node-wrk2 ; do
+          ~/actionutils.sh add_osd_to_node $c
+        done
+        ~/actionutils.sh headexec wait_for_osds 3
+
+    - name: Enable RGW
+      run: ~/actionutils.sh headexec enable_rgw
+
+    - name: Exercise RGW
+      run: ~/actionutils.sh headexec testrgw
+
+    - name: Upgrade to edge
+      run: ~/actionutils.sh refresh_snap squid/edge
+
+    - name: Wait until 3 OSDs are up
+      run:  ~/actionutils.sh headexec wait_for_osds 3
+
+    - name: Verify config
+      run: ~/actionutils.sh test_ceph_conf
+
+    - name: Exercise RGW again
+      run: ~/actionutils.sh headexec testrgw