Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: disk pressure flakes #2832

Merged
merged 16 commits into from
Aug 5, 2024
8 changes: 7 additions & 1 deletion .github/actions/cleanup-files/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,16 @@ runs:
- run: |
lsblk -f

echo "removing some github actions pre-installed tools to save space"
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo docker system prune --all --force

echo "removing zarf sboms, packages, cache"
sudo rm -rf zarf-sbom /tmp/zarf-*
sudo env "PATH=$PATH" CI=true make delete-packages
sudo build/zarf tools clear-cache
sudo docker system prune --all --force

lsblk -f
shell: bash
22 changes: 22 additions & 0 deletions .github/actions/debug-cluster/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
name: debug-cluster
description: "Setup Go binary and caching"

runs:
using: composite
steps:
- run: |
echo "***** Getting pods *****"
kubectl get pods -A

echo "***** Getting pods yaml *****"
kubectl get pods -A -o yaml

echo "***** Describing pods *****"
kubectl describe pods -A

echo "***** Getting nodes *****"
kubectl get nodes -A

echo "***** describing nodes *****"
kubectl describe nodes -A
shell: bash
16 changes: 2 additions & 14 deletions .github/workflows/nightly-eks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,21 +57,9 @@ jobs:
- name: Run tests
run: make test-e2e-with-cluster ARCH=amd64

- name: get pods
- name: show cluster logs
uses: ./.github/actions/debug-cluster
if: always()
run: kubectl get pods -n kiwix -o yaml

- name: describe pod
if: always()
run: kubectl describe pods -n kiwix

- name: get nodes
if: always()
run: kubectl get nodes -o yaml

- name: describe nodes
if: always()
run: kubectl describe nodes

- name: Teardown the cluster
if: always()
Expand Down
12 changes: 12 additions & 0 deletions .github/workflows/test-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,10 @@ jobs:
run: |
make test-e2e-with-cluster ARCH=amd64

- name: get cluster info
uses: ./.github/actions/debug-cluster
if: always()

- name: Save logs
if: always()
uses: ./.github/actions/save-logs
Expand Down Expand Up @@ -197,6 +201,10 @@ jobs:
run: |
make test-e2e-with-cluster ARCH=amd64

- name: get cluster info
uses: ./.github/actions/debug-cluster
if: always()

- name: Save logs
if: always()
uses: ./.github/actions/save-logs
Expand Down Expand Up @@ -235,6 +243,10 @@ jobs:
run: |
make test-e2e-with-cluster ARCH=amd64

- name: get cluster info
uses: ./.github/actions/debug-cluster
if: always()

- name: Save logs
if: always()
uses: ./.github/actions/save-logs
Expand Down
7 changes: 7 additions & 0 deletions .github/workflows/test-external.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,16 @@ jobs:
- name: Setup k3d
uses: ./.github/actions/k3d

- name: Cleanup files
uses: ./.github/actions/cleanup-files

- name: Run external service test
run: make test-external

- name: get cluster info
uses: ./.github/actions/debug-cluster
if: always()

- name: Save logs
if: always()
uses: ./.github/actions/save-logs