Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AKS: Unexpected error in logs while recording failure message #665

Open
cpinjani opened this issue Sep 10, 2024 · 3 comments
Open

AKS: Unexpected error in logs while recording failure message #665

cpinjani opened this issue Sep 10, 2024 · 3 comments
Assignees
Labels
kind/bug Something isn't working
Milestone

Comments

@cpinjani
Copy link
Contributor

Rancher version:
Rancher - v2.9-65df012014c22ba29006fca8c97aa6b8c2ba1105-head
aks-operator:v1.9.2-rc.1

Cluster Type: Downstream AKS cluster

Describe the bug:
Unexpected error in logs while recording failure message
See also: rancher/eks-operator#596

Logs:

time="2024-09-10T12:50:01Z" level=info msg="Create Azure Log Analytics Workspace \"cpinjani-aks4-EUS\" on Resource Group \"cpinjani-aks4\""
time="2024-09-10T12:53:38Z" level=error msg="Error recording akscc [ (id: )] failure message: resource name may not be empty"
time="2024-09-10T10:22:21Z" level=info msg="Importing config for cluster [cpinjani-imp (id: c-sflm4)]"
time="2024-09-10T10:22:22Z" level=info msg="Importing config for cluster [cpinjani-imp (id: c-sflm4)]"
time="2024-09-10T10:22:23Z" level=error msg="Error recording akscc [ (id: )] failure message: resource name may not be empty"
time="2024-09-10T10:22:25Z" level=error msg="Error recording akscc [ (id: )] failure message: resource name may not be empty"
time="2024-09-10T10:22:26Z" level=info msg="Checking configuration for cluster [cpinjani-imp (id: c-sflm4)]"
@cpinjani cpinjani added the kind/bug Something isn't working label Sep 10, 2024
@cpinjani cpinjani added this to the v2.9-Next1 milestone Sep 10, 2024
@cpinjani cpinjani changed the title Unexpected error in logs while recording failure message AKS: Unexpected error in logs while recording failure message Sep 10, 2024
@yiannistri yiannistri self-assigned this Sep 18, 2024
@yiannistri
Copy link
Contributor

The error originates here when a conflict occurs. When that happens, the config object returned is non-nil but empty which is why the log statement appears to be missing fields.

@thehejik
Copy link
Collaborator

thehejik commented Oct 2, 2024

Test report

The problem seems to be fixed in latest available version of AKS operator rancher-aks-operator:104.3.0+up1.9.3-rc.1 on rancher v2.9-43c67f5d91757836c8fa732be35c6171d89759c1-head.

Reproducer

• provision AKS from rancher with default values
• via Edit Config in Rancher add a NP to the AKS cluster with unsupported value for Max Pods per node = 1 (minimal allowed is 10)
• open and watch aks operator logs
• wait for RESPONSE 400: 400 Bad Request\nERROR CODE: InsufficientMaxPods
• Edit the cluster again and remove the problematic new NP and watch logs, something like this will appear:

time="2024-10-02T13:24:06Z" level=error msg="Error recording akscc [ (id: )] failure message: Operation cannot be fulfilled on aksclusterconfigs.aks.cattle.io \"c-6nz9j\": the object has been modified; please apply your changes to the latest version and try again"
time="2024-10-02T13:24:07Z" level=info msg="Checking configuration for cluster [th-aks292 (id: c-6nz9j)]"
time="2024-10-02T13:24:09Z" level=info msg="Configuration for cluster [th-aks292 (id: c-6nz9j)] was verified"

AFTER
When doing the same on fixed version the error is not present anymore:

time="2024-10-02T13:48:49Z" level=info msg="Configuration for cluster [th-aks-29head (id: c-rr2g9)] was verified"
Wed, Oct 2 2024 3:48:50 pm
time="2024-10-02T13:48:50Z" level=info msg="Checking configuration for cluster [th-aks-29head (id: c-rr2g9)]"
Wed, Oct 2 2024 3:48:51 pm
time="2024-10-02T13:48:51Z" level=info msg="Configuration for cluster [th-aks-29head (id: c-rr2g9)] was verified"

This issue can be closed.

@cpinjani
Copy link
Contributor Author

cpinjani commented Nov 7, 2024

Since the fix was reverted #684, re-opening to address this again.

@cpinjani cpinjani reopened this Nov 7, 2024
@cpinjani cpinjani modified the milestones: v2.9.3, v2.9.5 Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Development

No branches or pull requests

3 participants