Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OCPBUGS-41892: Single Watch on GWAPI CRD #1165

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Thealisyed
Copy link

A single CRD watch was added with a predicate to filter CRDs based on the gatewayapiv1.GroupName. This ensures that only relevant CRDs are watched, reducing unnecessary logging when the Gateway API feature is not enabled.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 4, 2024
@openshift-ci-robot openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Nov 4, 2024
@openshift-ci-robot
Copy link
Contributor

@Thealisyed: This pull request references Jira Issue OCPBUGS-41892, which is invalid:

  • expected the bug to target the "4.18.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

A single CRD watch was added with a predicate to filter CRDs based on the gatewayapiv1.GroupName. This ensures that only relevant CRDs are watched, reducing unnecessary logging when the Gateway API feature is not enabled.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested review from Miciah and miheer November 4, 2024 13:48
@Thealisyed
Copy link
Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Nov 4, 2024
@openshift-ci-robot
Copy link
Contributor

@Thealisyed: This pull request references Jira Issue OCPBUGS-41892, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.18.0) matches configured target version for branch (4.18.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @lihongan

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested a review from lihongan November 4, 2024 13:48
@Thealisyed
Copy link
Author

/test e2e-aws-ovn-single-node

@Thealisyed
Copy link
Author

/retest-required

@Thealisyed Thealisyed changed the title [WIP] OCPBUGS-41892: Single Watch on GWAPI CRD OCPBUGS-41892: Single Watch on GWAPI CRD Nov 6, 2024
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 6, 2024
Copy link
Contributor

@Miciah Miciah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Whoops, I've had this review pending, waiting for me to submit it, for 5 days...)

Is it true that the existing watch only generates reconciliation requests when the Gateway API feature is not enabled?

In any case, please add a little more detail in the commit message. For example:

gatewayapi: Use single watch for CRDs

Replace the multiple watches for Gateway API CRDs with a single CRD watch, with
a predicate to filter CRDs based on the gatewayapiv1.GroupName.  This ensures
that only relevant CRDs are watched, reducing unnecessary reconciliation
requests and log messages.

This commit fixes OCPBUGS-41892.

https://issues.redhat.com/browse/OCPBUGS-41892

pkg/operator/controller/gatewayapi/controller.go Outdated Show resolved Hide resolved
pkg/operator/controller/gatewayapi/controller.go Outdated Show resolved Hide resolved
@Miciah Miciah added the priority/backlog Higher priority than priority/awaiting-more-evidence. label Nov 13, 2024
@candita
Copy link
Contributor

candita commented Nov 13, 2024

/assign
/assign @Miciah

@candita
Copy link
Contributor

candita commented Nov 13, 2024

@Thealisyed please investigate the failures in the CI tests, especially e2e-aws-gatewayapi, which is the GWAPI test. I ran a retest to see if the issue with imagestreams had gone away.

@candita
Copy link
Contributor

candita commented Nov 13, 2024

/retest

@candita
Copy link
Contributor

candita commented Nov 25, 2024

@Thealisyed please note you will need to squash your three commits before merge.

@candita
Copy link
Contributor

candita commented Nov 25, 2024

Process did not finish before 4h0m0s timeout

/test e2e-aws-ovn-serial

@Miciah
Copy link
Contributor

Miciah commented Nov 27, 2024

/approve

The changes look fine to me. I did have a suggestion for the commit message: #1165 (review). We generally follow the upstream conventions: https://www.kubernetes.dev/docs/guide/pull-requests/#commit-message-guidelines

Copy link
Contributor

openshift-ci bot commented Nov 27, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Miciah

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 27, 2024
@candita
Copy link
Contributor

candita commented Nov 27, 2024

/test e2e-aws-ovn-techpreview

@Thealisyed
Copy link
Author

@Miciah Apologies for the confusion! I initially misunderstood your comment and thought you wanted a longer commit message. I now see you're suggesting adding more detail, like in the example you shared. I'll update the commit message to follow the upstream guidelines. Thanks for the clarification!

Replace the multiple watches for Gateway API CRDs with a single CRD watch, with
a predicate to filter CRDs based on the gatewayapiv1.GroupName.  This ensures
that only relevant CRDs are watched, reducing unnecessary reconciliation
requests and log messages.

This commit fixes OCPBUGS-41892.

https://issues.redhat.com/browse/OCPBUGS-41892
@Thealisyed
Copy link
Author

/retest-required

1 similar comment
@Thealisyed
Copy link
Author

/retest-required

@Thealisyed
Copy link
Author

/test e2e-aws-operator

@candita
Copy link
Contributor

candita commented Dec 9, 2024

@Thealisyed - before you test again, do a little analysis on the failures, and write them up like this. Then the next time they fail, if this failure keeps happening, you'll need to figure or get help to figure out why.

Two unit test failures:

TestAll/serial/TestRouterCompressionOperation
router_compression_test.go:245: failed to create service openshift-ingress/hello-pod-bpcnc: Internal error occurred: failed to allocate a serviceIP: rpc error: code = Unavailable desc = error reading from server: read tcp 10.0.89.66:43566->10.0.18.210:2379: read: connection reset by peer

TestAll/serial/TestCanaryWithMTLS
canary_test.go:343: Polling for canary success condition...
canary_test.go:346: Failed to get IngressController: Get "https://api.ci-op-ki1llnl8-9e7c5.origin-ci-int-aws.dev.rhcloud.com:6443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default": http2: client connection lost
canary_test.go:361: Polling terminated with unexpected error: Get "https://api.ci-op-ki1llnl8-9e7c5.origin-ci-int-aws.dev.rhcloud.com:6443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default": http2: client connection lost

/test e2e-aws-operator-techpreview

@Thealisyed
Copy link
Author

"Operator "console-operator\ + cluster-samples-operator" produces more watch requests than expected [for both]
seems unrelated to changes will test again and observe

/retest

Copy link
Contributor

openshift-ci bot commented Dec 12, 2024

@Thealisyed: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@candita
Copy link
Contributor

candita commented Dec 19, 2024

@Thealisyed, in the e2e-aws-gatewayapi test logs, where the featuregate is on, I'm seeing some errors that should be investigated: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1165/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-gatewayapi/1862083335728140288/artifacts/e2e-aws-gatewayapi/gather-extra/artifacts/pods/openshift-ingress-operator_ingress-operator-58d85999bb-l6cpv_ingress-operator.log

Can you check if this happens on a cluster without your changes? If so, please open a new bug that points to the logs. If not, we need to find out what's in this PR that would cause the errors.

Specifically, lots of these:

2024-11-28T11:51:57.292Z ERROR operator.init controller/controller.go:263 Reconciler error {"controller": "gatewayclass_controller", "object": {"name":"openshift-default"}, "namespace": "", "name": "openshift-default", "reconcileID": "e18f9589-f527-4802-87aa-bc6ebb8240fd", "error": "failed to create ServiceMeshControlPlane openshift-ingress/openshift-gateway: Internal error occurred: failed calling webhook "smcp.mutation.maistra.io": failed to call webhook: Post "https://istio-operator-service.openshift-operators.svc:443/mutate-smcp?timeout=10s\": no endpoints available for service "istio-operator-service"", "errorCauses": [{"error": "failed to create ServiceMeshControlPlane openshift-ingress/openshift-gateway: Internal error occurred: failed calling webhook "smcp.mutation.maistra.io": failed to call webhook: Post "https://istio-operator-service.openshift-operators.svc:443/mutate-smcp?timeout=10s": no endpoints available for service "istio-operator-service""}]}

and then eventually it is able to create the servicemeshcontrolplane:

2024-11-28T11:52:17.801Z INFO operator.gatewayclass_controller gatewayclass/servicemeshcontrolplane.go:46 created ServiceMeshControlPlane {"namespace": "openshift-ingress", "name": "openshift-gateway"}

@candita
Copy link
Contributor

candita commented Dec 19, 2024

Hold for a check on logs
/hold
Otherwise
/lgtm

@openshift-ci openshift-ci bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm Indicates that a PR is ready to be merged. labels Dec 19, 2024
@rhamini3
Copy link

rhamini3 commented Dec 19, 2024

Pre merge verified, Gateway logs does not flood the cluster-ingress-operator logs

% oc logs ingress-operator-747dd58b69-xwpsk -n openshift-ingress-operator | grep gateway
Defaulted container "ingress-operator" out of: ingress-operator, kube-rbac-proxy
2024-12-18T17:00:21.604Z	INFO	operator.init	controller/controller.go:231	Starting EventSource	{"controller": "gatewayapi_controller", "source": "kind source: *v1.FeatureGate"}
2024-12-18T17:00:21.605Z	INFO	operator.init	controller/controller.go:231	Starting EventSource	{"controller": "gatewayapi_controller", "source": "kind source: *v1.CustomResourceDefinition"}
2024-12-18T17:00:21.605Z	INFO	operator.init	controller/controller.go:231	Starting Controller	{"controller": "gatewayapi_controller"}
2024-12-18T17:00:22.130Z	INFO	operator.init	controller/controller.go:231	Starting workers	{"controller": "gatewayapi_controller", "worker count": 1}
2024-12-18T17:00:22.130Z	INFO	operator.gatewayapi_controller	controller/controller.go:116	reconciling	{"request": {"name":"cluster"}}

/label qe-approved

@openshift-ci openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Dec 19, 2024
@openshift-ci-robot openshift-ci-robot added jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. and removed jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Dec 19, 2024
@openshift-ci-robot
Copy link
Contributor

@Thealisyed: This pull request references Jira Issue OCPBUGS-41892, which is invalid:

  • expected the bug to target either version "4.19." or "openshift-4.19.", but it targets "4.18.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

A single CRD watch was added with a predicate to filter CRDs based on the gatewayapiv1.GroupName. This ensures that only relevant CRDs are watched, reducing unnecessary logging when the Gateway API feature is not enabled.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@Miciah
Copy link
Contributor

Miciah commented Dec 20, 2024

/jira refresh

@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Dec 20, 2024
@openshift-ci-robot
Copy link
Contributor

@Miciah: This pull request references Jira Issue OCPBUGS-41892, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.19.0) matches configured target version for branch (4.19.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira ([email protected]), skipping review request.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Dec 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. priority/backlog Higher priority than priority/awaiting-more-evidence. qe-approved Signifies that QE has signed off on this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants