Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix traefik lb being constantly re-created on controller restart #104

Merged
merged 8 commits into from
Jul 10, 2024
Merged

Fix traefik lb being constantly re-created on controller restart #104

merged 8 commits into from
Jul 10, 2024

Conversation

IbraAoad
Copy link
Contributor

@IbraAoad IbraAoad commented Jul 9, 2024 •
edited
Loading

Issue

Traefik k8s service is recreated on each controller restart, this causes the k8s api to request a new external IP for the LB when a pool of IPs exist

Solution

Call the _remove_service only when the on_remove hook is observed instead of calling it on each stop operation, also adjust the upgrade logic to delete any previous LB if service type is changed to ClusterIP

Context

From Managed Solutions

I am facing some issues with Traefik on top of OpenStack cloud. I see traefik now create an additional service called traefik-lb correctly b/c of #319 but any juju controller restart causes it to be recreated, which subsequently triggers a creation of a new LB in OpenStack Octavia and my LB Floating IP changes, but i have a lot of configuration around that LB (FIP, DNS, lxd loki addres and firewall) that makes COS unusable

Testing Instructions

  1. juju deploy traefik
  2. juju deploy catalogue
  3. juju relate traefik catalogue
  4. kubectl delete pod controller-0 -n controller-k8s (wait until it's completely deleted and comes back up again)
  5. kubectl get svc -A -o wide (observe that the traefik lb isn't restarted)

Tandem PR: canonical/traefik-k8s-operator#381

@IbraAoad IbraAoad requested a review from a team as a code owner July 9, 2024 20:22
@IbraAoad IbraAoad mentioned this pull request Jul 9, 2024
Merged
sed-i
sed-i reviewed Jul 9, 2024
View reviewed changes
lib/charms/observability_libs/v1/kubernetes_service_patch.py Show resolved Hide resolved
lib/charms/observability_libs/v1/kubernetes_service_patch.py Outdated Show resolved Hide resolved
lib/charms/observability_libs/v1/kubernetes_service_patch.py Outdated Show resolved Hide resolved
lib/charms/observability_libs/v1/kubernetes_service_patch.py Outdated Show resolved Hide resolved
lib/charms/observability_libs/v1/kubernetes_service_patch.py Outdated Show resolved Hide resolved
lib/charms/observability_libs/v1/kubernetes_service_patch.py Show resolved Hide resolved
lib/charms/observability_libs/v1/kubernetes_service_patch.py Outdated Show resolved Hide resolved
lib/charms/observability_libs/v1/kubernetes_service_patch.py Outdated Show resolved Hide resolved
@sed-i sed-i merged commit b2c5820 into main Jul 10, 2024
13 checks passed
@sed-i sed-i deleted the k8s_svc_patch branch July 10, 2024 20:32
hemanthnakkina added a commit to hemanthnakkina/snap-openstack that referenced this pull request Aug 30, 2024
@hemanthnakkina
Bump traefik-k8s channel to latest/beta
a175ff0 
Loadbalancer IPs on traefik services are
not reatined on restart. This is fixed in
traefik-k8s latest/beta [1][2].
Bump the traefik-k8s default channel to
latest/beta

[1] canonical/traefik-k8s-operator#381
[2] canonical/observability-libs#104
hemanthnakkina added a commit to hemanthnakkina/snap-openstack that referenced this pull request Aug 30, 2024
@hemanthnakkina
Bump traefik-k8s channel to latest/beta
18b3554 
Loadbalancer IPs on traefik services are
not retained on restart. This is fixed in
traefik-k8s latest/beta [1][2].
Bump the traefik-k8s default channel to
latest/beta

[1] canonical/traefik-k8s-operator#381
[2] canonical/observability-libs#104
@hemanthnakkina hemanthnakkina mentioned this pull request Aug 30, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sed-i sed-i sed-i approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@IbraAoad @sed-i