Exposing HAProxy behind a service LoadBalancer instead of Ingress

[alec-drw]

With the change from Nginx to HAProxy, I am curious as to why there is no support to bypass the ingress object and instead define the proxy Service object as type LoadBalancer and handle TLS termination from the HAProxy deployment. Notably, the current default values.yaml still set an ingress class name of nginx, so there is still a default ingress-nginx controller handling TLS.

Is this something that was excluded explicitly, or just not taken into consideration?

[nefelim4ag]

TLDR:
Because there is no need to bypass.

Long:
"Proxy" used only just because they are supported: https://www.jetbrains.com/help/teamcity/multinode-setup.html#Proxy+Configuration
They are used separately from Ingress because Ingress is not supposed to handle such custom load balancing stuff, like separate users/agents and route them by custom Headers/Cookie keys.

It is stupid to try to implement the same logic on every ingress (not even possible).

Nginx is just bad and obsolete (all fixed behavior exists in Nginx Plus) - I replaced it with Haproxy.
HaProxy is also obsolete to some degree but it is just better in ways where it is needed - stable health check and stable switching between nodes.
Envoy will be perfect, but let's stick to officially supported things.

So we need "proxy" layer as it is just simpler. We can't move it to Ingress.

About Load Balancers:
It is possible to further customize the "proxy", to make it support a load balancer. But LB by itself is badly defined, so we will end up in a situation where someone needs "advanced logging" or TLS support on the "proxy" side - and it is not cool to reimplement ingress again.

So all things are separated:
Ingress will handle incoming traffic and I don't care what sort of ingress is used.
"Proxy" implements support for TeamCity internals and "advanced" load balancing capabilities.

I believe this will explain current choices.

[alec-drw]

I see. A bit of a follow up with the current config:

On 2021.2 round robin is not supported - we are planning on using the HAProxy setup but disabling RR until we upgrade to 2023.05. Right now when I hit the main URL I see that I am redirected to the secondary node that is read-only. Should I not list it under proxy.ingress.hosts[] in order to ensure UI requests are not redirected there? Here is my current patch:

    proxy:
      ingress:
        hosts:
          - builds-uat.devx.kube
          - builds-member-0-uat.devx.kube
          - builds-member-1-uat.devx.kube
    teamcity:
      nodes: 
      - ingress:
          host: builds-uat.devx.kube
      - ingress:
          host: builds-member-0-uat.devx.kube
      - ingress:
          host: builds-member-1-uat.devx.kube

Although notably, with the new changes the teamcity.nodes.ingress.host value is not read:

As you can see hitting the main URL sends me to a read only node:

[nefelim4ag]

I'm not sure, but I think you must use different URLs for the Main ingress and the per-node ingress.
Per node ingress intended for direct access.

Example from my setup, from TF:

proxy:
  ingress:
    ingressClassName: contour
    hosts:
      - ${var.hostname}.${var.dns_domain}
      - ${var.hostname}.isolated.${var.dns_domain}|
      
teamcity:
  nodes:
  - ingress:
      host: ${var.hostname}.main.${var.dns_domain}

In your case, you can check the ingress object, and realize that because of duplicated host names ingress can not stably route you.

[nefelim4ag]

This can also be a problem with haproxy setup,
Because I used a recent one for 2023 and it slightly differs in detail from older ones <=2022.

Maybe I need to set some checks to alter the configuration for older setups like yours, I just did not expect that someone would stick to older versions =\

[alec-drw]

Thankfully part of this project is to get it up to 2023, so I will be making some changes to the HAProxy config to disable RR until we get there; this is still a better approach than what we are currently working with. Made some changes to teamcity.nodes and things are looking more so how I expected. Thanks for the quick response!