Self-Hosting FeatureHub: Liveness/Readiness Probes Always Fail for Management repository

[thedanchez]

Hello, I am trying to self host FeatureHub in my org's K8s cluster and cannot get the Management Repository pods to successfully start and be ready. The liveness/readiness probes fail all the time because the Pod cannot seem to connect to the remote host IP.

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  30m                   default-scheduler  Successfully assigned core/featurehub-management-repository-84d4959f6f-ldz5s to org-kubeletl048
  Normal   Pulling    30m                   kubelet            Pulling image "quay.org.com/core/featurehub-mr:1.5.7-amd64"
  Normal   Pulled     30m                   kubelet            Successfully pulled image "quay.org.com/core/featurehub-mr:1.5.7-amd64" in 21.310779353s
  Normal   Created    25m (x4 over 30m)     kubelet            Created container featurehub-management-repository
  Normal   Started    25m (x4 over 30m)     kubelet            Started container featurehub-management-repository
  Normal   Pulled     25m (x3 over 30m)     kubelet            Container image "quay.org.com/core/featurehub-mr:1.5.7-amd64" already present on machine
  Normal   Killing    25m                   kubelet            Container featurehub-management-repository failed liveness probe, will be restarted
  Warning  Unhealthy  14m (x10 over 26m)    kubelet            Liveness probe failed: Get "http://7.150.220.105:8701/health/liveness": dial tcp 7.150.220.105:8701: connect: connection refused
  Warning  Unhealthy  10m (x12 over 26m)    kubelet            Readiness probe failed: Get "http://7.150.220.105:8701/health/readiness": dial tcp 7.150.220.105:8701: connect: connection refused
  Warning  BackOff    5m45s (x22 over 29m)  kubelet            Back-off restarting failed container

I made only slight adjustments to the provided Helm charts (modified the initialDelaySeconds for the MR deployment). One thing I noticed is that the NATS pods have to be up and running before the other pods for Dacha, Edge and MR can even begin to become ready. I have already tried kubectl exec -it into the MR pod and attempted a wget to the probes from within the Pod and that still failed with connection refused. I do not know what else to do and could use guidance/direction here as I am not finding anything useful in the existing documentation.

Below are the logs from the running MR pod in my k8s cluster:

{"@timestamp":"2022-05-09T16:22:53.440+0000","message":"Loading configuration `/etc/common-config/common.properties` into system properties","priority":"INFO","path":"com.bluetrainsoftware.bathe.initializer.BatheTimeWatcher","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:53.521+0000","message":"Property loader - file `/etc/app-config/application.properties` does not exist.","priority":"WARN","path":"com.bluetrainsoftware.bathe.initializer.BatheTimeWatcher","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:53.521+0000","message":"Property loader - file `/etc/app-config/secrets.properties` does not exist.","priority":"WARN","path":"com.bluetrainsoftware.bathe.initializer.BatheTimeWatcher","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:53.923+0000","message":"Sticky Configuration is ready to read from System Properties.","priority":"INFO","path":"cd.connect.app.config.SystemPropertiesConfigurationSource","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:55.421+0000","message":"A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available.","priority":"WARN","path":"org.glassfish.jersey.message.internal.MessagingBinders","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:55.530+0000","message":"OpenTelemetry configured","priority":"INFO","path":"io.featurehub.lifecycle.TelemetryFeature","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:55.636+0000","message":"connecting to database jdbc:mysql://dbdevl2.org.com:3306/featurehub","priority":"INFO","path":"io.featurehub.app.db.utils.CommonDbFeature","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:56.155+0000","message":"DataSourcePool [generic] autoCommit[false] transIsolation[READ_COMMITTED] min[2] max[200] in[429ms]","priority":"INFO","path":"io.ebean.datasource","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:56.234+0000","message":"Local migrations:13  existing migrations:13  checkState:false","priority":"INFO","path":"io.ebean.migration","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:56.239+0000","message":"Database Master configured at jdbc:mysql://dbdevl2.org.com:3306/featurehub","priority":"INFO","path":"io.featurehub.app.db.utils.CommonDbFeature","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:56.240+0000","message":"ebean version: 12.15.0","priority":"INFO","path":"io.ebean","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:57.730+0000","message":"Started database[db] platform[MYSQL] in 1387ms","priority":"INFO","path":"io.ebean.core","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:57.737+0000","message":"No oauth2 providers in config, skipping oauth2.","priority":"INFO","path":"io.featurehub.web.security.oauth.OAuth2Feature","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:58.135+0000","message":"Using maximum of 100 connections to service request from Dacha","priority":"INFO","path":"io.featurehub.db.publish.DbCacheSource","thread":"ImmediateThread-1652113377823","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:58.530+0000","message":"NATS connection successfully established","priority":"INFO","path":"io.featurehub.publish.NATSConnectionSource","thread":"ImmediateThread-1652113377823","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:58.729+0000","message":"listening on management subject `default/cache-management-v2`","priority":"DEBUG","path":"io.featurehub.db.publish.NamedCacheListener","thread":"ImmediateThread-1652113377823","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:22:58.731+0000","message":"Listening for updates on /default/feature-updates","priority":"INFO","path":"io.featurehub.db.listener.FeatureUpdateListener","thread":"ImmediateThread-1652113377823","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:23:01.522+0000","message":"\nAll endpoints for Jersey application\n   GET     /health/liveness\n   GET     /health/readiness\n   GET     /health/readyness\n   GET     /info/version\n   GET     /metrics\n   DELETE  /mr-api/application/{appId}\n   GET     /mr-api/application/{appId}\n   PUT     /mr-api/application/{appId}\n   GET     /mr-api/application/{appId}/rollout-strategy\n   POST    /mr-api/application/{appId}/rollout-strategy\n   POST    /mr-api/application/{appId}/rollout-strategy-validation\n   DELETE  /mr-api/application/{appId}/rollout-strategy/{strategyIdOrName}\n   GET     /mr-api/application/{appId}/rollout-strategy/{strategyIdOrName}\n   PUT     /mr-api/application/{appId}/rollout-strategy/{strategyIdOrName}\n   GET     /mr-api/application/{id}/all-feature-environment\n   GET     /mr-api/application/{id}/environment\n   POST    /mr-api/application/{id}/environment\n   POST    /mr-api/application/{id}/environment-ordering\n   GET     /mr-api/application/{id}/feature-environments/{key}\n   PUT     /mr-api/application/{id}/feature-environments/{key}\n   GET     /mr-api/application/{id}/features\n   POST    /mr-api/application/{id}/features\n   DELETE  /mr-api/application/{id}/features/{key}\n   GET     /mr-api/application/{id}/features/{key}\n   PUT     /mr-api/application/{id}/features/{key}\n   POST    /mr-api/authentication\n   POST    /mr-api/authentication/{email}/expiredTokenReset\n   PUT     /mr-api/authentication/{id}/changePassword\n   PUT     /mr-api/authentication/{id}/replaceTempPassword\n   PUT     /mr-api/authentication/{id}/resetPassword\n   GET     /mr-api/authentication/{token}\n   DELETE  /mr-api/environment/{eid}\n   GET     /mr-api/environment/{eid}\n   PUT     /mr-api/environment/{eid}\n   GET     /mr-api/external-provider/{provider}\n   GET     /mr-api/features/{eid}\n   PUT     /mr-api/features/{eid}\n   DELETE  /mr-api/features/{eid}/feature/{key}\n   GET     /mr-api/features/{eid}/feature/{key}\n   POST    /mr-api/features/{eid}/feature/{key}\n   PUT     /mr-api/features/{eid}/feature/{key}\n   DELETE  /mr-api/group/{gid}\n   GET     /mr-api/group/{gid}\n   PUT     /mr-api/group/{gid}\n   DELETE  /mr-api/group/{gid}/person/{pId}\n   POST    /mr-api/group/{gid}/person/{pId}\n   GET     /mr-api/initialize\n   POST    /mr-api/initialize\n   POST    /mr-api/login\n   GET     /mr-api/logout\n   GET     /mr-api/organization/{id}/group\n   GET     /mr-api/person\n   POST    /mr-api/person\n   DELETE  /mr-api/person/{id}\n   GET     /mr-api/person/{id}\n   PUT     /mr-api/person/{id}\n   GET     /mr-api/portfolio\n   POST    /mr-api/portfolio\n   DELETE  /mr-api/portfolio/{id}\n   GET     /mr-api/portfolio/{id}\n   PUT     /mr-api/portfolio/{id}\n   GET     /mr-api/portfolio/{id}/application\n   POST    /mr-api/portfolio/{id}/application\n   GET     /mr-api/portfolio/{id}/group\n   POST    /mr-api/portfolio/{id}/group\n   GET     /mr-api/portfolio/{id}/service-account\n   POST    /mr-api/portfolio/{id}/service-account\n   DELETE  /mr-api/service-account/{id}\n   GET     /mr-api/service-account/{id}\n   PUT     /mr-api/service-account/{id}\n   POST    /mr-api/service-account/{id}/reset-api-key\n   GET     /mr-api/user-data/state/{appId}/hidden-environments\n   POST    /mr-api/user-data/state/{appId}/hidden-environments\n","priority":"INFO","path":"io.featurehub.jersey.config.EndpointLoggingListener$ResourceLogDetails","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:23:01.632+0000","message":"Started listener bound to [0.0.0.0:8903]","priority":"INFO","path":"org.glassfish.grizzly.http.server.NetworkListener","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:23:01.722+0000","message":"[HttpServer] Started.","priority":"INFO","path":"org.glassfish.grizzly.http.server.HttpServer","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:23:01.722+0000","message":"server started on http://0.0.0.0:8903/ with http/2 enabled","priority":"INFO","path":"io.featurehub.jersey.FeatureHubJerseyHost","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}
{"@timestamp":"2022-05-09T16:23:01.722+0000","message":"MR Launched - (HTTP/2 payloads enabled!)","priority":"INFO","path":"io.featurehub.Application","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}

[rvowles]

Hey @thedanchez - sorry, I never saw this message come up 😟

Did you manage to figure out what the problem was? And yes, Nats healthy is a requirement, the other servers include a check on Nats.

From your logs it's almost like it wasn't even asking the right urls for health?

Richard

[thedanchez]

Hi @rvowles, no haven't figured out the problem yet unfortunately. I haven't touched anything from the default provided Helm charts with regards to ports or IP addresses. As far as I know, the only things I've changed are the image target paths, setting the enabled fields for all the individual infra pieces of dacha, edge and managementRepository and setting the hosts/tls for those sections.

[thedanchez]

Unfortunately, due to all the problems I've been facing in getting the individual pieces working, I'm at this point going to resort to just using the all-in-one party server solution because the alternative has not been working despite all the efforts.

[rvowles]

Ok, i'll take a look tonight - our Slack is a great way to get a hold of us for feedback quickly, for some reason I miss these discussion messages :-(

[rvowles]

I'll see if I can get it sorted and ping you for any testing if thats suitable?

[rvowles]

Hi @thedanchez - I spent a bit of time with this tonight.

I deleted my kind cluster and recreated it, and then reinstalled the dependent helm charts and reinstalled the app as per the readme. It all came up and was healthy, but I couldn't hit the UI - and that is a configuration error (run.nginx=true has been removed from the env file for MR for some reason).

What I can see in your startup above is that it is definitely not starting properly based on the helm chart that is in our current repo on Master. It should have two environment variables or system properties - server.port=8085 and monitor.port=8701. Your server has neither set and is starting on the default (unconfigured) port of 8903 - and exposes all of the endpoints there.

You can see the liveness check here:

Readiness probe failed: Get "http://7.150.220.105:8701/health/readiness": dial tcp 7.150.220.105:8701: c

and your server logs show this:

{"@timestamp":"2022-05-09T16:23:01.722+0000","message":"server started on http://0.0.0.0:8903/ with http/2 enabled","priority":"INFO","path":"io.featurehub.jersey.FeatureHubJerseyHost","thread":"main","host":"featurehub-management-repository-84d4959f6f-z46fz"}

there is no port 8701 being bound.

Another telltale sign for me is this log:

All endpoints for Jersey application\n GET /health/l

All the health endpoints and application endpoints are loaded onto a single port - which you normally don't want :-)

So you need to get those two variables into your config - in the standard one, its in helm/featurehub/templates/management-repository/configmap-env.yaml - if that helps?

Also - really please contact me on Slack, I have done concalls with other people and we are happy to help you out :-)

[rvowles]

@thedanchez i have a PR: featurehub-io/featurehub-install#39

[thedanchez]

@rvowles this is my current configmap-env for the MR.

{{- if .Values.managementRepository.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "featurehub.fullname" . }}-management-repository-env
  labels:
    {{- include "featurehub.managementRepository.labels" . | nindent 4 }}
data:
{{- with .Values.managementRepository.environmentVars }}
  {{- toYaml . | nindent 2 }}
{{- end }}
  monitor.port: "8701"
  server.port: "8085"
  run.nginx: "true"
{{- end }}

The monitor.port and server.port fields were already there, but I didn't have run.nginx defined. Despite these being set, the MR server still starts on port 8903. I don't know how or why it does and there's nothing in the featurehub-install repo that has 8903 defined anywhere. The readiness/liveness probes still fail with these things defined.

Didn't realize you had a Slack channel. I recently joined in and will start a convo soon.

[rvowles]

If you have time, you should also be able to do a kubectl get deployment featurehub-management-repository -o yaml as well - see what the environment variables are there. There is this section in the template that extracts those env vars from that same location and puts them in your deployment.