Clarification Needed: Proper use of tezos-node-rpc service over nodes' headless services

[harryttd]

Background:
Currently, when deploying Tezos nodes, a headless service is created for each statefulset. From a discussion we had, our rationale behind possibly eliminating the headless service for public chain node sts's was because the nodes don't require individual DNS records; they do not rely on each other. However for private chains, nodes need to bootstrap together and utilize DNS records created by the headless service.

After further investigation, It's evident that even public chain nodes could require the headless services, for things such as indexers. Indexers target specific Tezos node pods. Using multiple nodes isn't ideal since different nodes can possess different states.

Required Action:

1. This discussion came up due to confusion of which service to forward ingress RPC traffic to. We need clear documentation and in-code comments emphasizing that for RPC traffic targeting, the tezos-node-rpc service should be preferred over headless services.
2. More instructions should be provided to configure label selectors for the tezos-node-rpc service in values.yaml allowing users to select only a subset of nodes, if needed, for RPC traffic. By default all nodes get RPC traffic forwarded to them by the service.

Relevant snippets:

tezos-k8s/charts/tezos/values.yaml

Lines 272 to 280 in 5ec120a

	## Configuration for K8s Service resources. Configuring the labels selector of a
	## service will result in the service forwarding traffic only to pods that have
	## matching labels.
	services:
	## Configure the labels selector of the Tezos node RPC service. Traffic will
	## be forwarded to all nodes if no labels are specified.
	tezos_node_rpc:
	selector:
	# rpc_node: "true"

tezos-k8s/charts/tezos/templates/static.yaml

Lines 1 to 34 in 5ec120a

	apiVersion: v1
	kind: Service
	metadata:
	name: tezos-node-rpc
	namespace: {{ .Release.Namespace }}
	spec:
	ports:
	- port: 8732
	name: rpc
	{{- with .Values.services.tezos_node_rpc.selector | default (dict "appType" "octez-node") }}
	selector:
	{{- toYaml . | nindent 4 }}
	{{- end }}
	type: NodePort
	---
	{{- range $key, $val := .Values.nodes }}
	apiVersion: v1
	kind: Service
	metadata:
	name: {{ $key }}
	labels:
	appType: octez-node
	spec:
	ports:
	- port: 8732
	name: rpc
	- port: 9732
	name: p2p
	- port: 9932
	name: metrics
	publishNotReadyAddresses: true
	clusterIP: None
	selector:
	node_class: {{ $key }}

[puppetninja]

"After further investigation, It's evident that even public chain nodes could require the headless services, for things such as indexers. Indexers target specific Tezos node pods. Using multiple nodes isn't ideal since different nodes can possess different states."

Can you elaborate on this ? I don't think this applies, at least for tzkt there is mechanism to handle 404 blocks with a retry, and it works fine with different rpcs behind a loadbalancer, they are aware of this since the beginning of the development I believe.

[harryttd]

It would be safer to index a single node vs creating the index from multiple different sources. Because each node could theoretically have different state (as far as i understand).

I saw your issue baking-bad/tzkt#146 about the 404s, but reorgs and nodes with diff state is still a possibility.

I see that tzkt has a lag option to handle potential reorgs: See https://github.com/baking-bad/tzkt#chain-reorgs-and-indexing-lag and baking-bad/tzkt#88.

We also support BCD indexer and I'm not aware if it handles nodes reorgs.

[puppetninja]

Ok, thanks for the clarification!