Dynamically select pod instead of statically passing Pods/PodIPs into ext proc

[Xunzhuo]

Currently we pass the static pod names and IPs to ext proc server, we should use more dynamic approach to fetch the data from kubernetes cluster like by selectors.

[Joffref]

In the near future, a new CR, tentatively called BackendPool, will be introduced.

llm-instance-gateway/docs/proposals/002-api-proposal/proposal.md

Lines 197 to 221 in d385c80

	**BackendPool**
	```golang
	// The BackendPool is a construct for pooling compute (often model servers) to
	// serve large models, that have the ability to share capacity across multiple
	// use cases (such as through prompt engineering, LoRA adapters, etc).
	// BackendPools have a dependency on a Gateway that is compatible with ext-proc
	// (External Processing). When a new BP object is created, a new ext proc
	// deployment is created. BackendPools require at minimum a single UseCase to
	// be subscribed to them to accept traffic, any traffic with a model not
	// definied within a UseCase will be rejected.
	type BackendPool struct {
	metav1.ObjectMeta
	metav1.TypeMeta
	Spec BackendPoolSpec
	}
	type BackendPoolSpec struct {
	// Select the distinct services to include in the backend pool. These
	// services should be consumed by only the backendpool they are part
	// of. Should this behavior be breached, routing behavior is not
	// guaranteed.
	ServiceRef []corev1.ObjectReference
	}
	```

This resource will reference the services exposing inference servers that share certain characteristics, mainly the same set of loaded adapters. The final name of this resource is still being discussed, and you can review the related documents for more information: https://docs.google.com/document/d/1v1Rp6v_AfY5EfwpLqDadDpAaCg7OcnrUutzBUNxGoJE/edit?pli=1

Once introduced, BackendPool will be referenced in the backendRefs field of HTTPRoute to manage routing.

The idea of referencing services directly could be re-evaluated, given that using selectors was the original approach. However, directly referencing pods would involve managing a structure similar to EndpointSlices within the gateway. IMO, this adds unnecessary complexity and introduces potential security risks.

Let me know your thoughts on this—I’d be happy to discuss it further.

[liu-cong]

cc @kfswain @robscott

[liu-cong]

/close

Closed by #36

[k8s-ci-robot]

@liu-cong: Closing this issue.

In response to this:

/close

Closed by #36

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.