Agent for granite 3/ llama 3 model running on OpenShift

[gskumar1010]

If there is a granite3/ llama 3 model running on OpenShift/Kubernetes,

Is there a way of building agent using the Bee framework?

Thanks

[gskumar1010]

Just to clarify:

If I have the agent running on OpenShift, how can it interact with a model that is also running on OpenShift?

https://github.com/i-am-bee/bee-agent-framework/blob/main/.env.template#L9
expects Watsonx details. Is there a way of just providing the url of the LLM thats being served on OpenShift?

Thanks

[Tomas2D]

In that case, I assume you would like to use a vLLM adapter.

[gskumar1010]

Thanks @Tomas2D for the response. A couple of quick questions.

1. Can we use the a vllm adapter like ibm-vllm or do you see a need to implement a custom vllm adapter
2. So, if the llm is running on OpenShift on vllm, is there a way of providing the url?
   vllm-adapter seems to just have option for modelId

[Tomas2D]

See https://github.com/i-am-bee/bee-agent-framework/blob/main/examples/llms/providers/ibm-vllm.ts.

The Client class takes URL which is then processed. See implementation

bee-agent-framework/src/adapters/ibm-vllm/client.ts

Lines 91 to 110 in b5e10fa

	protected getClient(modelId: string) {
	const modelSpecificUrl = this.options.url.replace(/{model_id}/, modelId.replaceAll("/", "--"));
	const client = new generationPackageObject.fmaas.GenerationService(
	modelSpecificUrl,
	grpc.credentials.createSsl(
	Buffer.from(this.options.credentials.rootCert),
	Buffer.from(this.options.credentials.privateKey),
	Buffer.from(this.options.credentials.certChain),
	),
	this.options.grpcClientOptions,
	);
	setTimeout(() => {
	try {
	client.close();
	} catch {
	/* empty */
	}
	}, GRPC_CLIENT_TTL + this.options.clientShutdownDelay).unref();
	return client;
	}