add k8s docs for getting started, K8s Manifest and Helm

[devpramod]

This PR contains the following docs:

• Getting Started for k8s - Installation, basic introduction to k8s and has a section for helm and k8s manifest. As more k8s deployment modes are added, corresponding sections will be created in this doc

• Deploy using helm charts, a doc that follows the xeon.md template as much as possible to deploy ChatQnA on k8s using Helm

• Deploy using K8s Manifest, a doc that follows the xeon.md template as much as possible to deploy ChatQnA on k8s using a K8s manifest yaml

[dbkinder]

some suggested edits

Also, when you add new documents, they need to be linked into the table of contents structure. There's an index.rst file in this folder you can edit to add these two documents.

I'd suggest you add an edit to the index.rst doc in this deploy folder, and replace the existing Kubernetes section with this:

Kubernetes
**********

.. toctree::
   :maxdepth: 1

   k8s_getting_started
   TGI on Xeon with Helm Charts <k8s_helm>

* Xeon & Gaudi with GMC
* Xeon & Gaudi without GMC

[dbkinder]

LGTM, thanks!

[tylertitsworth]

Some of the stuff I see in the docs, is just a tutorial on things that already have docs. Like TGI/TEI, Helm, and Kubernetes. It feels a lot like we're overexplaining a concept that can be answered by a link to the source docs of another tool and a command for how it's relevant to use with ChatQnA.

For reference, this is the most handholding I would do in the case of deploying TGI:

---

Configure Model Server

Before we deploy a model, we need to configure the model server with information like, what model to use and how many max tokens to use. We will be using the tgi-on-intel helm chart. This chart uses XPU to the serve model normally, but we are going to configure it to use gaudi2 instead.

First, look at the configuration files in the tgi directory and add/remove any configuration options relevant to your workflow:

cd tgi
# Create a new configmap for your model server to use
kubectl apply -f cm.yaml

Tip

Here is the reference to the Huggingface Launcher Environment Variables and the TGI-Gaudi Environment Variables.

Deploy Model Server

Now that we have configured the model server, we can deploy it to Kubernetes. Using the provided config.yaml file in the tgi directory, we can deploy the model server.

Modify any values like resources or replicas in the config.yaml file to suit your needs. Then, deploy the model server:

# Encode HF Token for secret.encodedToken
echo -n '<token>' | base64
# Install Chart
git clone https://github.com/intel/ai-containers
helm install model-server -f config.yaml ai-containers/workflows/charts/tgi
# Check the pod status
kubectl get pod
kubectl logs -f <pod-name>

---

Please use a tool like markdownlint to ensure consistent styling.

[dbkinder]

I've got a script in docs/scripts/checkmd.sh that uses pymarkdown (lint) to scan markdown files, with a bunch of checks disabled. Alas, if I wasn't retiring today, including a markdown linter was on my list to add to the CI checks. :)

[arun-gupta]

The initial setup is confusing as the docs point to set up k8s in multiple ways where as the tested configuration is minikube. We need to clearly document that. Other than that, there are some more clarifications required.

[arun-gupta]

Lets remove this one, it is no longer maintained.

[arun-gupta]

Keep it simple, just publish only Helm charts.

[arun-gupta]

Its not clear whether this script is standard for all Helm charts or unique to OPEA?

[eero-t]

Unique to OPEA. Normally one just calls Helm dep command directly.

[arun-gupta]

Then we should use the standard tools instead of creating our own mechanisms

[eero-t]

I'm rather new to Helm, so all of this is very much AFAIK:

• Script is there because OPEA has many of the dependencies locally instead of in a Helm repo
• Normally users would update the repo first, and that would basically do what the script is needed for (refresh the deps to latest, not just minimum version)
• OPEA Helm charts are moving to repo, so I think in future that script is needed only in case when one wants to use local checkout

@poussa ?

[arun-gupta]

We need to provide guidance on the memory/disk requirements for k8s cluster, otherwise ChatQnA may not run.

[arun-gupta]

It was tested on a single node k8s cluster so the "multi-node" part is not accurate

[arun-gupta]

this should be lines: 58-62

56 # If you would like to switch to traditional UI image
 57 # Uncomment the following lines
 58 # chatqna-ui:
 59 #   image:
 60 #     repository: "opea/chatqna-ui"
 61 #     tag: "1.1"
 62 #   containerPort: "5173"

Also, just uncommenting may not be sufficient as that messes up with formatting. There is an additional space that needs to be deleted too.

[arun-gupta]

Could not get all pods to run, filed opea-project/GenAIExamples#1202

[arun-gupta]

This caused an error:

Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    10m                 default-scheduler  Successfully assigned default/chatqna-tei-645548d7f-7crjj to minikube
  Warning  FailedMount  17s (x13 over 10m)  kubelet            MountVolume.SetUp failed for volume "model-volume" : hostPath type check failed: /mnt/opea-models is not a directory

Using this command export MODELDIR="null" solved the issue.

[lianhao]

I would not recommend to set global.modelUseHostPath=${MODELDIR} in helm install in multi-node environment. This options is meant to share the pre downloaded model data files in a single node environment for a quick setup to test. To use this in a multiple nodes environment , you need to go to every K8S worker node to make sure there is a ${MODELDIR} directory exists and writable. Just don't set global.modelUseHostPath and leave it as the default value(empty string) which will download the models without sharing.

Another option to share the model files is to set global.modelUsePVC to use K8S persistent volume to share the model data files. In order to use that, users need to do some preparation work out-of-band:

1. Setup the K8S storage class, persistent volumes.
2. Create a K8S persistent volume claim which can be shared by multi pods.
3. pass the created k8s persistent volume claim to global.modelUsePVC

[eero-t]

I would not recommend setting it empty either, because not sharing the data will much more easily cause node to run out disk space, meaning that k8s evicts pods from it, and things in general break.

[eero-t]

Not all associated resources, only replicaset and pods. E.g. associated Service, ServiceMonitor, configMap, access control etc resources still remain.

Things should be deleted using same manifest or Helm chart that was used to create them!

[eero-t]

Use of "current" in all of these is wrong.

When namespace is not specified, removal is done from "default" namespace, not "current" one, whatever that is.

=> I think all examples should include namespace option,.

[eero-t]

IMHO would make sense to have this section before above one.

[eero-t]

Somewhat misleading. Maybe?

Suggested change

	| `values.yaml` | Stores configuration values that can be customized depending on the deployment environment. These values override defaults set in the chart templates. |
	| `values.yaml` | Overridable configuration values for the Helm chart deployment, used in the chart k8s object templates. |

[eero-t]

Incorrect. Typically template directory has one file per k8s object, with object type being used as the file name.

Better to just mention template dir, and maybe have pointer to Helm docs: https://helm.sh/docs/chart_best_practices/templates/

[eero-t]

IMHO this section should be be removed. Manifests will be removed because they're generated from Helm charts, they don't catch all the configurations possible with Helm charts, and are continuously out of sync.

[eero-t]

"Docker image sources" => "container image name/version"?

(k8s supports also OCI standard images in addition to legacy Docker format.)

[eero-t]

Query specified here seems outdated based on answer, and data-prep being given OPEA material, and re-run of query specified below?

[arun-gupta]

Getting the following error when running this query:

ubuntu@ip-172-31-7-63:~/.minikube$ curl http://localhost:8888/v1/chatqna -H "Content-Type: application/json" -d '{
     "model": "Intel/neural-chat-7b-v3-3",
     "messages": "What is the revenue of Nike in 2023?"
     }'
curl: (7) Failed to connect to localhost port 8888 after 0 ms: Couldn't connect to server

[eero-t]

It does not look like this. Each data: output is on its own line, with extra empty lines between.

(Same issue in output below.)

[eero-t]

Link is wrong (can be tested with "View file" functionality):

Suggested change

	For advanced usage of the dataprep microservice refer [here](#dataprep-microservice-%28advanced%29)
	For advanced usage of the dataprep microservice refer [here](#dataprep-microservice-advanced).

[eero-t]

Please remove these redundant empty lines from all the command examples. They do not look good.

[eero-t]

Please use correct formatting for commands and command lines:

Suggested change

	In this example the embedding model used is "BAAI/bge-base-en-v1.5", which has a vector size of 768. So the output of the curl command is a embedded vector of
	In this example the embedding model used is "BAAI/bge-base-en-v1.5", which has a vector size of 768. So the output of the `curl` command is a embedded vector of

[eero-t]

Both export and embedding var are redundant:

Suggested change

	export your_embedding=$(python3 -c "import random; embedding = [random.uniform(-1, 1) for _ in range(768)]; print(embedding)")
	your_embedding=$(python3 -c "import random; print([random.uniform(-1, 1) for _ in range(768)])")

[eero-t]

With the uploaded document being about OPEA, output being about Nike fiscals seems unlikely.

[eero-t]

Suggested change

	TGI service generate text for the input prompt. Here is the expected result from TGI:
	TGI service generates text for the input prompt. Here is the expected result from TGI:

[eero-t]

And it has resources (memory) for running all of them...

[eero-t]

Suggested change

We recommend using newer CPUs, such as 4th Gen Intel Xeon Scalable processors (code-named Sapphire Rapids) and later, that support the bfloat16 data type. If your hardware includes such CPUs and your model is compatible with bfloat16, adding the `--dtype bfloat16` argument to the HuggingFace `text-generation-inference` server can significantly reduce memory usage by half and provide a moderate speed boost. This change has already been configured in the `chatqna_bf16.yaml` file. To use it, follow these steps:

We recommend using newer CPUs, such as 4th Gen Intel Xeon Scalable processors (code-named Sapphire Rapids) and later, that support the bfloat16 data type. If your hardware includes such CPUs and your model is compatible with bfloat16, adding the `--dtype bfloat16` argument to the HuggingFace `text-generation-inference` server halves its memory usage and provides a moderate speed boost. This change has already been configured in the `chatqna_bf16.yaml` file. To use it, follow these steps:

[eero-t]

With Helm charts, one can e.g. enable the relevant BF16 option in the values file: https://github.com/opea-project/GenAIInfra/blob/main/helm-charts/common/tgi/values.yaml#L23

[eero-t]

Does langchain really provide weights too, and not just manipulate them?

[eero-t]

Suggested change

	The ChatQnA deployment starts 9 Kubernetes services. Ensure that all associated pods are running, i.e., all the pods' statuses are 'Running'.
	The ChatQnA deployment starts 9 Kubernetes services. Ensure that all associated pods have `Running` status and have initialized themselves successfully, i.e. show `1/1` as their `Ready` state.