Skip to content

Latest commit

 

History

History
122 lines (86 loc) · 4.6 KB

tensorflow.md

File metadata and controls

122 lines (86 loc) · 4.6 KB

Inference of MNIST using TensorFlow on Amazon EKS

This document explains how to perform inference of Fashion-MNIST model using TensorFlow and Keras on Amazon EKS.

Pre-requisite

  1. Create EKS cluster using GPU with Kubeflow
  2. Basic understanding of TensorFlow Serving

Upload model

  1. If you've gone through the Training MNIST using TensorFlow and Keras on Amazon EKS, a model is already stored in the identified S3 bucket. Initialize S3_BUCKET variable with the bucket:

    export S3_BUCKET=eks-ml-data

    Skip rest of this section.

  2. If you've not done the training, a pre-trained model is already available at samples/mnist/training/tensorflow/saved_model. This model requires your inference cluster has GPU. Use an S3 bucket in your region and upload this model:

    export S3_BUCKET=eks-ml-data
cd samples/mnist/training/tensorflow/saved_model
aws s3 sync . s3://$S3_BUCKET/mnist/tf_saved_model/

Install TensorFlow Serving component

  1. Install TensorFlow Serving package:

    cd ${KUBEFLOW_SRC}/${KFAPP}/ks_app
ks pkg install kubeflow/tf-serving

  2. Use Store AWS Credentials in Kubernetes Secret to configure AWS credentials in your Kubernetes cluster with the name aws-secret. This needs to be created in the kubeflow namespace. So the command may look like:

    kubectl create -f secret.yaml -n kubeflow

  3. Install Tensorflow Serving AWS Component (Deployment + Service):

    cd $KUBEFLOW_SRC/$KFAPP/ks_app
export TF_SERVING_SERVICE=mnist-service
export TF_SERVING_DEPLOYMENT=mnist

ks generate tf-serving-service ${TF_SERVING_SERVICE}
# match your deployment mode name
ks param set ${TF_SERVING_SERVICE} modelName ${TF_SERVING_DEPLOYMENT}
# optional, change type to LoadBalancer to expose external IP.
ks param set ${TF_SERVING_SERVICE} serviceType ClusterIP

ks generate tf-serving-deployment-aws ${TF_SERVING_DEPLOYMENT}
# make sure to match the bucket name used for model
ks param set ${TF_SERVING_DEPLOYMENT} modelBasePath s3://$S3_BUCKET/mnist/tf_saved_model
ks param set ${TF_SERVING_DEPLOYMENT} s3Enable true
ks param set ${TF_SERVING_DEPLOYMENT} s3SecretName aws-secret
ks param set ${TF_SERVING_DEPLOYMENT} s3UseHttps true
ks param set ${TF_SERVING_DEPLOYMENT} s3VerifySsl true
ks param set ${TF_SERVING_DEPLOYMENT} s3AwsRegion us-west-2
ks param set ${TF_SERVING_DEPLOYMENT} s3Endpoint s3.us-west-2.amazonaws.com

ks param set ${TF_SERVING_DEPLOYMENT} numGpus 1

  4. Deploy Tensorflow Serving components

    ks apply default -c ${TF_SERVING_SERVICE}
ks apply default -c ${TF_SERVING_DEPLOYMENT}

    Wait for the containers to start:

    kubectl get pods -n kubeflow --selector=app=mnist -w
NAME                    READY   STATUS              RESTARTS   AGE
mnist-cfdb9c99b-fgbz2   0/1     ContainerCreating   0          38s
mnist-cfdb9c99b-fgbz2   1/1     Running             0          41s

  5. Port forward inference endpoint for local testing:

    kubectl port-forward -n kubeflow `kubectl get pods -n kubeflow --selector=app=mnist -o jsonpath='{.items[0].metadata.name}' --field-selector=status.phase=Running` 8500:8500

  6. Original datasets are feature vectors and we use matplotlib to draw picture to compare results. Install the following TensorFlow client-side components:

    pip install tensorflow matplotlib --user

  7. Use the script inference_client.py to make prediction request. It will randomly pick one image from test dataset and make prediction.

    $ python inference_client.py --endpoint http://localhost:8500/v1/models/mnist:predict

 Data: {"instances": [[[[0.0], [0.0], [0.0], [0.0], [0.0] ... 0.0], [0.0]]]], "signature_name": "serving_default"}
 The model thought this was a Ankle boot (class 9), and it was actually a Ankle boot (class 9)

    inference-random-example

    Running this client shows an image and the output text indicates what kind of object it is.

  8. Get serving pod name:

    TF_MNIST_POD=$(kubectl get pods -n kubeflow --selector=app=mnist --field-selector=status.phase=Running -o jsonpath={'.items[0].metadata.name'})
NAME                     READY   STATUS    RESTARTS   AGE
mnist-7cc4468bc5-wm8kx   1/1     Running   0          2h

    Check the logs:

    kubectl logs $TF_MNIST_POD -n kubeflow