KServe: Highly Scalable Machine Learning Deployment with Kubernetes | by Lloyd Hamilton | May, 2023

Estimating the Kubenetes model just got easier.

In the wake of the release of ChatGTP, it is becoming increasingly difficult to avoid techniques that leverage machine learning. From text prediction on your messaging app to facial recognition on your smart doorbell, machine learning (ML) can be found in almost every technology we use today.

How to deliver machine learning technology to consumers is one of the many challenges that organizations must address as they grow. Deployment strategies for ML products have a significant impact on the end users of your product. It can mean the difference between Siri on your iPhone or Chat GPT in your web browser.

Beyond the flashy user interface and highly articulate chat dialogs of ChatGPT, the large language hides the complex mechanisms required to deploy ML models. ChatGPT is built on a highly scalable framework designed to deliver and support the model during its exponential adoption. In fact, the actual ML model will be only a small part of the entire project. Such projects are often cross-disciplinary and require expertise in data engineering, data science and software development. Therefore, frameworks that simplify model deployment processes are becoming increasingly important in delivering models in order to save time and money to organizations that go into production.

Without the proper operational framework to support and manage ML models, organizations will often face roadblocks when attempting to scale up the number of ML models in production.

While no single tool has emerged as a clear winner in the highly saturated market of MLOps toolkits, KServe is becoming an increasingly popular tool to help organizations meet the scalability requirements of ML models.

Kserv is a highly scalable machine learning deployment toolkit for Kubernetes. It is an orchestration tool built on top of Kubernetes and leverages two other open source projects, native-serving and Istio; More on this later.

KServe significantly simplifies the deployment process of ML models in a Kubernetes cluster by unifying the deployment into a single resource definition. This makes machine learning deployment of any ML project easier to learn and ultimately lowers the barrier to entry. Therefore, models deployed using KServ may be easier to maintain than models deployed using traditional Kubernetes deployments, which require a Flask or FastAPI service.

With KServe, there’s no need to wrap your models inside a FastAPI or Flask app before exposing them via HTTPs to the intenet. KServe has built in functionality that essentially replicates this process but without the overhead of maintaining API endpoints, configuring pod replicas, or internal routing networks on Kubernetes. All we have to do is point KServe towards your model and it will handle the rest.

Beyond the simplification of deployment processes, KServe also provides a number of features including canary deployment, predictive autoscaling, and request batching. These features will not be discussed further as it is out of scope, however, this guide will establish the foundation of understanding needed for further exploration.

First, let’s talk about the two major technologies, Istio and Knative, that come with KServe.

Without Istio a lot of the functionality offered by KServe would be difficult. Istio is a service mesh that extends your applications deployed in Kubernetes. It is a dedicated infrastructure layer that combines capabilities such as observability, traffic management, and security. For those familiar with Kubernetes, Istio will replace the standard ingress definitions commonly found in Kubernetes clusters.

The complexity of managing traffic and maintaining observability will increase as Kubernetes-based systems scale. One of the best features of Istio is centralizing the control of service level communication. This gives developers more control and transparency over communication between services.

With Istio, there is no need for developers to design applications that need to be able to handle traffic authentication or authorization. Ultimately, Istio helps reduce the complexity of deployed apps and allows developers to focus on the critical components of the app.

By leveraging Istio’s networking features, KServe is able to bring features that include canary deployment, inference graphs, and custom transformers.

Knative, on the other hand, is an open source enterprise level solution for building serverless and event driven applications. Knative is built on top of Istio to bring serverless code execution capabilities similar to those offered by AWS Lambdas and Azure Functions. Knative is a platform-agnostic solution for running serverless deployments in Kubernetes.

One of the best features of Knetiv is the zero feature scale which automatically scales down deployments when there is no demand. This is a critical component of KServe’s ability to scale up or scale down ML model deployments that maximize resource utilization and save costs.

Like many other tools, K-Serve is not a one-size-fits-all solution that best suits your organization’s needs. It has a high cost of entry due to the fact that working with Kubernetes requires some experience. If you are just starting out with Kubernetes, there are many resources online and I highly recommend checking out resources like the DevOps guy on Youtube. Nonetheless, it is possible to learn how to use KServe, even without a deep understanding of Kubernetes.

KServ will be ideal for organizations already leveraging Kubernetes where there is existing knowledge of working with Kubernetes. It may also be suitable for organizations that want to build off or complement managed services like SageMaker or Azure Machine Learning to have more control over their model deployment process. Increased ownership can result in significant cost reductions and increased configurability to meet project specific requirements.

Nevertheless, the right cloud infrastructure decision will depend on a case by case basis as infrastructure requirements will vary across companies.

This guide will take you through the steps required to get setup with KServe. You will be walked through the steps to set up KServe and render your first model.

Several prerequisites must be met before proceeding. You will need the following:

kubernetes cluster

For this tutorial, I recommend experimenting with a Kubernetes cluster using Kind. It is a tool for running a local Kubernetes cluster without the need to spin up cloud resources. Also, I recommend kubectux as a tool to easily switch between Kubernetes contexts if you are working in multiple clusters.

However, when running production workloads, you will need access to a fully functional Kubernetes cluster to configure DNS and HTTPS.

Deploy the Kubernetes cluster as follows:

kind create cluster --name kserve-demo

Switch to the correct Kubernetes context with:

kubectx kind-kserve-demo

The following steps will install Istio v1.16, Knative Serving v1.7.2 and KServe v0.10.0. These versions are best suited for this tutorial as Knative v1.8 onwards access will require DNS configuration which adds a layer of complexity that is currently out of scope.

1. Istio installation.

# Install istio
curl -L  | ISTIO_VERSION=1.16.0 TARGET_ARCH=x86_64 sh -
istioctl install --set profile=default -y

2. Install KNative Serving.

# Install the Knative Serving component
export KNATIVE_VERSION="v1.7.2"
kubectl apply -f https://github.com/knative/serving/releases/download/knative-$KNATIVE_VERSION/serving-crds.yaml
kubectl apply -f https://github.com/knative/serving/releases/download/knative-$KNATIVE_VERSION/serving-core.yaml
# Install istio-controller for knative
kubectl apply -f 

3. Install Cert Manager. A certificate manager is required to manage valid certificates for HTTPs traffic.

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.11.0 --set installCRDs=true

4. Create a namespace for the model.

kubectl create namespace kserve

5. Clone the Kserv repository.

git clone git@github.com:kserve/kserve.git

6. Install the KServe Cutom resource definitions and the KServe runtime in the model namespace in your cluster.

cd kserve
helm install kserve-crd charts/kserve-crd -n kserve
helm install kserve-resources charts/kserve-resources -n kserve

Great! Now we have kserv installed on the cluster. Let’s deploy!

To ensure that the deployment goes smoothly, let us deploy a demo ingress service. The source code for deployment can be found here.

kubectl apply -n kserve -f - <<EOF
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
name: "sklearn-iris"
spec:
predictor:
model:
modelFormat:
name: sklearn
storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
EOF

The above YAML resource definition deploys a test inference service that sources publicly available models trained using the SciKit-Learn library. Kserv supports several different flavors of machine learning libraries. These include MLFlow, PyTorch or XGBoost models; More are added in each release. If none of these offbeat libraries meet your needs, KServe also supports custom predictors.

It is possible to monitor the status of the current deployment by getting the pods available in the namespace.

kubectl get pods -n kserve

If you run into issues with deployment, use the following to debug:

kubectl describe pod <name_of_pod> -n kserve

We can also see the status of the inference service deployment with:

kubectl get isvc -A

If the guess service is marked as true, we are ready to make our first prediction.

To make predictions, we’ll need to determine whether our Kubernetes cluster is running in an environment that supports external load balancers.

kubectl get svc istio-ingressgateway -n istio-system

Dayalu Cluster

Clusters deployed using Kind do not support external load balancers so you will have a gateway that looks like the one below.

In this case we need to port-forward the istio-ingressgateway which will allow us to access it through localhost,

port-forward istio gateway service to port 8080 But localhost Together:

kubectl port-forward -n istio-system service/istio-ingressgateway 8080:80

Then set the ingress host and port with:

export INGRESS_HOST=localhost
export INGRESS_PORT=8080

kubernetes cluster

If external IP is valid and not displayed <pending>We are able to send an estimate request via internet to IP address.

Set the ingress host and port with:

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath=".status.loadBalancer.ingress(0).ip")
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath=".spec.ports(?(@.name=="http2")).port")

To draw conclusions

create an input request json File for findings request.

cat <<EOF > "./iris-input.json"

"instances": (
(6.8,  2.8,  4.8,  1.4),
(6.0,  3.4,  4.5,  1.6)
)

EOF

Then do a guess with curl:

SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -n kserve -o jsonpath=".status.url" | cut -d "/" -f 3)
curl -v -H "Host: $SERVICE_HOSTNAME" " -d @./iris-input.json

The request will be forwarded to the KServe deployment through the istio-ingress gateway. If everything is in order, we will get a json Answer from guess service with prediction of (1,1) for each example.

By leveraging the features of KNative, KServe supports scale to zero capabilities. This feature effectively manages the limited resources in the cluster by scaling the pod down to zero. Scaling down to zero capabilities allows for the creation of a reactive system that responds to requests as opposed to a system that is always up. This will facilitate the deployment of a greater number of models in a cluster as compared to the traditional deployment configuration.

Note, however, that there is a cold start penalty for the lowered pods. This will vary depending on the size of the image/model and available cluster resources. Cold start can take 5 minutes if the cluster takes an additional node or 10 seconds if the model is already cached on the node.

Let us modify the existing scikit-learn inference service and enable scale to zero by defining minReplicas: 0,

kubectl apply -n kserve -f - <<EOF
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
name: "sklearn-iris"
spec:
predictor:
minReplicas: 0
model:
modelFormat:
name: sklearn
storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
EOF

Setting minReplicas to 0 will instruct Knative to scale down the inference service to zero when there is no HTTP traffic. You will notice that after a period of 30 seconds, the pods of the Sklearn-Iris model will be truncated.

kubectl get pods -n kserve

To restart the prediction service, send a prediction request to the same endpoint.

SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -n kserve -o jsonpath=".status.url" | cut -d "/" -f 3)
curl -v -H "Host: $SERVICE_HOSTNAME" " -d @./iris-input.json

This will trigger pod initialization from a cold start and return a prediction.

KServe simplifies the process of machine learning deployment and shortens the path to production. When combined with Knative and Istio, KServe has the added bonus of being highly customizable and brings a range of features that easily rivals those offered in managed cloud solutions.

Of course, bringing the model deployment process in-house has its own inherent complexities. However, increased platform ownership will provide greater flexibility in meeting project specific requirements. With the right Kubernetes expertise, KServe can be a powerful tool that will allow organizations to easily scale their machine learning deployments on any cloud provider to meet growing demand.