Kfserving

InferenceService V1Beta1
:ship: TorchServe Integration!

- Add TorchServe to v1beta1 spec 1161 by jagadeeshi2i

📝 Documentation
https://github.com/kubeflow/kfserving/tree/master/docs/samples/v1beta1/torchserve

InferenceService V1Beta1
:ship: KFServing 0.5 promotes the core InferenceService from v1alpha2 to v1beta1!

The minimum required versions are Kubernetes 1.15 and Istio 1.3.1. Conversion webhook is installed to automatically convert v1alpha2 inference service to v1beta1.

:new: What's new ?
- You can now specify container fields on ML Framework spec such as env variable, liveness/readiness probes etc.
- You can now specify pod template fields on component spec such as NodeAffinity etc.
- gRPC support Tensorflow Serving.
- Triton Inference server V2 inference REST/gRPC protocol support
- SKLearn/XGBoost V2 inference REST/gRPC protocol support with MLServer
- PMMLServer support
- Allow specifying timeouts on component spec
- Simplified canary rollout, traffic split at knative revisions level instead of services level
- Transformer to predictor call is now made async

What's gone ?
- Default/Canary level is removed, canaryTrafficPercent is moved to the component level
- Parallelism field is renamed to containerConcurrency

What actions are needed to take to upgrade?
- Make sure canary traffic is all rolled out before upgrade as v1alpha2 canary spec is deprecated, please use v1beta1 spec for canary rollout feature.
- Although KFServing automatically converts the InferenceService to v1beta1, we recommend rewriting all your spec with V1beta1 API as we plan to drop the support for v1alpha2 in later versions.

Contribution list
* Make KFServer HTTP requests asynchronous 983 by salanki

* Add support for generic HTTP/HTTPS URI for Storage Initializer  979 by tduffy000

*



InferenceService v1beta1 API  991 by yuzisun

* Validation check for InferenceService Name 1079 by jazzsir

* Set KFServing default worker to 1  1106 by yuzliu

* Add support for MLServer in the SKLearn predictor  1155 by adriangonz

* Add V2 support to XGBoost predictor 1196 by adriangonz

* Support PMML server 1141 by AnyISalIn

* Generate SDK for KFServing v1beta1  1150 by jinchihe


* Support Kubernetes 1.18 1128 by pugangxa

Multi Model Serving V1Alpha1
:rainbow: KFServing 0.5 introduces Multi Model Serving with V1Alpha1 TrainedModel CR, this is currently for experiment only and we are looking for your feedbacks!

* Multi-Model Puller 989 by ifilonenko 

* Add multi model configmap 992 by wengyao04

* Trained model v1alpha1 api  1009 by yuzliu
* TrainedModel controller 1013 by yuzliu

* Harden model puller logic and add tests  1055 by yuzisun
* Puller streamlining/simplification 1057 by njhill
*
Integrate MMS inferenceservice controller, configmap controller, model agent 1132 by yuzliu
* Add load/unload endpoint for SKLearn/XGBoost KFServer 1082 by wengyao04

* Sync from model config on agent startup 1204 by yuzisun

Explanation

*
Add support for AIX360 explanations 1094 by drewbutlerbb4

* Alibi 0.5.5 1168 by cliveseldon

Documentation

* Docs/custom domain  1036 by adamkgray

* Update ingress gateway access instruction 1008 by yuzisun

* Document working k8s version 1062 by riklopfer
* Add triton torchscript example with prediction v2 protocol  1131 by yuzisun

* Add torchserve custom server with pv storage example 1182 by jagadeeshi2i 
* Add torchserve custom server example 1156 by jagadeeshi2i 
*
Add torchserve custom server bert sample 1185 by jagadeeshi2i 


* Bump up minimal Kube and Istio requirements  1166 by animeshsingh

Developer Experience

* Migrate controller tests to use BDD testing style 936 by yuzisun
*
Genericized component logic. 1018 by ellistarn

* Use github action for kfserving controller tests 1056 by yuzisun
* Make standalone installation kustomizable 1103 by jazzsir
*
Move KFServing CI to AWS 1170 by yuzisun
* Upgrade k8s and kn go library versions 1144 by ryandawsonuk

KFServing patch release on top of v0.4 to enable deployment on OpenShift. Fixes include

* Fixed issues on openshift (1122)
- change to use port 9443
- add rbac for finalizer
* Add inferenceservice finalizer rbac rules (1134)
* Fixes KFServing SDK 0.4 import error while running the custom built image (1117)

Action Required
- KFServing has added object selector on pod mutator webhook configuration which requires minimally Kubernetes 1.15 to take effect.
- The generated KFServing InferenceService openAPI schema validation now includes markers like `x-kubernetes-list-map-keys` and `x-kubernetes-map-type` which requires minimally Kubernetes 1.16, if you are on kubernetes 1.15 or lower version please install KFServing with `--validate=false` flag.
- Tensorrt inference server has been renamed to Triton inference server, if you are using `tensorrt` predictor on inference service yaml please rename to `triton`.
- KFserving has removed the default percentage based queue proxy resource limit due to 844, please set queue proxy requests/limits in the knative `config-deployment.yaml` config map which is introduced in knative 0.16 or add the queue proxy resource limit annotation if you are on lower version and your cluster has resource quota turned on, we highly recommend upgrading linux kernel if you are hitting the same [cpu throttling issue](kubernetes/kubernetes67577).
- The default S3 credential name has been updated to follow the convention from `awsAccessKeyID` and `awsSecretAccessKey` to `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, if you have secrets configured with the old way please update accordingly.
- KFServing has stopped maintaining the model server image versions in the configmap, user now can set the corresponding model server version on `runtimeVersion` field if you need the version different from the default.

New features
- Add batcher module as sidecar 847 zhangrongguo
- Add Default LivenessProbe to Tensorflow Predictor 925 salanki
- Remove framework image version list from configmap  917 yuzisun
- Record Events when InferenceService goes in and out of readiness state 876 ifilonenko
- Triton inference server rename and integrations 747 deadeyegoodwin
- Alibi explainer upgrade to 0.4.0 803 cliveseldon
- Make default request logger url more flexible 837 ryandawsonuk 
- Allow customized url paths on data plane 907 iamlovingit
- Add object selector for KFServing pod mutator webhook configuration 893 yuzisun
- Update logger to CloudEvents V1 protocol 886 cliveseldon
- Set ContainerConcurrency to Parallelism 806 salanki

Bug Fixes
- Disable retries in Istio VirtualService  807 salanki 
- Remove default queue proxy resource limit and Add KFServing benchmarking 894 yuzisun
- Enhance SDK watch API to avoid traceback  889 jinchihe
- Update KNative annotation when modifying minReplicas to 0 963 salanki
- Allow configurable region name when creating minio client 823 harshavardhana
- Return 503 from healthhandler when model is not ready 818 kolasanichaitanya
- Updated S3 credential variable names to commonly used en var names 704 karlschriek
- Fix duplicated volume issue when attaching GCS secret 766 kangwoo

Documentations
- Add BERT example for triton inference server integration 750 yuzisun
- Add KFServing Debugging guide 829 yuzisun
- Add new KFServing sample for GCP IAP 853 owennewo 
- Add KFServing on Kubeflow with Istio-Dex Example 821 822 sachua 
- Add Outlier Detection and Drift Detection Examples 764 cliveseldon
- Update pipeline sample to point to mnist e2e one  926 animeshsingh 
- Add custom gRPC sample  921 Iamlovingit
- Add custom inference example using BentoML 800 yubozhao
- Update KFServing roadmap for Q3/Q4 861 yuzisun

Developer Experience
- Migrate KFServing to Go Module  796 yuzisun
- Add tabular explainer e2e test 865 janeman yuzisun
- Add logger and improve batcher e2e tests 938 yuzisun

Features
- Pytorch model server with GPU inference 540
- Support internal mesh routing to inference service e.g routing from Kafka event source 583
- Add storage URI for transformer 643
- Add parallelism field to allow setting autoscaling target concurrency and number of tornado workers 637
- SKLearn model server to support pickled model 560
- Add extra information for Logger 699
- Default min replica to 1 instead of 0 655
- Upgrade knative API from v1alpha1 to v1 for KFServing 585
- Upgrade KFServing Kubernetes dependency 1.15 and knative dependency to 1.11 630
- Upgrade openapi-gen 600
- Expose containerPort to let knative listen on logger port, support logger for custom spec 592
- Self-signed certs generation script 650

Bug Fixes
- Fix default queue proxy container resource limit which was too low 608
- Allow configuring max buffer size for tornado server 665
- Relax data plane "instances" key validation 705
- Return application/json in response header 615
- Fix top level virtual service for HTTPS 726

Developer Experience, Tools & Testing, Examples
- Enable local development for model servers, explainer and storage initializer 591
- Add wait inference service SDK api 610
- Adding custom examples 678 698
- Add canary rollout examples 691
- Add e2e tests for canary rollout 658