Kuber

- Update to kubernetes 1.6 spec 169

API Change
- '`kube-apiserver`: adds `--authentication-config` flag for reading `AuthenticationConfiguration`
files. `--authentication-config` flag is mutually exclusive with the existing `--oidc-*`
flags.' ([kubernetes/kubernetes119142](https://github.com/kubernetes/kubernetes/pull/119142), [aramase](https://github.com/aramase))
- '`kube-scheduler` component config (`KubeSchedulerConfiguration`) `kubescheduler.config.k8s.io/v1beta3`
is removed in `v1.29`. Migrated `kube-scheduler` configuration files to `kubescheduler.config.k8s.io/v1`.' ([kubernetes/kubernetes119994](https://github.com/kubernetes/kubernetes/pull/119994), [SataQiu](https://github.com/SataQiu))
- A new sleep action for the `PreStop` lifecycle hook was added, allowing containers to pause for a specified duration before termination. ([kubernetes/kubernetes119026](https://github.com/kubernetes/kubernetes/pull/119026), [AxeZhan](https://github.com/AxeZhan))
- Added CEL expressions to `v1alpha1 AuthenticationConfiguration`. ([kubernetes/kubernetes121078](https://github.com/kubernetes/kubernetes/pull/121078), [aramase](https://github.com/aramase))
- Added Windows support for InPlace Pod Vertical Scaling feature. ([kubernetes/kubernetes112599](https://github.com/kubernetes/kubernetes/pull/112599), [fabi200123](https://github.com/fabi200123)) [SIG Autoscaling, Node, Scalability, Scheduling and Windows]
- Added `ImageMaximumGCAge` field to Kubelet configuration, which allows a user to set the maximum age an image is unused before it's garbage collected. ([kubernetes/kubernetes121275](https://github.com/kubernetes/kubernetes/pull/121275), [haircommander](https://github.com/haircommander))
- Added `UserNamespacesPodSecurityStandards` feature gate to enable user namespace support for Pod Security Standards.
Enabling this feature will modify all Pod Security Standard rules to allow setting: `spec[.*].securityContext.[runAsNonRoot,runAsUser]`.
This feature gate should only be enabled if all nodes in the cluster support the user namespace feature and have it enabled.
The feature gate will not graduate or be enabled by default in future Kubernetes releases. ([kubernetes/kubernetes118760](https://github.com/kubernetes/kubernetes/pull/118760), [saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery, Auth, Node and Release]
- Added `optionalOldSelf` to `x-kubernetes-validations` to support ratcheting CRD schema constraints. ([kubernetes/kubernetes121034](https://github.com/kubernetes/kubernetes/pull/121034), [alexzielenski](https://github.com/alexzielenski))
- Added a new `ServiceCIDR` type that allows to dynamically configure the cluster range used to allocate `Service ClusterIPs` addresses. ([kubernetes/kubernetes116516](https://github.com/kubernetes/kubernetes/pull/116516), [aojea](https://github.com/aojea))
- Added a new `ipMode` field to the `.status` of Services where `type` is set to `LoadBalancer`.
The new field is behind the `LoadBalancerIPMode` feature gate. ([kubernetes/kubernetes119937](https://github.com/kubernetes/kubernetes/pull/119937), [RyanAoh](https://github.com/RyanAoh)) [SIG API Machinery, Apps, Cloud Provider, Network and Testing]
- Added options for configuring `nf_conntrack_udp_timeout`, and `nf_conntrack_udp_timeout_stream` variables of netfilter conntrack subsystem. ([kubernetes/kubernetes120808](https://github.com/kubernetes/kubernetes/pull/120808), [aroradaman](https://github.com/aroradaman))
- Added support for CEL expressions to `v1alpha1 AuthorizationConfiguration` webhook `matchConditions`. ([kubernetes/kubernetes121223](https://github.com/kubernetes/kubernetes/pull/121223), [ritazh](https://github.com/ritazh))
- Added support for projecting `certificates.k8s.io/v1alpha1` ClusterTrustBundle objects into pods. ([kubernetes/kubernetes113374](https://github.com/kubernetes/kubernetes/pull/113374), [ahmedtd](https://github.com/ahmedtd))
- Added the `DisableNodeKubeProxyVersion` feature gate. If `DisableNodeKubeProxyVersion` is enabled, the `kubeProxyVersion` field is not set. ([kubernetes/kubernetes120954](https://github.com/kubernetes/kubernetes/pull/120954), [HirazawaUi](https://github.com/HirazawaUi))
- Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps.
The incorrect cost was evident when the result of a function was used in subsequent operations. ([kubernetes/kubernetes119800](https://github.com/kubernetes/kubernetes/pull/119800), [jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth and Cloud Provider]
- Fixed the API comments for the Job `Ready` field in status. ([kubernetes/kubernetes121765](https://github.com/kubernetes/kubernetes/pull/121765), [mimowo](https://github.com/mimowo))
- Fixed the API comments for the `FailIndex` Job pod failure policy action. ([kubernetes/kubernetes121764](https://github.com/kubernetes/kubernetes/pull/121764), [mimowo](https://github.com/mimowo))
- Go API: the `ResourceRequirements` struct was replaced with `VolumeResourceRequirements` for use with volumes. ([kubernetes/kubernetes118653](https://github.com/kubernetes/kubernetes/pull/118653), [pohly](https://github.com/pohly))
- Graduated `Job BackoffLimitPerIndex` feature to `beta`. ([kubernetes/kubernetes121356](https://github.com/kubernetes/kubernetes/pull/121356), [mimowo](https://github.com/mimowo))
- Marked the `onPodConditions` field as optional in `Job`'s pod failure policy. ([kubernetes/kubernetes120204](https://github.com/kubernetes/kubernetes/pull/120204), [mimowo](https://github.com/mimowo))
- Promoted `PodReadyToStartContainers` condition to `beta`. ([kubernetes/kubernetes119659](https://github.com/kubernetes/kubernetes/pull/119659), [kannon92](https://github.com/kannon92))
- The `flowcontrol.apiserver.k8s.io/v1beta3` `FlowSchema` and `PriorityLevelConfiguration` APIs has been promoted to `flowcontrol.apiserver.k8s.io/v1`, with the following changes:
- `PriorityLevelConfiguration`: the `.spec.limited.nominalConcurrencyShares` field defaults to `30` only if the field is omitted (v1beta3 also defaulted an explicit `0` value to `30`). Specifying an explicit `0` value is not allowed in the `v1` version in v1.29 to ensure compatibility with `v1.28` API servers. In `v1.30`, explicit `0` values will be allowed in this field in the `v1` API.
The `flowcontrol.apiserver.k8s.io/v1beta3` APIs are deprecated and will no longer be served in v1.32. All existing objects are available via the `v1` APIs. Transition clients and manifests to use the `v1` APIs before upgrading to `v1.32`. ([kubernetes/kubernetes121089](https://github.com/kubernetes/kubernetes/pull/121089), [tkashem](https://github.com/tkashem))
- The `kube-proxy` command-line documentation was updated to clarify that
`--bind-address` does not actually have anything to do with binding to an
address, and you probably don't actually want to be using it. ([kubernetes/kubernetes120274](https://github.com/kubernetes/kubernetes/pull/120274), [danwinship](https://github.com/danwinship))
- The `kube-scheduler` `selectorSpread` plugin has been removed, please use the `podTopologySpread` plugin instead. ([kubernetes/kubernetes117720](https://github.com/kubernetes/kubernetes/pull/117720), [kerthcet](https://github.com/kerthcet))
- The `matchLabelKeys/mismatchLabelKeys` feature is introduced to the hard/soft `PodAffinity/PodAntiAffinity`. ([kubernetes/kubernetes116065](https://github.com/kubernetes/kubernetes/pull/116065), [sanposhiho](https://github.com/sanposhiho))
- When updating a CRD, per-expression cost limit check are now skipped for `x-kubernetes-validations` rules of versions that are not mutated. ([kubernetes/kubernetes121460](https://github.com/kubernetes/kubernetes/pull/121460), [jiahuif](https://github.com/jiahuif))
- `CSINodeExpandSecret` feature has been promoted to `GA` in this release and is enabled
by default. The CSI drivers can make use of the `secretRef` values passed in `NodeExpansion`
request optionally sent by the CSI Client from this release onwards. ([kubernetes/kubernetes121303](https://github.com/kubernetes/kubernetes/pull/121303), [humblec](https://github.com/humblec))
- `NodeStageVolume` calls will now be retried if the CSI node driver is not running. ([kubernetes/kubernetes120330](https://github.com/kubernetes/kubernetes/pull/120330), [rohitssingh](https://github.com/rohitssingh))
- `PersistentVolumeLastPhaseTransitionTime` is now beta and enabled by default. ([kubernetes/kubernetes120627](https://github.com/kubernetes/kubernetes/pull/120627), [RomanBednar](https://github.com/RomanBednar))
- `ValidatingAdmissionPolicy` type checking now supports CRDs and API extensions types. ([kubernetes/kubernetes119109](https://github.com/kubernetes/kubernetes/pull/119109), [jiahuif](https://github.com/jiahuif))
- `kube-apiserver`: added `--authorization-config` flag for reading a configuration file containing an `apiserver.config.k8s.io/v1alpha1 AuthorizationConfiguration` object. The `--authorization-config` flag is mutually exclusive with `--authorization-modes` and `--authorization-webhook-*` flags. The `alpha` `StructuredAuthorizationConfiguration` feature flag must be enabled for `--authorization-config` to be specified. ([kubernetes/kubernetes120154](https://github.com/kubernetes/kubernetes/pull/120154), [palnabarun](https://github.com/palnabarun))
- `kube-proxy` now has a new nftables-based mode, available by running

`kube-proxy --feature-gates NFTablesProxyMode=true --proxy-mode nftables`

This is currently an alpha-level feature and while it probably will not
eat your data, it may nibble at it a bit. (It passes e2e testing but has
not yet seen real-world use.)

At this point it should be functionally mostly identical to the iptables
mode, except that it does not (and will not) support Service NodePorts on
127.0.0.1. (Also note that there are currently no command-line arguments
for the nftables-specific config; you will need to use a config file if
you want to set the equivalent of any of the `--iptables-xxx` options.)

As this code is still very new, it has not been heavily optimized yet;
while it is expected to _eventually_ have better performance than the
iptables backend, very little performance testing has been done so far. ([kubernetes/kubernetes121046](https://github.com/kubernetes/kubernetes/pull/121046), [danwinship](https://github.com/danwinship))
- `kube-proxy`: Added an option/flag for configuring the `nf_conntrack_tcp_be_liberal` sysctl (in the kernel's netfilter conntrack subsystem). When enabled, `kube-proxy` will not install the `DROP` rule for invalid conntrack states, which currently breaks users of asymmetric routing. ([kubernetes/kubernetes120354](https://github.com/kubernetes/kubernetes/pull/120354), [aroradaman](https://github.com/aroradaman))
- Added support for projecting certificates.k8s.io/v1alpha1 ClusterTrustBundle objects into pods. ([kubernetes/kubernetes113374](https://github.com/kubernetes/kubernetes/pull/113374), [ahmedtd](https://github.com/ahmedtd)) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
- Adds `optionalOldSelf` to `x-kubernetes-validations` to support ratcheting CRD schema constraints ([kubernetes/kubernetes121034](https://github.com/kubernetes/kubernetes/pull/121034), [alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery]
- Fix API comment for the Job Ready field in status ([kubernetes/kubernetes121765](https://github.com/kubernetes/kubernetes/pull/121765), [mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps]
- Fix API comments for the FailIndex Job pod failure policy action. ([kubernetes/kubernetes121764](https://github.com/kubernetes/kubernetes/pull/121764), [mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps]
- A new sleep action for the PreStop lifecycle hook is added, allowing containers to pause for a specified duration before termination. ([kubernetes/kubernetes119026](https://github.com/kubernetes/kubernetes/pull/119026), [AxeZhan](https://github.com/AxeZhan)) [SIG API Machinery, Apps, Node and Testing]
- Add ImageMaximumGCAge field to Kubelet configuration, which allows a user to set the maximum age an image is unused before it's garbage collected. ([kubernetes/kubernetes121275](https://github.com/kubernetes/kubernetes/pull/121275), [haircommander](https://github.com/haircommander)) [SIG API Machinery and Node]
- Add a new ServiceCIDR type that allows to dynamically configure the cluster range used to allocate Service ClusterIPs addresses ([kubernetes/kubernetes116516](https://github.com/kubernetes/kubernetes/pull/116516), [aojea](https://github.com/aojea)) [SIG API Machinery, Apps, Auth, CLI, Network and Testing]
- Add the DisableNodeKubeProxyVersion feature gate. If DisableNodeKubeProxyVersion is enabled, the kubeProxyVersion field is not set. ([kubernetes/kubernetes120954](https://github.com/kubernetes/kubernetes/pull/120954), [HirazawaUi](https://github.com/HirazawaUi)) [SIG API Machinery, Apps and Node]
- Added Windows support for InPlace Pod Vertical Scaling feature. ([kubernetes/kubernetes112599](https://github.com/kubernetes/kubernetes/pull/112599), [fabi200123](https://github.com/fabi200123)) [SIG Autoscaling, Node, Scalability, Scheduling and Windows]
- Added `UserNamespacesPodSecurityStandards` feature gate to enable user namespace support for Pod Security Standards.
Enabling this feature will modify all Pod Security Standard rules to allow setting: `spec[.*].securityContext.[runAsNonRoot,runAsUser]`.
This feature gate should only be enabled if all nodes in the cluster support the user namespace feature and have it enabled.
The feature gate will not graduate or be enabled by default in future Kubernetes releases. ([kubernetes/kubernetes118760](https://github.com/kubernetes/kubernetes/pull/118760), [saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery, Auth, Node and Release]
- Added options for configuring nf_conntrack_udp_timeout, and nf_conntrack_udp_timeout_stream variables of netfilter conntrack subsystem. ([kubernetes/kubernetes120808](https://github.com/kubernetes/kubernetes/pull/120808), [aroradaman](https://github.com/aroradaman)) [SIG API Machinery and Network]
- Adds CEL expressions to v1alpha1 AuthenticationConfiguration. ([kubernetes/kubernetes121078](https://github.com/kubernetes/kubernetes/pull/121078), [aramase](https://github.com/aramase)) [SIG API Machinery, Auth and Testing]
- Adds support for CEL expressions to v1alpha1 AuthorizationConfiguration webhook matchConditions. ([kubernetes/kubernetes121223](https://github.com/kubernetes/kubernetes/pull/121223), [ritazh](https://github.com/ritazh)) [SIG API Machinery and Auth]
- CSINodeExpandSecret feature has been promoted to GA in this release and enabled by default. The CSI drivers can make use of the `secretRef` values passed in NodeExpansion request optionally sent by the CSI Client from this release onwards. ([kubernetes/kubernetes121303](https://github.com/kubernetes/kubernetes/pull/121303), [humblec](https://github.com/humblec)) [SIG API Machinery, Apps and Storage]
- Graduate Job BackoffLimitPerIndex feature to Beta ([kubernetes/kubernetes121356](https://github.com/kubernetes/kubernetes/pull/121356), [mimowo](https://github.com/mimowo)) [SIG Apps]
- Kube-apiserver: adds --authorization-config flag for reading a configuration file containing an apiserver.config.k8s.io/v1alpha1 AuthorizationConfiguration object. --authorization-config flag is mutually exclusive with --authorization-modes and --authorization-webhook-* flags. The alpha StructuredAuthorizationConfiguration feature flag must be enabled for --authorization-config to be specified. ([kubernetes/kubernetes120154](https://github.com/kubernetes/kubernetes/pull/120154), [palnabarun](https://github.com/palnabarun)) [SIG API Machinery, Auth and Testing]
- Kube-proxy now has a new nftables-based mode, available by running

kube-proxy --feature-gates NFTablesProxyMode=true --proxy-mode nftables

This is currently an alpha-level feature and while it probably will not
eat your data, it may nibble at it a bit. (It passes e2e testing but has
not yet seen real-world use.)

At this point it should be functionally mostly identical to the iptables
mode, except that it does not (and will not) support Service NodePorts on
127.0.0.1. (Also note that there are currently no command-line arguments
for the nftables-specific config; you will need to use a config file if
you want to set the equivalent of any of the `--iptables-xxx` options.)

As this code is still very new, it has not been heavily optimized yet;
while it is expected to _eventually_ have better performance than the
iptables backend, very little performance testing has been done so far. ([kubernetes/kubernetes121046](https://github.com/kubernetes/kubernetes/pull/121046), [danwinship](https://github.com/danwinship)) [SIG API Machinery and Network]
- Kube-proxy: Added an option/flag for configuring the `nf_conntrack_tcp_be_liberal` sysctl (in the kernel's netfilter conntrack subsystem). When enabled, kube-proxy will not install the DROP rule for invalid conntrack states, which currently breaks users of asymmetric routing. ([kubernetes/kubernetes120354](https://github.com/kubernetes/kubernetes/pull/120354), [aroradaman](https://github.com/aroradaman)) [SIG API Machinery and Network]
- PersistentVolumeLastPhaseTransitionTime is now beta, enabled by default. ([kubernetes/kubernetes120627](https://github.com/kubernetes/kubernetes/pull/120627), [RomanBednar](https://github.com/RomanBednar)) [SIG Storage]
- Promote PodReadyToStartContainers condition to beta. ([kubernetes/kubernetes119659](https://github.com/kubernetes/kubernetes/pull/119659), [kannon92](https://github.com/kannon92)) [SIG Node and Testing]
- The flowcontrol.apiserver.k8s.io/v1beta3 FlowSchema and PriorityLevelConfiguration APIs has been promoted to flowcontrol.apiserver.k8s.io/v1, with the following changes:
- PriorityLevelConfiguration: the `.spec.limited.nominalConcurrencyShares` field defaults to `30` only if the field is omitted (v1beta3 also defaulted an explicit `0` value to `30`). Specifying an explicit `0` value is not allowed in the `v1` version in v1.29 to ensure compatibility with 1.28 API servers. In v1.30, explicit `0` values will be allowed in this field in the `v1` API.
The flowcontrol.apiserver.k8s.io/v1beta3 APIs are deprecated and will no longer be served in v1.32. All existing objects are available via the `v1` APIs. Transition clients and manifests to use the `v1` APIs before upgrading to v1.32. ([kubernetes/kubernetes121089](https://github.com/kubernetes/kubernetes/pull/121089), [tkashem](https://github.com/tkashem)) [SIG API Machinery and Testing]
- The kube-proxy command-line documentation was updated to clarify that
`--bind-address` does not actually have anything to do with binding to an
address, and you probably don't actually want to be using it. ([kubernetes/kubernetes120274](https://github.com/kubernetes/kubernetes/pull/120274), [danwinship](https://github.com/danwinship)) [SIG Network]
- The matchLabelKeys/mismatchLabelKeys feature is introduced to the hard/soft PodAffinity/PodAntiAffinity. ([kubernetes/kubernetes116065](https://github.com/kubernetes/kubernetes/pull/116065), [sanposhiho](https://github.com/sanposhiho)) [SIG API Machinery, Apps, Cloud Provider, Scheduling and Testing]
- ValidatingAdmissionPolicy Type Checking now supports CRDs and API extensions types. ([kubernetes/kubernetes119109](https://github.com/kubernetes/kubernetes/pull/119109), [jiahuif](https://github.com/jiahuif)) [SIG API Machinery, Apps, Auth and Testing]
- When updating a CRD, per-expression cost limit check is skipped for x-kubernetes-validations rules of versions that are not mutated. ([kubernetes/kubernetes121460](https://github.com/kubernetes/kubernetes/pull/121460), [jiahuif](https://github.com/jiahuif)) [SIG API Machinery]
- Added a new `ipMode` field to the `.status` of Services where `type` is set to `LoadBalancer`.
The new field is behind the `LoadBalancerIPMode` feature gate. ([kubernetes/kubernetes119937](https://github.com/kubernetes/kubernetes/pull/119937), [RyanAoh](https://github.com/RyanAoh)) [SIG API Machinery, Apps, Cloud Provider, Network and Testing]
- Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps.
The incorrect cost was evident when the result of a function was used in subsequent operations. ([kubernetes/kubernetes119800](https://github.com/kubernetes/kubernetes/pull/119800), [jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth and Cloud Provider]
- Go API: the ResourceRequirements struct needs to be replaced with VolumeResourceRequirements for use with volumes. ([kubernetes/kubernetes118653](https://github.com/kubernetes/kubernetes/pull/118653), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node, Scheduling, Storage and Testing]
- Kube-apiserver: adds --authentication-config flag for reading AuthenticationConfiguration files. --authentication-config flag is mutually exclusive with the existing --oidc-* flags. ([kubernetes/kubernetes119142](https://github.com/kubernetes/kubernetes/pull/119142), [aramase](https://github.com/aramase)) [SIG API Machinery, Auth and Testing]
- Kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta3 is removed in v1.29. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. ([kubernetes/kubernetes119994](https://github.com/kubernetes/kubernetes/pull/119994), [SataQiu](https://github.com/SataQiu)) [SIG Scheduling and Testing]
- Mark the onPodConditions field as optional in Job's pod failure policy. ([kubernetes/kubernetes120204](https://github.com/kubernetes/kubernetes/pull/120204), [mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps]
- Retry NodeStageVolume calls if CSI node driver is not running ([kubernetes/kubernetes120330](https://github.com/kubernetes/kubernetes/pull/120330), [rohitssingh](https://github.com/rohitssingh)) [SIG Apps, Storage and Testing]
- The kube-scheduler `selectorSpread` plugin has been removed, please use the `podTopologySpread` plugin instead. ([kubernetes/kubernetes117720](https://github.com/kubernetes/kubernetes/pull/117720), [kerthcet](https://github.com/kerthcet)) [SIG Scheduling]

API Change
- Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps.
The incorrect cost was evident when the result of a function was used in subsequent operations. ([kubernetes/kubernetes119807](https://github.com/kubernetes/kubernetes/pull/119807), [jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth and Cloud Provider]
- Mark Job onPodConditions as optional in pod failure policy ([kubernetes/kubernetes120208](https://github.com/kubernetes/kubernetes/pull/120208), [mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps]

API Change
- A CDIDevice field is included in the Device Plugin's `ContainerAllocateResponse`. This field maps to the CDIDevice field in the CRI protocol. ([kubernetes/kubernetes118254](https://github.com/kubernetes/kubernetes/pull/118254), [elezar](https://github.com/elezar)) [SIG Node and Testing]
- ACTION_REQUIRED
When an Indexed Job has a number of completions higher than 10^5 and parallelism higher than 10^4, and a big number of Indexes fail, Kubernetes might not be able to track the termination of the Job. Kubernetes now emits a warning, at Job creation, when the Job manifest exceeds both of these limits. ([kubernetes/kubernetes118420](https://github.com/kubernetes/kubernetes/pull/118420), [alculquicondor](https://github.com/alculquicondor)) [SIG Apps]
- Added `ServedVersions` field to `StorageVersion` API. ([kubernetes/kubernetes118386](https://github.com/kubernetes/kubernetes/pull/118386), [Richabanker](https://github.com/Richabanker))
- Added `IP mode` field to loadbalancer status ingress. ([kubernetes/kubernetes118895](https://github.com/kubernetes/kubernetes/pull/118895), [RyanAoh](https://github.com/RyanAoh))
- Added `podReplacementPolicy` and terminating field to job api. ([kubernetes/kubernetes119301](https://github.com/kubernetes/kubernetes/pull/119301), [kannon92](https://github.com/kannon92))
- Added a new `namespaceParamRef` field to `admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy`. ([kubernetes/kubernetes119215](https://github.com/kubernetes/kubernetes/pull/119215), [alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery and Testing]
- Added a warning that TLS 1.3 ciphers are not configurable. ([kubernetes/kubernetes115399](https://github.com/kubernetes/kubernetes/pull/115399), [3u13r](https://github.com/3u13r)) [SIG API Machinery and Node]
- Added error handling for seccomp localhost configurations that do not properly set a `localhostProfile`. ([kubernetes/kubernetes117020](https://github.com/kubernetes/kubernetes/pull/117020), [cji](https://github.com/cji))
- Added fields `reason` and `fieldPath` into CRD validation rules to allow users to specify reason and field path when validation failed. ([kubernetes/kubernetes118041](https://github.com/kubernetes/kubernetes/pull/118041), [cici37](https://github.com/cici37)) [SIG API Machinery]
- Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a `namespaceObject`
variable with expressions. ([kubernetes/kubernetes118267](https://github.com/kubernetes/kubernetes/pull/118267), [cici37](https://github.com/cici37)) [SIG API Machinery and Testing]
- Added new `CRDValidationRatcheting` alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. ([kubernetes/kubernetes118990](https://github.com/kubernetes/kubernetes/pull/118990), [alexzielenski](https://github.com/alexzielenski))
- Added new annotation `batch.kubernetes.io/cronjob-scheduled-timestamp` to Job objects scheduled from CronJobs. ([kubernetes/kubernetes118137](https://github.com/kubernetes/kubernetes/pull/118137), [helayoty](https://github.com/helayoty))
- Added new config option `delayCacheUntilActive` to `KubeSchedulerConfiguration` that can provide a tradeoff between memory efficiency and scheduling speed when their leadership is updated in `kube-scheduler` ([kubernetes/kubernetes115754](https://github.com/kubernetes/kubernetes/pull/115754), [linxiulei](https://github.com/linxiulei)) [SIG API Machinery and Scheduling]
- Changed how KMS v2 encryption at rest can generate data encryption keys.
When you enable the `KMSv2KDF` feature gate (off by default), KMS v2 uses a key derivation function to generate single use data encryption keys from a secret seed combined with some random data. This eliminates the need for a counter based nonce while avoiding nonce collision concerns associated with AES-GCM's 12 byte nonce. ([kubernetes/kubernetes118828](https://github.com/kubernetes/kubernetes/pull/118828), [enj](https://github.com/enj))
- Exposed `rest.DefaultServerUrlFor` function. ([kubernetes/kubernetes118055](https://github.com/kubernetes/kubernetes/pull/118055), [timofurrer](https://github.com/timofurrer))
- Extended the Job API for alpha version of `BackoffLimitPerIndex`. ([kubernetes/kubernetes119294](https://github.com/kubernetes/kubernetes/pull/119294), [mimowo](https://github.com/mimowo))
- Graduated `AdmissionWebhookMatchCondition` feature to beta. ([kubernetes/kubernetes119380](https://github.com/kubernetes/kubernetes/pull/119380), [a-hilaly](https://github.com/a-hilaly))
- If using cgroups v2, then the cgroup aware OOM killer will be enabled for container cgroups via `memory.oom.group` . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup. ([kubernetes/kubernetes117793](https://github.com/kubernetes/kubernetes/pull/117793), [tzneal](https://github.com/tzneal)) [SIG Apps, Node and Testing]
- In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . ([kubernetes/kubernetes118782](https://github.com/kubernetes/kubernetes/pull/118782), [MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery]
- Indexed Job pods now have the pod completion index set as a pod label. ([kubernetes/kubernetes118883](https://github.com/kubernetes/kubernetes/pull/118883), [danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps]
- Kube-proxy: added `--logging-format` flag to support structured logging. ([kubernetes/kubernetes117800](https://github.com/kubernetes/kubernetes/pull/117800), [cyclinder](https://github.com/cyclinder))
- NodeVolumeLimits implement the `PreFilter` extension point for skipping the Filter phase if the Pod doesn't use volumes with limits. ([kubernetes/kubernetes115398](https://github.com/kubernetes/kubernetes/pull/115398), [tangwz](https://github.com/tangwz)) [SIG Scheduling]
- PersistentVolumes have a new `LastPhaseTransitionTime` field which holds a timestamp of when the volume last transitioned its phase. ([kubernetes/kubernetes116469](https://github.com/kubernetes/kubernetes/pull/116469), [RomanBednar](https://github.com/RomanBednar))
- Pods which set `hostNetwork: true` and declare ports, get the `hostPort` field set automatically. Previously this would happen in the PodTemplate of a Deployment, DaemonSet or other workload API. Now `hostPort` will only be set when an actual Pod is being created. If this presents a problem, setting the feature gate "DefaultHostNetworkHostPortsInPodTemplates" to true will revert this behavior. Please file a kubernetes bug if you need to do this. ([kubernetes/kubernetes117696](https://github.com/kubernetes/kubernetes/pull/117696), [thockin](https://github.com/thockin)) [SIG Apps]
- Promoted API groups `ValidatingAdmissionPolicy` and `ValidatingAdmissionPolicyBinding` to `v1beta1`. ([kubernetes/kubernetes118644](https://github.com/kubernetes/kubernetes/pull/118644), [alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Apps and Testing]
- Promoted the feature gate `ValidtaingAdmissionPolicy` to beta, and it is turned off by default. ([kubernetes/kubernetes119409](https://github.com/kubernetes/kubernetes/pull/119409), [alexzielenski](https://github.com/alexzielenski))
- Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to `BETA` stability. ([kubernetes/kubernetes119264](https://github.com/kubernetes/kubernetes/pull/119264), [logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]
- Removed `resizeStatus` enum from `pvc.Status` and replaced with `AllocatedResourceStatus`. ([kubernetes/kubernetes116335](https://github.com/kubernetes/kubernetes/pull/116335), [gnufied](https://github.com/gnufied)) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
- Removed `WindowsHostProcessContainers` feature-gate. ([kubernetes/kubernetes117570](https://github.com/kubernetes/kubernetes/pull/117570), [marosset](https://github.com/marosset)) [SIG API Machinery, Apps, Auth, Node and Windows]
- Revised the comment about the feature-gate level for `PodFailurePolicy` from alpha to beta. ([kubernetes/kubernetes117802](https://github.com/kubernetes/kubernetes/pull/117802), [kerthcet](https://github.com/kerthcet)) [SIG API Machinery and Apps]
- StatefulSet pods now have the pod index set as a pod label `statefulset.kubernetes.io/pod-index`. ([kubernetes/kubernetes119232](https://github.com/kubernetes/kubernetes/pull/119232), [danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps]
- Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver ([kubernetes/kubernetes117740](https://github.com/kubernetes/kubernetes/pull/117740), [Richabanker](https://github.com/Richabanker)) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing]
- Supported `BackoffLimitPerIndex` in Jobs. ([kubernetes/kubernetes118009](https://github.com/kubernetes/kubernetes/pull/118009), [mimowo](https://github.com/mimowo))
- The `IPTablesOwnershipCleanup` feature (KEP-3178) is now GA; kubelet no longer
creates the `KUBE-MARK-DROP` chain (which has been unused for several releases)
or the `KUBE-MARK-MASQ` chain (which is now only created by kube-proxy). ([kubernetes/kubernetes119374](https://github.com/kubernetes/kubernetes/pull/119374), [danwinship](https://github.com/danwinship))
- The `SelfSubjectReview` API is promoted to `authentication.k8s.io/v1` and the `kubectl auth whoami` command is GA. ([kubernetes/kubernetes117713](https://github.com/kubernetes/kubernetes/pull/117713), [nabokihms](https://github.com/nabokihms)) [SIG API Machinery, Architecture, Auth, CLI and Testing]
- The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still `<pod>-<claim name>`, but a random suffix will avoid name collisions. ([kubernetes/kubernetes117351](https://github.com/kubernetes/kubernetes/pull/117351), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
- The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. ([kubernetes/kubernetes116429](https://github.com/kubernetes/kubernetes/pull/116429), [gjkim42](https://github.com/gjkim42)) [SIG API Machinery, Apps, Node, Scheduling and Testing]
- Updated the comment about the feature-gate level for `PodFailurePolicy` from alpha to beta ([kubernetes/kubernetes118278](https://github.com/kubernetes/kubernetes/pull/118278), [mimowo](https://github.com/mimowo))
- `client-go`: Improved memory use of reflector caches when watching large numbers
of objects which do not change frequently. ([kubernetes/kubernetes113362](https://github.com/kubernetes/kubernetes/pull/113362), [sxllwx](https://github.com/sxllwx))
- `component-base/logs` is now stricter about not applying configurations multiple
times and will return an error when that is attempted. Can be overridden by binaries
which need to do that. ([kubernetes/kubernetes117108](https://github.com/kubernetes/kubernetes/pull/117108), [pohly](https://github.com/pohly))
- `kube-controller-manager`: The `LegacyServiceAccountTokenCleanUp` feature gate
is now available as alpha (off by default). When enabled, the `legacy-service-account-token-cleaner`
controller loop removes service account token secrets that have not been used
in the time specified by `--legacy-service-account-token-clean-up-period` (defaulting
to one year), **and are** referenced from the `.secrets` list of a ServiceAccount
object, **and are not** referenced from pods. ([kubernetes/kubernetes115554](https://github.com/kubernetes/kubernetes/pull/115554), [yt2985](https://github.com/yt2985))
- `kube-scheduler` component config (KubeSchedulerConfiguration) `kubescheduler.config.k8s.io/v1beta2`
is removed in `v1.28`. Migrate `kube-scheduler` configuration files to `kubescheduler.config.k8s.io/v1`. ([kubernetes/kubernetes117649](https://github.com/kubernetes/kubernetes/pull/117649), [SataQiu](https://github.com/SataQiu))
- Aggregated discovery now returns `responseKind: {}` for resources which are missing group/version/kind information, to ensure compatibility with v0.26.0-v0.26.3 clients. ([kubernetes/kubernetes119835](https://github.com/kubernetes/kubernetes/pull/119835), [liggitt](https://github.com/liggitt)) [SIG API Machinery and Testing]
- Fix CustomResourceDefinition status.storedVersions validation error messages. ([kubernetes/kubernetes119653](https://github.com/kubernetes/kubernetes/pull/119653), [sttts](https://github.com/sttts)) [SIG API Machinery]
- Kube-proxy in Kubernetes >= 1.28 up until v1.28.0-beta.0 ignored the `-v` command line flag when combined with `--config`. ([kubernetes/kubernetes119867](https://github.com/kubernetes/kubernetes/pull/119867), [pohly](https://github.com/pohly)) [SIG Network]
- PersistentVolumes have a new LastPhaseTransitionTime field which holds a timestamp of when the volume last transitioned its phase. ([kubernetes/kubernetes116469](https://github.com/kubernetes/kubernetes/pull/116469), [RomanBednar](https://github.com/RomanBednar)) [SIG API Machinery, Apps, Auth, Node, Release, Storage and Testing]
- Promoted API groups `ValidatingAdmissionPolicy` and `ValidatingAdmissionPolicyBinding` to `v1beta1`. ([kubernetes/kubernetes118644](https://github.com/kubernetes/kubernetes/pull/118644), [alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Apps and Testing]
- Promoted the feature gate `ValidtaingAdmissionPolicy` to beta and it is turned off by default. ([kubernetes/kubernetes119409](https://github.com/kubernetes/kubernetes/pull/119409), [alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Apps, Auth, Instrumentation, Node, Release, Storage and Testing]
- Changed how KMS v2 encryption at rest can generate data encryption keys. When you enable the `KMSv2KDF` feature gate (off by default), KMS v2 uses a key derivation function to generate single use data encryption keys from a secret seed combined with some random data. This eliminates the need for a counter based nonce while avoiding nonce collision concerns associated with AES-GCM's 12 byte nonce. ([kubernetes/kubernetes118828](https://github.com/kubernetes/kubernetes/pull/118828), [enj](https://github.com/enj)) [SIG API Machinery, Auth and Testing]
- A CDIDevice field is includes in the Device Plugin's `ContainerAllocateResponse`. This field maps to the CDIDevice field in the CRI protocol. ([kubernetes/kubernetes118254](https://github.com/kubernetes/kubernetes/pull/118254), [elezar](https://github.com/elezar)) [SIG Node and Testing]
- Add new annotation `batch.kubernetes.io/cronjob-scheduled-timestamp` to Job objects scheduled from CronJobs. ([kubernetes/kubernetes118137](https://github.com/kubernetes/kubernetes/pull/118137), [helayoty](https://github.com/helayoty)) [SIG Apps]
- Add podReplacementPolicy and terminating field to job api ([kubernetes/kubernetes119301](https://github.com/kubernetes/kubernetes/pull/119301), [kannon92](https://github.com/kannon92)) [SIG API Machinery and Apps]
- Added fields `reason` and `fieldPath` into CRD validation rules to allow users to specify reason and field path when validation failed. ([kubernetes/kubernetes118041](https://github.com/kubernetes/kubernetes/pull/118041), [cici37](https://github.com/cici37)) [SIG API Machinery]
- Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a `namespaceObject`
variable with expressions. ([kubernetes/kubernetes118267](https://github.com/kubernetes/kubernetes/pull/118267), [cici37](https://github.com/cici37)) [SIG API Machinery and Testing]
- Adds new CRDValidationRatcheting alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. ([kubernetes/kubernetes118990](https://github.com/kubernetes/kubernetes/pull/118990), [alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
- Adds new namespaceParamRef to admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy ([kubernetes/kubernetes119215](https://github.com/kubernetes/kubernetes/pull/119215), [alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery and Testing]
- Extend the Job API for alpha version of BackoffLimitPerIndex ([kubernetes/kubernetes119294](https://github.com/kubernetes/kubernetes/pull/119294), [mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps]
- Graduate `AdmissionWebhookMatchCondition` feature to beta ([kubernetes/kubernetes119380](https://github.com/kubernetes/kubernetes/pull/119380), [a-hilaly](https://github.com/a-hilaly)) [SIG API Machinery]
- In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . ([kubernetes/kubernetes118782](https://github.com/kubernetes/kubernetes/pull/118782), [MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery]
- Indexed Job pods now have the pod completion index set as a pod label. ([kubernetes/kubernetes118883](https://github.com/kubernetes/kubernetes/pull/118883), [danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps]
- Kube-proxy: add '--logging-format' flag to support structured logging ([kubernetes/kubernetes117800](https://github.com/kubernetes/kubernetes/pull/117800), [cyclinder](https://github.com/cyclinder)) [SIG API Machinery, Architecture, Instrumentation and Network]
- Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to `BETA` stability. ([kubernetes/kubernetes119264](https://github.com/kubernetes/kubernetes/pull/119264), [logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]
- Removed `resizeStatus` enum from `pvc.Status` and replaced with `AllocatedResourceStatus` ([kubernetes/kubernetes116335](https://github.com/kubernetes/kubernetes/pull/116335), [gnufied](https://github.com/gnufied)) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
- StatefulSet pods now have the pod index set as a pod label `statefulset.kubernetes.io/pod-index`. ([kubernetes/kubernetes119232](https://github.com/kubernetes/kubernetes/pull/119232), [danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps]
- Support BackoffLimitPerIndex in Jobs ([kubernetes/kubernetes118009](https://github.com/kubernetes/kubernetes/pull/118009), [mimowo](https://github.com/mimowo)) [SIG API Machinery, Apps and Testing]
- Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver ([kubernetes/kubernetes117740](https://github.com/kubernetes/kubernetes/pull/117740), [Richabanker](https://github.com/Richabanker)) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing]
- The IPTablesOwnershipCleanup feature (KEP-3178) is now GA; kubelet no longer
creates the KUBE-MARK-DROP chain (which has been unused for several releases)
or the KUBE-MARK-MASQ chain (which is now only created by kube-proxy). ([kubernetes/kubernetes119374](https://github.com/kubernetes/kubernetes/pull/119374), [danwinship](https://github.com/danwinship)) [SIG API Machinery, Network and Node]
- The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still `<pod>-<claim name>`, but a random suffix will avoid name collisions. ([kubernetes/kubernetes117351](https://github.com/kubernetes/kubernetes/pull/117351), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
- The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. ([kubernetes/kubernetes116429](https://github.com/kubernetes/kubernetes/pull/116429), [gjkim42](https://github.com/gjkim42)) [SIG API Machinery, Apps, Node, Scheduling and Testing]
- Add ServedVersions field to StorageVersion API ([kubernetes/kubernetes118386](https://github.com/kubernetes/kubernetes/pull/118386), [Richabanker](https://github.com/Richabanker)) [SIG API Machinery and Testing]
- Component-base/logs is now more strict about not applying configurations multiple times and will return an error when that is attempted. Can be overridden by binaries which need to do that. ([kubernetes/kubernetes117108](https://github.com/kubernetes/kubernetes/pull/117108), [pohly](https://github.com/pohly)) [SIG API Machinery, Architecture, Cloud Provider, Instrumentation, Scheduling and Testing]
- ACTION_REQUIRED
When an Indexed Job has a number of completions higher than 10^5 and parallelism higher than 10^4, and a big number of Indexes fail, Kubernetes might not be able to track the termination of the Job. Kubernetes now emits a warning, at Job creation, when the Job manifest exceeds both of these limits. ([kubernetes/kubernetes118420](https://github.com/kubernetes/kubernetes/pull/118420), [alculquicondor](https://github.com/alculquicondor)) [SIG Apps]
- Expose rest.DefaultServerUrlFor function ([kubernetes/kubernetes118055](https://github.com/kubernetes/kubernetes/pull/118055), [timofurrer](https://github.com/timofurrer)) [SIG API Machinery]
- If using cgroups v2, then the cgroup aware OOM killer will be enabled for container cgroups via `memory.oom.group` . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup. ([kubernetes/kubernetes117793](https://github.com/kubernetes/kubernetes/pull/117793), [tzneal](https://github.com/tzneal)) [SIG Apps, Node and Testing]
- Update the comment about the feature-gate level for PodFailurePolicy from alpha to beta ([kubernetes/kubernetes118278](https://github.com/kubernetes/kubernetes/pull/118278), [mimowo](https://github.com/mimowo)) [SIG Apps]
- Added a warning that TLS 1.3 ciphers are not configurable. ([kubernetes/kubernetes115399](https://github.com/kubernetes/kubernetes/pull/115399), [3u13r](https://github.com/3u13r)) [SIG API Machinery and Node]
- Added error handling for seccomp localhost configurations that do not properly set a localhostProfile ([kubernetes/kubernetes117020](https://github.com/kubernetes/kubernetes/pull/117020), [cji](https://github.com/cji)) [SIG API Machinery and Node]
- Added new config option `delayCacheUntilActive` to `KubeSchedulerConfiguration` that can provide a tradeoff between memory efficiency and scheduling speed when their leadership is updated in `kube-scheduler` ([kubernetes/kubernetes115754](https://github.com/kubernetes/kubernetes/pull/115754), [linxiulei](https://github.com/linxiulei)) [SIG API Machinery and Scheduling]
- Client-go: Improved memory use of reflector caches when watching large numbers of objects which do not change frequently ([kubernetes/kubernetes113362](https://github.com/kubernetes/kubernetes/pull/113362), [sxllwx](https://github.com/sxllwx)) [SIG API Machinery]
- Kube-controller-manager: The `LegacyServiceAccountTokenCleanUp` feature gate is now available as alpha (off by default). When enabled, the `legacy-service-account-token-cleaner` controller loop removes service account token secrets that have not been used in the time specified by `--legacy-service-account-token-clean-up-period` (defaulting to one year), **and are** referenced from the `.secrets` list of a ServiceAccount object, **and are not** referenced from pods. ([kubernetes/kubernetes115554](https://github.com/kubernetes/kubernetes/pull/115554), [yt2985](https://github.com/yt2985)) [SIG API Machinery, Apps, Auth, Release and Testing]
- Kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta2 is removed in v1.28. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. ([kubernetes/kubernetes117649](https://github.com/kubernetes/kubernetes/pull/117649), [SataQiu](https://github.com/SataQiu)) [SIG API Machinery, Scheduling and Testing]
- NodeVolumeLimits implement the PreFilter extension point for skipping the Filter phase if the Pod doesn't use volumes with limits. ([kubernetes/kubernetes115398](https://github.com/kubernetes/kubernetes/pull/115398), [tangwz](https://github.com/tangwz)) [SIG Scheduling]
- Pods which set `hostNetwork: true` and declare ports get the `hostPort` field set automatically. Previously this would happen in the PodTemplate of a Deployment, DaemonSet or other workload API. Now `hostPort` will only be set when an actual Pod is being created. If this presents a problem, setting the feature gate "DefaultHostNetworkHostPortsInPodTemplates" to true will revert this behavior. Please file a kubernetes bug if you need to do this. ([kubernetes/kubernetes117696](https://github.com/kubernetes/kubernetes/pull/117696), [thockin](https://github.com/thockin)) [SIG Apps]
- Removing WindowsHostProcessContainers feature-gate ([kubernetes/kubernetes117570](https://github.com/kubernetes/kubernetes/pull/117570), [marosset](https://github.com/marosset)) [SIG API Machinery, Apps, Auth, Node and Windows]
- Revised the comment about the feature-gate level for PodFailurePolicy from alpha to beta ([kubernetes/kubernetes117802](https://github.com/kubernetes/kubernetes/pull/117802), [kerthcet](https://github.com/kerthcet)) [SIG API Machinery and Apps]
- The `SelfSubjectReview` API is promoted to `authentication.k8s.io/v1` and the `kubectl auth whoami` command is GA. ([kubernetes/kubernetes117713](https://github.com/kubernetes/kubernetes/pull/117713), [nabokihms](https://github.com/nabokihms)) [SIG API Machinery, Architecture, Auth, CLI and Testing]

Documentation
- Fix request_timeout example and doc. Arg name should be _request_timeout. Single value type should be int or long. (2071, hemslo)

API Change
- Added error handling for seccomp localhost configurations that do not properly set a localhostProfile ([kubernetes/kubernetes117020](https://github.com/kubernetes/kubernetes/pull/117020), [cji](https://github.com/cji)) [SIG API Machinery and Node]
- Fixed an issue where kubelet does not set case-insensitive headers for http probes. (117182, dddddai) ([kubernetes/kubernetes117324](https://github.com/kubernetes/kubernetes/pull/117324), [dddddai](https://github.com/dddddai)) [SIG API Machinery, Apps and Node]
- Revised the comment about the feature-gate level for PodFailurePolicy from alpha to beta ([kubernetes/kubernetes117815](https://github.com/kubernetes/kubernetes/pull/117815), [kerthcet](https://github.com/kerthcet)) [SIG Apps]
- A fix in the `resource.k8s.io/v1alpha1/ResourceClaim` API avoids harmless (?) ".status.reservedFor: element 0: associative list without keys has an element that's a map type" errors in the apiserver. Validation now rejects the incorrect reuse of the same UID in different entries. ([kubernetes/kubernetes115354](https://github.com/kubernetes/kubernetes/pull/115354), [pohly](https://github.com/pohly))
- A terminating pod on a node that is not caused by preemption no longer prevents `kube-scheduler` from preempting pods on that node
- Rename `PreemptionByKubeScheduler` to `PreemptionByScheduler` ([kubernetes/kubernetes114623](https://github.com/kubernetes/kubernetes/pull/114623), [Huang-Wei](https://github.com/Huang-Wei))
- API: resource.k8s.io/v1alpha1.PodScheduling was renamed to resource.k8s.io/v1alpha2.PodSchedulingContext. ([kubernetes/kubernetes116556](https://github.com/kubernetes/kubernetes/pull/116556), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, CLI, Node, Scheduling and Testing]
- Added CEL runtime cost calculation into ValidatingAdmissionPolicy, matching the evaluation cost
restrictions that already apply to CustomResourceDefinition.
If rule evaluation uses more compute than the limit, the API server aborts the evaluation and the
admission check that was being performed is aborted; the `failurePolicy` for the ValidatingAdmissionPolicy
determines the outcome. ([kubernetes/kubernetes115747](https://github.com/kubernetes/kubernetes/pull/115747), [cici37](https://github.com/cici37))
- Added `auditAnnotations` to `ValidatingAdmissionPolicy`, enabling CEL to be used to add audit annotations to request audit events.
Added `validationActions` to `ValidatingAdmissionPolicyBinding`, enabling validation failures to be handled by any combination of the warn, audit and deny enforcement actions. ([kubernetes/kubernetes115973](https://github.com/kubernetes/kubernetes/pull/115973), [jpbetz](https://github.com/jpbetz))
- Added `messageExpression` field to `ValidationRule`. ([kubernetes/kubernetes115969](https://github.com/kubernetes/kubernetes/pull/115969), [DangerOnTheRanger](https://github.com/DangerOnTheRanger))
- Added `messageExpression` to `ValidatingAdmissionPolicy`, to set custom failure message via CEL expression. ([kubernetes/kubernetes116397](https://github.com/kubernetes/kubernetes/pull/116397), [jiahuif](https://github.com/jiahuif)) [SIG API Machinery]
- Added a new IPAddress object kind
- Added a new ClusterIP allocator. The new allocator removes previous Service CIDR block size limitations for IPv4, and limits IPv6 size to a /64 ([kubernetes/kubernetes115075](https://github.com/kubernetes/kubernetes/pull/115075), [aojea](https://github.com/aojea)) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Network and Testing]
- Added a new alpha API: ClusterTrustBundle (`certificates.k8s.io/v1alpha1`).
A ClusterTrustBundle may be used to distribute [X.509](https://www.itu.int/rec/T-REC-X.509) trust anchors to workloads within the cluster. ([kubernetes/kubernetes#113218](https://github.com/kubernetes/kubernetes/pull/113218), [ahmedtd](https://github.com/ahmedtd)) [SIG API Machinery, Auth and Testing]
- Added authorization check support to the CEL expressions of ValidatingAdmissionPolicy via a `authorizer`
variable with expressions. The new variable provides a builder that allows expressions such `authorizer.group('').resource('pods').check('create').allowed()`. ([kubernetes/kubernetes116054](https://github.com/kubernetes/kubernetes/pull/116054), [jpbetz](https://github.com/jpbetz)) [SIG API Machinery and Testing]
- Added matchConditions field to ValidatingAdmissionPolicy and enabled support for CEL based custom match criteria. ([kubernetes/kubernetes116350](https://github.com/kubernetes/kubernetes/pull/116350), [maxsmythe](https://github.com/maxsmythe))
- Added new option to the `InterPodAffinity` scheduler plugin to ignore existing
pods` preferred inter-pod affinities if the incoming pod has no preferred inter-pod
affinities. This option can be used as an optimization for higher scheduling throughput
(at the cost of an occasional pod being scheduled non-optimally/violating existing
pods preferred inter-pod affinities). To enable this scheduler option, set the
`InterPodAffinity` scheduler plugin arg `ignorePreferredTermsOfExistingPods: true` ([kubernetes/kubernetes114393](https://github.com/kubernetes/kubernetes/pull/114393), [danielvegamyhre](https://github.com/danielvegamyhre))
- Added the `MatchConditions` field to `ValidatingWebhookConfiguration` and `MutatingWebhookConfiguration` for the v1beta and v1 apis.

The `AdmissionWebhookMatchConditions` featuregate is now in Alpha ([kubernetes/kubernetes116261](https://github.com/kubernetes/kubernetes/pull/116261), [ivelichkovich](https://github.com/ivelichkovich)) [SIG API Machinery and Testing]
- Added validation to ensure that if `service.kubernetes.io/topology-aware-hints` and `service.kubernetes.io/topology-mode` annotations are both set, they are set to the same value.Also Added deprecation warning if `service.kubernetes.io/topology-aware-hints` annotation is used. ([kubernetes/kubernetes116612](https://github.com/kubernetes/kubernetes/pull/116612), [robscott](https://github.com/robscott))
- Added warnings about workload resources (Pods, ReplicaSets, Deployments, Jobs, CronJobs, or ReplicationControllers) whose names are not valid DNS labels. ([kubernetes/kubernetes114412](https://github.com/kubernetes/kubernetes/pull/114412), [thockin](https://github.com/thockin))
- Adds feature gate `NodeLogQuery` which provides cluster administrators with a streaming view of logs using kubectl without them having to implement a client side reader or logging into the node. ([kubernetes/kubernetes96120](https://github.com/kubernetes/kubernetes/pull/96120), [LorbusChris](https://github.com/LorbusChris))
- Api: validation of a `PodSpec` now rejects invalid `ResourceClaim` and `ResourceClaimTemplate` names. For a pod, the name generated for the `ResourceClaim` when using a template also must be valid. ([kubernetes/kubernetes116576](https://github.com/kubernetes/kubernetes/pull/116576), [pohly](https://github.com/pohly))
- Bump default API QPS limits for Kubelet. ([kubernetes/kubernetes116121](https://github.com/kubernetes/kubernetes/pull/116121), [wojtek-t](https://github.com/wojtek-t))
- Enabled the `StatefulSetStartOrdinal` feature gate in beta ([kubernetes/kubernetes115260](https://github.com/kubernetes/kubernetes/pull/115260), [pwschuurman](https://github.com/pwschuurman))
- Enabled usage of `kube-proxy`, `kube-scheduler` and `kubelet` HTTP APIs for changing the logging
verbosity at runtime for JSON output. ([kubernetes/kubernetes114609](https://github.com/kubernetes/kubernetes/pull/114609), [pohly](https://github.com/pohly))
- Encryption of API Server at rest configuration now allows the use of wildcards in the list of resources. For example, *.* can be used to encrypt all resources, including all current and future custom resources. ([kubernetes/kubernetes115149](https://github.com/kubernetes/kubernetes/pull/115149), [nilekhc](https://github.com/nilekhc))
- Extended the kubelet's PodResources API to include resources allocated in `ResourceClaims` via `DynamicResourceAllocation`. Additionally, added a new `Get()` method to query a specific pod for its resources. ([kubernetes/kubernetes115847](https://github.com/kubernetes/kubernetes/pull/115847), [moshe010](https://github.com/moshe010)) [SIG Node]
- Forbid to set matchLabelKeys when labelSelector is not set in topologySpreadConstraints ([kubernetes/kubernetes116535](https://github.com/kubernetes/kubernetes/pull/116535), [denkensk](https://github.com/denkensk))
- GCE does not support LoadBalancer Services with ports with different protocols (TCP and UDP) ([kubernetes/kubernetes115966](https://github.com/kubernetes/kubernetes/pull/115966), [aojea](https://github.com/aojea)) [SIG Apps and Cloud Provider]
- GRPC probes are now a GA feature. `GRPCContainerProbe` feature gate was locked to default value and will be removed in v1.29. If you were setting this feature gate explicitly, please remove it now. ([kubernetes/kubernetes116233](https://github.com/kubernetes/kubernetes/pull/116233), [SergeyKanzhelev](https://github.com/SergeyKanzhelev))
- Graduated `Kubelet Topology Manager` to GA. ([kubernetes/kubernetes116093](https://github.com/kubernetes/kubernetes/pull/116093), [swatisehgal](https://github.com/swatisehgal))
- Graduated `KubeletTracing` to beta, which means that the feature gate is now enabled by default. ([kubernetes/kubernetes115750](https://github.com/kubernetes/kubernetes/pull/115750), [saschagrunert](https://github.com/saschagrunert))
- Graduated seccomp profile defaulting to GA.

Set the kubelet `--seccomp-default` flag or `seccompDefault` kubelet configuration field to `true` to make pods on that node default to using the `RuntimeDefault` seccomp profile.

Enabling seccomp for your workload can have a negative performance impact depending on the kernel and container runtime version in use.

Guidance for identifying and mitigating those issues is outlined in the Kubernetes [seccomp tutorial](https://k8s.io/docs/tutorials/security/seccomp). ([kubernetes/kubernetes#115719](https://github.com/kubernetes/kubernetes/pull/115719), [saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery, Node, Storage and Testing]
- Graduated the container resource metrics feature on `HPA` to beta. ([kubernetes/kubernetes116046](https://github.com/kubernetes/kubernetes/pull/116046), [sanposhiho](https://github.com/sanposhiho))
- Implemented API streaming for the `watch-cache`

When `sendInitialEvents` `ListOption` is set together with `watch=true`, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. ([kubernetes/kubernetes110960](https://github.com/kubernetes/kubernetes/pull/110960), [p0lyn0mial](https://github.com/p0lyn0mial))
- Introduced API for streaming.

Added `SendInitialEvents` field to the `ListOptions`. When the new option is set together with `watch=true`, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. ([kubernetes/kubernetes115402](https://github.com/kubernetes/kubernetes/pull/115402), [p0lyn0mial](https://github.com/p0lyn0mial))
- Introduced a breaking change to the `resource.k8s.io` API in its `AllocationResult` struct. This change allows a kubelet plugin for the `DynamicResourceAllocation` feature to service allocations from multiple resource driver controllers. ([kubernetes/kubernetes116332](https://github.com/kubernetes/kubernetes/pull/116332), [klueska](https://github.com/klueska))
- Introduces new alpha functionality to the reflector, allowing user to enable API streaming.

To activate this feature, users can set the `ENABLE_CLIENT_GO_WATCH_LIST_ALPHA` environmental variable.
It is important to note that the server must support streaming for this feature to function properly.
If streaming is not supported by the server, the reflector will revert to the previous method
of obtaining data through LIST/WATCH semantics. ([kubernetes/kubernetes110772](https://github.com/kubernetes/kubernetes/pull/110772), [p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery]
- K8s.io/client-go/tools/record.EventBroadcaster: after Shutdown() is called, the broadcaster now gives up immediately after a failure to write an event to a sink. Previously it tried multiple times for 12 seconds in a goroutine. ([kubernetes/kubernetes115514](https://github.com/kubernetes/kubernetes/pull/115514), [pohly](https://github.com/pohly)) [SIG API Machinery]
- K8s.io/component-base/logs: usage of the pflag values in a normal Go flag set led to panics when printing the help message ([kubernetes/kubernetes114680](https://github.com/kubernetes/kubernetes/pull/114680), [pohly](https://github.com/pohly)) [SIG Instrumentation]
- Kubeadm: explicitly set `priority` for static pods with `priorityClassName: system-node-critical` ([kubernetes/kubernetes114338](https://github.com/kubernetes/kubernetes/pull/114338), [champtar](https://github.com/champtar)) [SIG Cluster Lifecycle]
- Kubelet: a "maxParallelImagePulls" field can now be specified in the kubelet configuration file to control how many image pulls the kubelet can perform in parallel. ([kubernetes/kubernetes115220](https://github.com/kubernetes/kubernetes/pull/115220), [ruiwen-zhao](https://github.com/ruiwen-zhao)) [SIG API Machinery, Node and Scalability]
- Kubelet: changed `MemoryThrottlingFactor` default value to `0.9` and formulas to calculate `memory.high` ([kubernetes/kubernetes115371](https://github.com/kubernetes/kubernetes/pull/115371), [pacoxu](https://github.com/pacoxu))
- Kubernetes components that perform leader election now only support using `Leases` for this. ([kubernetes/kubernetes114055](https://github.com/kubernetes/kubernetes/pull/114055), [aimuz](https://github.com/aimuz))
- Migrated the `DaemonSet` controller (within `kube-controller-manager`) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging) ([kubernetes/kubernetes113622](https://github.com/kubernetes/kubernetes/pull/113622), [249043822](https://github.com/249043822))
- New `service.kubernetes.io/topology-mode` annotation has been introduced as a replacement for the `service.kubernetes.io/topology-aware-hints` annotation.
- `service.kubernetes.io/topology-aware-hints` annotation has been deprecated.
- kube-proxy now accepts any value that is not "disabled" for these annotations, enabling custom implementation-specific and/or future built-in heuristics to be used. ([kubernetes/kubernetes116522](https://github.com/kubernetes/kubernetes/pull/116522), [robscott](https://github.com/robscott)) [SIG Apps, Network and Testing]
- Pods owned by a Job now uses the labels `batch.kubernetes.io/job-name` and `batch.kubernetes.io/controller-uid`.
The legacy labels `job-name` and `controller-uid` are still added for compatibility. ([kubernetes/kubernetes114930](https://github.com/kubernetes/kubernetes/pull/114930), [kannon92](https://github.com/kannon92))
- Promoted `CronJobTimeZone` feature to GA ([kubernetes/kubernetes115904](https://github.com/kubernetes/kubernetes/pull/115904), [soltysh](https://github.com/soltysh))
- Promoted `SelfSubjectReview` to Beta ([kubernetes/kubernetes116274](https://github.com/kubernetes/kubernetes/pull/116274), [nabokihms](https://github.com/nabokihms)) [SIG API Machinery, Auth, CLI and Testing]
- Relaxed API validation to allow pod node selector to be mutable for gated pods (additions only, no deletions or mutations). ([kubernetes/kubernetes116161](https://github.com/kubernetes/kubernetes/pull/116161), [danielvegamyhre](https://github.com/danielvegamyhre))
- Remove `kubernetes.io/grpc` standard appProtocol ([kubernetes/kubernetes116866](https://github.com/kubernetes/kubernetes/pull/116866), [LiorLieberman](https://github.com/LiorLieberman)) [SIG API Machinery and Apps]
- Remove deprecated `--enable-taint-manager` and `--pod-eviction-timeout` CLI ([kubernetes/kubernetes115840](https://github.com/kubernetes/kubernetes/pull/115840), [atosatto](https://github.com/atosatto))
- Removed support for the `v1alpha1` kubeletplugin API of `DynamicResourceManagement`. All plugins must be updated to `v1alpha2` in order to function properly. ([kubernetes/kubernetes116558](https://github.com/kubernetes/kubernetes/pull/116558), [klueska](https://github.com/klueska))
- The API server now re-uses data encryption keys while the kms v2 plugin key ID is stable. Data encryption keys are still randomly generated on server start but an atomic counter is used to prevent nonce collisions. ([kubernetes/kubernetes116155](https://github.com/kubernetes/kubernetes/pull/116155), [enj](https://github.com/enj))
- The PodDisruptionBudget `spec.unhealthyPodEvictionPolicy` field has graduated to beta and is enabled by default. On servers with the feature enabled, this field may be set to `AlwaysAllow` to always allow unhealthy pods covered by the PodDisruptionBudget to be evicted. ([kubernetes/kubernetes115363](https://github.com/kubernetes/kubernetes/pull/115363), [ravisantoshgudimetla](https://github.com/ravisantoshgudimetla)) [SIG Apps, Auth and Node]
- The `DownwardAPIHugePages` kubelet feature graduated to stable / GA. ([kubernetes/kubernetes115721](https://github.com/kubernetes/kubernetes/pull/115721), [saschagrunert](https://github.com/saschagrunert)) [SIG Apps and Node]
- The following feature gates for volume expansion GA features have now been removed and must no longer be referenced in `--feature-gates` flags: `ExpandCSIVolumes`, `ExpandInUsePersistentVolumes`, `ExpandPersistentVolumes` ([kubernetes/kubernetes113942](https://github.com/kubernetes/kubernetes/pull/113942), [mengjiao-liu](https://github.com/mengjiao-liu))
- The list-type of the alpha `resourceClaims` field introduced to `Pods` in `1.26.0` was modified from `set` to `map`, resolving an incompatibility with use of this schema in `CustomResourceDefinitions` and with server-side apply. ([kubernetes/kubernetes114585](https://github.com/kubernetes/kubernetes/pull/114585), [JoelSpeed](https://github.com/JoelSpeed))
- Updated API reference for Requests, specifying they must not exceed limits ([kubernetes/kubernetes115434](https://github.com/kubernetes/kubernetes/pull/115434), [ehashman](https://github.com/ehashman))
- Updated `KMSv2` to beta ([kubernetes/kubernetes115123](https://github.com/kubernetes/kubernetes/pull/115123), [aramase](https://github.com/aramase))
- Updated: Redefine AppProtocol field description and add new standard values ([kubernetes/kubernetes115433](https://github.com/kubernetes/kubernetes/pull/115433), [LiorLieberman](https://github.com/LiorLieberman)) [SIG API Machinery, Apps and Network]
- `/metrics/slis` is now available for control plane components allowing you to scrape health check metrics. ([kubernetes/kubernetes114997](https://github.com/kubernetes/kubernetes/pull/114997), [Richabanker](https://github.com/Richabanker))
- `APIServerTracing` feature gate is now enabled by default. Tracing in the API
Server is still disabled by default, and requires a config file to enable. ([kubernetes/kubernetes116144](https://github.com/kubernetes/kubernetes/pull/116144), [dashpole](https://github.com/dashpole))
- `NodeResourceFit` and `NodeResourcesBalancedAllocation` implement the `PreScore`
extension point for a more performant calculation. ([kubernetes/kubernetes115655](https://github.com/kubernetes/kubernetes/pull/115655), [tangwz](https://github.com/tangwz))
- `PodSchedulingReadiness` is graduated to beta. ([kubernetes/kubernetes115815](https://github.com/kubernetes/kubernetes/pull/115815), [Huang-Wei](https://github.com/Huang-Wei))
- `PodSpec.Container.Resources` became mutable for CPU and memory resource types.
- `PodSpec.Container.ResizePolicy` (new object) gives users control over how their containers are resized.
- `PodStatus.Resize` status describes the state of a requested Pod resize.
- `PodStatus.ResourcesAllocated` describes node resources allocated to Pod.
- `PodStatus.Resources` describes node resources applied to running containers by CRI.
- `UpdateContainerResources` CRI API now supports both Linux and Windows. ([kubernetes/kubernetes102884](https://github.com/kubernetes/kubernetes/pull/102884), [vinaykul](https://github.com/vinaykul))
- `SELinuxMountReadWriteOncePod` graduated to Beta. ([kubernetes/kubernetes116425](https://github.com/kubernetes/kubernetes/pull/116425), [jsafrane](https://github.com/jsafrane))
- `StatefulSetAutoDeletePVC` feature gate promoted to beta. ([kubernetes/kubernetes116501](https://github.com/kubernetes/kubernetes/pull/116501), [mattcary](https://github.com/mattcary))
- `StatefulSet` names must be DNS labels, rather than subdomains. Any `StatefulSet`
which took advantage of subdomain validation (by having dots in the name) can't
possibly have worked, because we eventually set `pod.spec.hostname` from the `StatefulSetName`,
and that is validated as a DNS label. ([kubernetes/kubernetes114172](https://github.com/kubernetes/kubernetes/pull/114172), [thockin](https://github.com/thockin))
- `ValidatingAdmissionPolicy` now provides a status field that contains results of type checking the validation expression.
The type checking is fully informational, and the behavior of the policy is unchanged. ([kubernetes/kubernetes115668](https://github.com/kubernetes/kubernetes/pull/115668), [jiahuif](https://github.com/jiahuif))
- `cacheSize` field in `EncryptionConfiguration` is not supported for KMSv2 provider ([kubernetes/kubernetes113121](https://github.com/kubernetes/kubernetes/pull/113121), [aramase](https://github.com/aramase))
- `k8s.io/component-base/logs` now also supports adding command line flags to a `flag.FlagSet`. ([kubernetes/kubernetes114731](https://github.com/kubernetes/kubernetes/pull/114731), [pohly](https://github.com/pohly))
- `kubelet`: migrated `--container-runtime-endpoint` and `--image-service-endpoint`
to kubelet config ([kubernetes/kubernetes112136](https://github.com/kubernetes/kubernetes/pull/112136), [pacoxu](https://github.com/pacoxu))
- `resource.k8s.io/v1alpha1` was replaced with `resource.k8s.io/v1alpha2`. Before
upgrading a cluster, all objects in resource.k8s.io/v1alpha1 (ResourceClaim, ResourceClaimTemplate,
ResourceClass, PodScheduling) must be deleted. The changes are internal, so
YAML files which create pods and resource claims don't need changes except for
the newer `apiVersion`. ([kubernetes/kubernetes116299](https://github.com/kubernetes/kubernetes/pull/116299), [pohly](https://github.com/pohly))
- `volumes`: `resource.claims` is now cleared for PVC specs during create or update of a pod spec with inline PVC template or of a PVC because it has no effect. ([kubernetes/kubernetes115928](https://github.com/kubernetes/kubernetes/pull/115928), [pohly](https://github.com/pohly))
- Added a new alpha API: ClusterTrustBundle (`certificates.k8s.io/v1alpha1`).
A ClusterTrustBundle may be used to distribute [X.509](https://www.itu.int/rec/T-REC-X.509) trust anchors to workloads within the cluster. ([kubernetes/kubernetes#113218](https://github.com/kubernetes/kubernetes/pull/113218), [ahmedtd](https://github.com/ahmedtd)) [SIG API Machinery, Auth and Testing]
- Remove `kubernetes.io/grpc` standard appProtocol ([kubernetes/kubernetes116866](https://github.com/kubernetes/kubernetes/pull/116866), [LiorLieberman](https://github.com/LiorLieberman)) [SIG API Machinery and Apps]
- API: resource.k8s.io/v1alpha1.PodScheduling was renamed to resource.k8s.io/v1alpha2.PodSchedulingContext. ([kubernetes/kubernetes116556](https://github.com/kubernetes/kubernetes/pull/116556), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, CLI, Node, Scheduling and Testing]
- APIServerTracing feature gate is now enabled by default. Tracing in the API Server is still disabled by default, and requires a config file to enable. ([kubernetes/kubernetes116144](https://github.com/kubernetes/kubernetes/pull/116144), [dashpole](https://github.com/dashpole)) [SIG API Machinery and Testing]
- Added CEL runtime cost calculation into ValidatingAdmissionPolicy, matching the evaluation cost
restrictions that already apply to CustomResourceDefinition.
If rule evaluation uses more compute than the limit, the API server aborts the evaluation and the
admission check that was being performed is aborted; the `failurePolicy` for the ValidatingAdmissionPolicy
determines the outcome. ([kubernetes/kubernetes115747](https://github.com/kubernetes/kubernetes/pull/115747), [cici37](https://github.com/cici37)) [SIG API Machinery]
- Added `messageExpression` to `ValidatingAdmissionPolicy`, to set custom failure message via CEL expression. ([kubernetes/kubernetes116397](https://github.com/kubernetes/kubernetes/pull/116397), [jiahuif](https://github.com/jiahuif)) [SIG API Machinery]
- Added a new IPAddress object kind
- Added a new ClusterIP allocator. The new allocator removes previous Service CIDR block size limitations for IPv4, and limits IPv6 size to a /64 ([kubernetes/kubernetes115075](https://github.com/kubernetes/kubernetes/pull/115075), [aojea](https://github.com/aojea)) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Network and Testing]
- Added a new alpha API: ClusterTrustBundle (`certificates.k8s.io/v1alpha1`).
A ClusterTrustBundle may be used to distribute [X.509](https://www.itu.int/rec/T-REC-X.509) trust anchors to workloads within the cluster. ([kubernetes/kubernetes#113218](https://github.com/kubernetes/kubernetes/pull/113218), [ahmedtd](https://github.com/ahmedtd)) [SIG API Machinery, Auth and Testing]
- Added authorization check support to the CEL expressions of ValidatingAdmissionPolicy via a `authorizer`
variable with expressions. The new variable provides a builder that allows expressions such `authorizer.group('').resource('pods').check('create').allowed()`. ([kubernetes/kubernetes116054](https://github.com/kubernetes/kubernetes/pull/116054), [jpbetz](https://github.com/jpbetz)) [SIG API Machinery and Testing]
- Added matchConditions field to ValidatingAdmissionPolicy, enabled support for CEL based custom match criteria. ([kubernetes/kubernetes116350](https://github.com/kubernetes/kubernetes/pull/116350), [maxsmythe](https://github.com/maxsmythe)) [SIG API Machinery and Testing]
- Added messageExpression field to ValidationRule. (115969, DangerOnTheRanger) ([kubernetes/kubernetes115969](https://github.com/kubernetes/kubernetes/pull/115969), [DangerOnTheRanger](https://github.com/DangerOnTheRanger)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Instrumentation, Node and Testing]
- Added the `MatchConditions` field to `ValidatingWebhookConfiguration` and `MutatingWebhookConfiguration` for the v1beta and v1 apis.

The `AdmissionWebhookMatchConditions` featuregate is now in Alpha ([kubernetes/kubernetes116261](https://github.com/kubernetes/kubernetes/pull/116261), [ivelichkovich](https://github.com/ivelichkovich)) [SIG API Machinery and Testing]
- Added validation to ensure that if `service.kubernetes.io/topology-aware-hints` and `service.kubernetes.io/topology-mode` annotations are both set, they are set to the same value.
- Added deprecation warning if `service.kubernetes.io/topology-aware-hints` annotation is used. ([kubernetes/kubernetes116612](https://github.com/kubernetes/kubernetes/pull/116612), [robscott](https://github.com/robscott)) [SIG Apps, Network and Testing]
- Adds auditAnnotations to ValidatingAdmissionPolicy, enabling CEL to be used to add audit annotations to request audit events.
Adds validationActions to ValidatingAdmissionPolicyBinding, enabling validation failures to be handled by any combination of the warn, audit and deny enforcement actions. ([kubernetes/kubernetes115973](https://github.com/kubernetes/kubernetes/pull/115973), [jpbetz](https://github.com/jpbetz)) [SIG API Machinery and Testing]
- Adds feature gate `NodeLogQuery` which provides cluster administrators with a streaming view of logs using kubectl without them having to implement a client side reader or logging into the node. ([kubernetes/kubernetes96120](https://github.com/kubernetes/kubernetes/pull/96120), [LorbusChris](https://github.com/LorbusChris)) [SIG API Machinery, Apps, CLI, Node, Testing and Windows]
- Api: validation of a PodSpec now rejects invalid ResourceClaim and ResourceClaimTemplate names. For a pod, the name generated for the ResourceClaim when using a template also must be valid. ([kubernetes/kubernetes116576](https://github.com/kubernetes/kubernetes/pull/116576), [pohly](https://github.com/pohly)) [SIG Apps]
- Bump default API QPS limits for Kubelet. ([kubernetes/kubernetes116121](https://github.com/kubernetes/kubernetes/pull/116121), [wojtek-t](https://github.com/wojtek-t)) [SIG API Machinery and Node]
- Enable the "StatefulSetStartOrdinal" feature gate in beta ([kubernetes/kubernetes115260](https://github.com/kubernetes/kubernetes/pull/115260), [pwschuurman](https://github.com/pwschuurman)) [SIG API Machinery and Apps]
- Extended the kubelet's PodResources API to include resources allocated in `ResourceClaims` via `DynamicResourceAllocation`. Additionally, added a new `Get()` method to query a specific pod for its resources. ([kubernetes/kubernetes115847](https://github.com/kubernetes/kubernetes/pull/115847), [moshe010](https://github.com/moshe010)) [SIG Node]
- Forbid to set matchLabelKeys when labelSelector isn’t set in topologySpreadConstraints ([kubernetes/kubernetes116535](https://github.com/kubernetes/kubernetes/pull/116535), [denkensk](https://github.com/denkensk)) [SIG API Machinery, Apps and Scheduling]
- GCE does not support LoadBalancer Services with ports with different protocols (TCP and UDP) ([kubernetes/kubernetes115966](https://github.com/kubernetes/kubernetes/pull/115966), [aojea](https://github.com/aojea)) [SIG Apps and Cloud Provider]
- GRPC probes are now a GA feature. GRPCContainerProbe feature gate was locked to default value and will be removed in v1.29. If you were setting this feature gate explicitly, please remove it now. ([kubernetes/kubernetes116233](https://github.com/kubernetes/kubernetes/pull/116233), [SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG API Machinery, Apps and Node]
- Graduate Kubelet Topology Manager to GA. ([kubernetes/kubernetes116093](https://github.com/kubernetes/kubernetes/pull/116093), [swatisehgal](https://github.com/swatisehgal)) [SIG API Machinery, Node and Testing]
- Graduate `KubeletTracing` to beta, which means that the feature gate is now enabled by default. ([kubernetes/kubernetes115750](https://github.com/kubernetes/kubernetes/pull/115750), [saschagrunert](https://github.com/saschagrunert)) [SIG Instrumentation and Node]
- Graduate the container resource metrics feature on HPA to beta. ([kubernetes/kubernetes116046](https://github.com/kubernetes/kubernetes/pull/116046), [sanposhiho](https://github.com/sanposhiho)) [SIG Autoscaling]
- Introduced a breaking change to the `resource.k8s.io` API in its `AllocationResult` struct. This change allows a kubelet plugin for the `DynamicResourceAllocation` feature to service allocations from multiple resource driver controllers. ([kubernetes/kubernetes116332](https://github.com/kubernetes/kubernetes/pull/116332), [klueska](https://github.com/klueska)) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]
- Introduces new alpha functionality to the reflector, allowing user to enable API streaming.

To activate this feature, users can set the `ENABLE_CLIENT_GO_WATCH_LIST_ALPHA` environmental variable.
It is important to note that the server must support streaming for this feature to function properly.
If streaming is not supported by the server, the reflector will revert to the previous method
of obtaining data through LIST/WATCH semantics. ([kubernetes/kubernetes110772](https://github.com/kubernetes/kubernetes/pull/110772), [p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery]
- Kubelet: change MemoryThrottlingFactor default value to 0.9 and formulas to calculate memory.high ([kubernetes/kubernetes115371](https://github.com/kubernetes/kubernetes/pull/115371), [pacoxu](https://github.com/pacoxu)) [SIG API Machinery, Apps and Node]
- Migrated the DaemonSet controller (within `kube-controller-manager) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging) ([kubernetes/kubernetes113622](https://github.com/kubernetes/kubernetes/pull/113622), [249043822](https://github.com/249043822)) [SIG API Machinery, Apps, Instrumentation and Testing]
- New `service.kubernetes.io/topology-mode` annotation has been introduced as a replacement for the `service.kubernetes.io/topology-aware-hints` annotation.
- `service.kubernetes.io/topology-aware-hints` annotation has been deprecated.
- kube-proxy now accepts any value that is not "disabled" for these annotations, enabling custom implementation-specific and/or future built-in heuristics to be used. ([kubernetes/kubernetes116522](https://github.com/kubernetes/kubernetes/pull/116522), [robscott](https://github.com/robscott)) [SIG Apps, Network and Testing]
- NodeResourceFit and NodeResourcesBalancedAllocation implement the PreScore extension point for a more performant calculation. ([kubernetes/kubernetes115655](https://github.com/kubernetes/kubernetes/pull/115655), [tangwz](https://github.com/tangwz)) [SIG Scheduling]
- Pods owned by a Job will now use the labels `batch.kubernetes.io/job-name` and `batch.kubernetes.io/controller-uid`.
The legacy labels `job-name` and `controller-uid` are still added for compatibility. ([kubernetes/kubernetes114930](https://github.com/kubernetes/kubernetes/pull/114930), [kannon92](https://github.com/kannon92)) [SIG Apps]
- Promote CronJobTimeZone feature to GA ([kubernetes/kubernetes115904](https://github.com/kubernetes/kubernetes/pull/115904), [soltysh](https://github.com/soltysh)) [SIG API Machinery and Apps]
- Promoted `SelfSubjectReview` to Beta ([kubernetes/kubernetes116274](https://github.com/kubernetes/kubernetes/pull/116274), [nabokihms](https://github.com/nabokihms)) [SIG API Machinery, Auth, CLI and Testing]
- Relax API validation to allow pod node selector to be mutable for gated pods (additions only, no deletions or mutations). ([kubernetes/kubernetes116161](https://github.com/kubernetes/kubernetes/pull/116161), [danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps, Scheduling and Testing]
- Remove deprecated `--enable-taint-manager` and `--pod-eviction-timeout` CLI flags ([kubernetes/kubernetes115840](https://github.com/kubernetes/kubernetes/pull/115840), [atosatto](https://github.com/atosatto)) [SIG API Machinery, Apps, Node and Testing]
- Resource.k8s.io/v1alpha1 was replaced with resource.k8s.io/v1alpha2. Before upgrading a cluster, all objects in resource.k8s.io/v1alpha1 (ResourceClaim, ResourceClaimTemplate, ResourceClass, PodScheduling) must be deleted. The changes will be internal, so YAML files which create pods and resource claims don't need changes except for the newer `apiVersion`. ([kubernetes/kubernetes116299](https://github.com/kubernetes/kubernetes/pull/116299), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]
- SELinuxMountReadWriteOncePod graduated to Beta. ([kubernetes/kubernetes116425](https://github.com/kubernetes/kubernetes/pull/116425), [jsafrane](https://github.com/jsafrane)) [SIG Storage and Testing]
- StatefulSetAutoDeletePVC feature gate promoted to beta. ([kubernetes/kubernetes116501](https://github.com/kubernetes/kubernetes/pull/116501), [mattcary](https://github.com/mattcary)) [SIG Apps, Auth and Testing]
- The API server now re-uses data encryption keys while the kms v2 plugin's key ID is stable. Data encryption keys are still randomly generated on server start but an atomic counter is used to prevent nonce collisions. ([kubernetes/kubernetes116155](https://github.com/kubernetes/kubernetes/pull/116155), [enj](https://github.com/enj)) [SIG API Machinery, Auth and Testing]
- The API server's encryption at rest configuration now allows the use of wildcards in the list of resources. For example, '*.*' can be used to encrypt all resources, including all current and future custom resources. ([kubernetes/kubernetes115149](https://github.com/kubernetes/kubernetes/pull/115149), [nilekhc](https://github.com/nilekhc)) [SIG API Machinery, Auth and Testing]
- Update KMSv2 to beta ([kubernetes/kubernetes115123](https://github.com/kubernetes/kubernetes/pull/115123), [aramase](https://github.com/aramase)) [SIG API Machinery, Auth and Testing]
- Updated: Redefine AppProtocol field description and add new standard values ([kubernetes/kubernetes115433](https://github.com/kubernetes/kubernetes/pull/115433), [LiorLieberman](https://github.com/LiorLieberman)) [SIG API Machinery, Apps and Network]
- ValidatingAdmissionPolicy now provides a status field that contains results of type checking the validation expression.
The type checking is fully informational, and the behavior of the policy is unchanged. ([kubernetes/kubernetes115668](https://github.com/kubernetes/kubernetes/pull/115668), [jiahuif](https://github.com/jiahuif)) [SIG API Machinery, Auth, Cloud Provider and Testing]
- We have removed support for the v1alpha1 kubeletplugin API of DynamicResourceManagement. All plugins must update to v1alpha2 in order to function properly going forward. ([kubernetes/kubernetes116558](https://github.com/kubernetes/kubernetes/pull/116558), [klueska](https://github.com/klueska)) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]
- Graduated seccomp profile defaulting to GA.

Set the kubelet `--seccomp-default` flag or `seccompDefault` kubelet configuration field to `true` to make pods on that node default to using the `RuntimeDefault` seccomp profile.

Enabling seccomp for your workload can have a negative performance impact depending on the kernel and container runtime version in use.

Guidance for identifying and mitigating those issues is outlined in the Kubernetes [seccomp tutorial](https://k8s.io/docs/tutorials/security/seccomp). ([kubernetes/kubernetes#115719](https://github.com/kubernetes/kubernetes/pull/115719), [saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery, Node, Storage and Testing]
- Implements API for streaming for the watch-cache

When sendInitialEvents ListOption is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. ([kubernetes/kubernetes110960](https://github.com/kubernetes/kubernetes/pull/110960), [p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery]
- Introduce API for streaming.

Add SendInitialEvents field to the ListOptions. When the new option is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. ([kubernetes/kubernetes115402](https://github.com/kubernetes/kubernetes/pull/115402), [p0lyn0mial](https://github.com/p0lyn0mial)) [SIG API Machinery]
- Kubelet: a "maxParallelImagePulls" field can now be specified in the kubelet configuration file to control how many image pulls the kubelet can perform in parallel. ([kubernetes/kubernetes115220](https://github.com/kubernetes/kubernetes/pull/115220), [ruiwen-zhao](https://github.com/ruiwen-zhao)) [SIG API Machinery, Node and Scalability]
- PodSchedulingReadiness is graduated to beta. ([kubernetes/kubernetes115815](https://github.com/kubernetes/kubernetes/pull/115815), [Huang-Wei](https://github.com/Huang-Wei)) [SIG API Machinery, Apps, Scheduling and Testing]
- In-place resize feature for Kubernetes Pods
- Changed the Pod API so that the `resources` defined for containers are mutable for `cpu` and `memory` resource types.
- Added `resizePolicy` for containers in a pod to allow users control over how their containers are resized.
- Added `allocatedResources` field to container status in pod status that describes the node resources allocated to a pod.
- Added `resources` field to container status that reports actual resources applied to running containers.
- Added `resize` field to pod status that describes the state of a requested pod resize.
For details, see KEPs below. ([kubernetes/kubernetes102884](https://github.com/kubernetes/kubernetes/pull/102884), [vinaykul](https://github.com/vinaykul)) [SIG API Machinery, Apps, Instrumentation, Node, Scheduling and Testing]
- The PodDisruptionBudget `spec.unhealthyPodEvictionPolicy` field has graduated to beta and is enabled by default. On servers with the feature enabled, this field may be set to `AlwaysAllow` to always allow unhealthy pods covered by the PodDisruptionBudget to be evicted. ([kubernetes/kubernetes115363](https://github.com/kubernetes/kubernetes/pull/115363), [ravisantoshgudimetla](https://github.com/ravisantoshgudimetla)) [SIG Apps, Auth and Node]
- The `DownwardAPIHugePages` kubelet feature graduated to stable / GA. ([kubernetes/kubernetes115721](https://github.com/kubernetes/kubernetes/pull/115721), [saschagrunert](https://github.com/saschagrunert)) [SIG Apps and Node]
- Volumes: `resource.claims` gets cleared for PVC specs during create or update of a pod spec with inline PVC template or of a PVC because it has no effect. ([kubernetes/kubernetes115928](https://github.com/kubernetes/kubernetes/pull/115928), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps and Storage]
- A fix in the resource.k8s.io/v1alpha1/ResourceClaim API avoids harmless (?) ".status.reservedFor: element 0: associative list without keys has an element that's a map type" errors in the apiserver. Validation now rejects the incorrect reuse of the same UID in different entries. ([kubernetes/kubernetes115354](https://github.com/kubernetes/kubernetes/pull/115354), [pohly](https://github.com/pohly)) [SIG API Machinery]
- CacheSize field in EncryptionConfiguration is not supported for KMSv2 provider ([kubernetes/kubernetes113121](https://github.com/kubernetes/kubernetes/pull/113121), [aramase](https://github.com/aramase)) [SIG API Machinery, Auth and Testing]
- K8s.io/client-go/tools/record.EventBroadcaster: after Shutdown() is called, the broadcaster now gives up immediately after a failure to write an event to a sink. Previously it tried multiple times for 12 seconds in a goroutine. ([kubernetes/kubernetes115514](https://github.com/kubernetes/kubernetes/pull/115514), [pohly](https://github.com/pohly)) [SIG API Machinery]
- K8s.io/component-base/logs now also supports adding command line flags to a flag.FlagSet. ([kubernetes/kubernetes114731](https://github.com/kubernetes/kubernetes/pull/114731), [pohly](https://github.com/pohly)) [SIG Architecture]
- Update API reference for Requests, specifying they must not exceed limits ([kubernetes/kubernetes115434](https://github.com/kubernetes/kubernetes/pull/115434), [ehashman](https://github.com/ehashman)) [SIG Architecture, Docs and Node]
- `/metrics/slis` is made available for control plane components allowing you to scrape health check metrics. ([kubernetes/kubernetes114997](https://github.com/kubernetes/kubernetes/pull/114997), [Richabanker](https://github.com/Richabanker)) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing]
- A terminating pod on a node that is not caused by preemption won't prevent kube-scheduler from preempting pods on that node
- Rename 'PreemptionByKubeScheduler' to 'PreemptionByScheduler' ([kubernetes/kubernetes114623](https://github.com/kubernetes/kubernetes/pull/114623), [Huang-Wei](https://github.com/Huang-Wei)) [SIG Scheduling]
- Added new option to the InterPodAffinity scheduler plugin to ignore existing pods` preferred inter-pod affinities if the incoming pod has no preferred inter-pod affinities. This option can be used as an optimization for higher scheduling throughput (at the cost of an occasional pod being scheduled non-optimally/violating existing pods' preferred inter-pod affinities). To enable this scheduler option, set the InterPodAffinity scheduler plugin arg "ignorePreferredTermsOfExistingPods: true". ([kubernetes/kubernetes114393](https://github.com/kubernetes/kubernetes/pull/114393), [danielvegamyhre](https://github.com/danielvegamyhre)) [SIG API Machinery and Scheduling]
- Added warnings about workload resources (Pods, ReplicaSets, Deployments, Jobs, CronJobs, or ReplicationControllers) whose names are not valid DNS labels. ([kubernetes/kubernetes114412](https://github.com/kubernetes/kubernetes/pull/114412), [thockin](https://github.com/thockin)) [SIG API Machinery and Apps]
- K8s.io/component-base/logs: usage of the pflag values in a normal Go flag set led to panics when printing the help message ([kubernetes/kubernetes114680](https://github.com/kubernetes/kubernetes/pull/114680), [pohly](https://github.com/pohly)) [SIG Instrumentation]
- Kube-proxy, kube-scheduler and kubelet have HTTP APIs for changing the logging verbosity at runtime. This now also works for JSON output. ([kubernetes/kubernetes114609](https://github.com/kubernetes/kubernetes/pull/114609), [pohly](https://github.com/pohly)) [SIG API Machinery, Architecture, Cloud Provider, Instrumentation and Testing]
- Kubeadm: explicitly set `priority` for static pods with `priorityClassName: system-node-critical` ([kubernetes/kubernetes114338](https://github.com/kubernetes/kubernetes/pull/114338), [champtar](https://github.com/champtar)) [SIG Cluster Lifecycle]
- Kubelet: migrate "--container-runtime-endpoint" and "--image-service-endpoint" to kubelet config ([kubernetes/kubernetes112136](https://github.com/kubernetes/kubernetes/pull/112136), [pacoxu](https://github.com/pacoxu)) [SIG API Machinery, Node and Scalability]
- Kubernetes components that perform leader election now only support using Leases for this. ([kubernetes/kubernetes114055](https://github.com/kubernetes/kubernetes/pull/114055), [aimuz](https://github.com/aimuz)) [SIG API Machinery, Cloud Provider and Scheduling]
- StatefulSet names must be DNS labels, rather than subdomains. Any StatefulSet which took advantage of subdomain validation (by having dots in the name) can't possibly have worked, because we eventually set `pod.spec.hostname` from the StatefulSetName, and that is validated as a DNS label. ([kubernetes/kubernetes114172](https://github.com/kubernetes/kubernetes/pull/114172), [thockin](https://github.com/thockin)) [SIG Apps]
- The following feature gates for volume expansion GA features have been removed and must no longer be referenced in `--feature-gates` flags: ExpandCSIVolumes, ExpandInUsePersistentVolumes, ExpandPersistentVolumes ([kubernetes/kubernetes113942](https://github.com/kubernetes/kubernetes/pull/113942), [mengjiao-liu](https://github.com/mengjiao-liu)) [SIG API Machinery, Apps and Testing]
- The list-type of the alpha resourceClaims field introduced to Pods in 1.26.0 was modified from "set" to "map", resolving an incompatibility with use of this schema in CustomResourceDefinitions and with server-side apply. ([kubernetes/kubernetes114585](https://github.com/kubernetes/kubernetes/pull/114585), [JoelSpeed](https://github.com/JoelSpeed)) [SIG API Machinery]