Even though we are a container management company, and have elected not to offer any KubeVirt management capability (or maybe because of that!!) we often get asked for our opinion on KubeVirt as a replacement for VMware. Its always hard to provide guidance without knowing the customer requirements, but a largely Windows Server estate makes the answer pretty quick.
KubeVirt is powered by KVM, the same hypervisor that carries a large share of the public cloud, and delivers near native raw performance (much like VMware). However where many people initially mess up is “forgetting” that with VMware, we engineered a storage subsystem specifically optimized for running VM disks (and they built a file system specifically optimized for this too), and unless you do the same for KubeVirt, you should expect a pretty significant IO performance loss, regardless of the VM operating system.
Memory management, however, is materially different between the two.
ESXi treats a VM’s memory as a promise; it schedules on the assumption that guests will leave some of their allocation unused, and it reclaims under pressure through ballooning, compression, and finally host swap. The balloon driver does most of the work; the hypervisor asks the guest to give memory back, and the guest, which knows which of its pages are cold, trims its own cache and pages to its own pagefile. That mechanism is what has allowed vSphere clusters to overcommit RAM 2:1 (or more) for decades.
Kubernetes schedules based on requests. A VM lands on a node based on the memory its virt-launcher pod requests, and if that request equals the guest’s configured memory, density is 1:1 (in practice slightly worse, because virt-launcher requests a little extra on top of guest memory to cover QEMU overhead). OpenShift Virtualization (the most commonly used KubeVirt platform out there) implements overcommit by setting the request below the guest size (the documented default when enabled is 150 percent). Free page reporting (kubevirt’s “ballooning”) runs in the reverse direction to ESXi ballooning; the guest volunteers pages it has freed and the host reclaims them. The safety net under real pressure is swap, managed by the wasp-agent component, which enables swap for the VMs. Red Hat’s guidance is a fast dedicated swap device per node, but swap even on an NVMe is materially slower than RAM.
Windows Server interacts poorly (from a resource efficiency standpoint) with the KubeVirt model.
Free page reporting is a Linux guest feature; the shipped Windows virtio-balloon driver does not implement it, so a Windows guest reports nothing back to the host, full stop. Windows also holds unused memory as standby cache and trims it only under pressure, so even a future Windows implementation would have little to offer. The common workloads compound it; SQL Server grows its buffer pool to max server memory and holds it, Exchange behaves the same way, and file servers cache to the ceiling. Under ESXi the balloon forced all three to release, and those workloads tolerated it in practice; under KubeVirt there is no host-driven reclaim, so those VMs stay fully committed for their lifetime regardless of actual load.
The planning consequence is that Windows Servers hosted on KubeVirt must be backed with in-host RAM sized at close to 1:1 to VM requirements. That exposes right-sizing debt, because most Windows estates carry years of 8GB and 16GB allocations that vSphere overcommit absorbed for free; on a request-based scheduler every gigabyte of that padding becomes a scheduled gigabyte that has to be purchased (and at todays CRAZY prices!!!).
Windows Server licensing then multiplies the density loss. Datacenter is licensed per physical core on every host eligible to run a Windows VM, so whatever percentage of additional nodes the density model produces applies three times: to hardware, to the OpenShift subscription, and to Windows Server core licensing.
Linux VMs sit at the other end of almost every one of these considerations. A Linux guest releases page cache readily, so free page reporting delivers real reclaim; the typical workloads (web tiers, middleware, application servers) are quiet outside business hours and release memory by design; the virtio drivers are in the mainline kernel, so nothing needs rolling out; and there is no per-core operating system licensing on the host. A Linux-heavy estate can approach vSphere density on OpenShift Virtualization with right-sizing and tuning; a Windows Server estate should plan to pay for what it allocates.
So looping back to the very start of this article… what is our advice when asked about KubeVirt vs VMware? Simple, if you predominately run Linux, then KubeVirt is a pretty good way to achieve server virtualization with a Kubernetes “feel”. If you run Windows, maybe think again.
Neil