Hypervisor fault causing VM instability

Random I/O stalls across VMs on one KVM node, with no pattern by workload. The cause was one failing disk behind a single Ceph OSD. No data lost.

5 September 2026 · 2 min read

SECTOR

Virtualization provider

SCALE

40 VMs across 3 nodes

PLATFORM

KVM + Ceph

ENGAGEMENT

Platform stability

40

VMs returned to stable I/O

4 h

Full cluster rebalance

0

Data loss

Environment

Scroll the diagram sideways to see all of it →

Placement-group distribution is what made a single failing disk look like a random fault.

The problem

Symptoms with no pattern. Multiple VMs on one hypervisor node were hitting I/O stalls and intermittent unresponsiveness. The issue looked random, tied to no particular VM or workload, and rebooting the affected VMs bought only temporary relief.

Investigation

Three layers of logs pointed at one disk. Hypervisor-level logs showed repeated libvirt timeout errors correlating with the stalls. Ceph OSD logs then pointed at elevated latency on one specific OSD, which mapped to a disk in the hypervisor’s local storage node. SMART data confirmed early-stage read errors on that underlying drive.

Randomness at the guest level is usually structure at the storage level.

Root cause

One failing disk, amplified by recovery logic. Slow reads on a single OSD meant that whenever a VM with placement groups on that OSD issued I/O, Ceph’s recovery and retry logic added enough latency to surface as a visible stall inside the guest. The apparent randomness was just placement-group distribution.

At a glance

Challenge

Intermittent I/O stalls on one node, with no correlation to VM or workload.

Approach

Correlate libvirt timeouts with Ceph OSD latency, then confirm against SMART data.

Result

Failing drive replaced, 4-hour rebalance, no data loss.

The fix

Out, replace, scrub, verify. We marked the failing OSD out of the cluster to trigger data rebalancing, replaced the physical drive, and re-added the OSD after a full scrub of the rebuilt placement groups. Cluster health and I/O latency were then monitored for 72 hours to confirm stability.