Eight write-ups from live hosting and virtualization environments: what broke, how we traced it, and what changed afterwards. Written the way we hand them over internally, with the root cause and the numbers intact.
Two shared IPs blacklisted by Spamhaus and Barracuda after a compromised contact form turned one account into a spam relay. Contained, delisted and rate-limited within 18 hours.
18 h To confirmed blacklist removal
Domains stopped resolving for a high-value downstream customer. The suspected cause was our own recent migration. The real cause was a healthy, symptom-free server nobody had connected to the authority chain.
0 Rollbacks required
60+ production VMs moved to a new SAN backend across a 5-day window, with no maintenance window and zero customer-reported downtime.
60+ VMs migrated live
A 15-second WordPress site that plugin-disabling could not fix. The bottleneck was 4.2 million orphaned rows in wp_options. Page load went from 12 seconds to 1.8.
1.8 s Page load, down from 12 s
Random I/O stalls across VMs on one KVM node, with no pattern by workload. The cause was one failing disk behind a single Ceph OSD. No data lost.
40 VMs returned to stable I/O
A production MariaDB server that would not start after an InnoDB failure. We extracted every database in forced-recovery mode, rebuilt InnoDB clean and restored through SQL, with no data loss identified.
0 Data loss identified in validation
A production server stayed up, but root could not log in with the correct password. The cause was a corrupted /etc/login.defs. We restored it from the package through rescue mode, with no password resets.
0 Passwords reset
A SaaS company provisioned every VM by hand on VMware. We moved the platform to Apache CloudStack with templates, a provisioning API and self-service networking. 30+ VMs migrated, with the SLA never breached.
30+ VMs migrated, SLA never breached
Most of the work above started as a symptom nobody could reproduce. If that sounds familiar, we should talk.
