A production MariaDB server that would not start after an InnoDB failure. We extracted every database in forced-recovery mode, rebuilt InnoDB clean and restored through SQL, with no data loss identified.
29 September 2026 · 3 min read
TRIGGER
OOM kill mid-write
PLATFORM
MariaDB, InnoDB storage engine
RECOVERY MODE
innodb_force_recovery=4
ENGAGEMENT
Emergency database recovery
Lowest force-recovery level that started
Data loss identified in validation
Checks before return to production
Scroll the diagram sideways to see all of it →
Nothing damaged was deleted, and nothing damaged was copied back.
The database would not come back. The MariaDB service on a production server stopped unexpectedly and would not start again. Normal service restarts failed every time, and every application depending on the database server was down.
InnoDB could not finish initializing. The MariaDB error log showed the failure inside InnoDB startup: the storage engine could not initialize, so the server could not start at all. Before changing anything, we stopped MariaDB and copied the complete datadir aside as a rollback point. We then enabled innodb_force_recovery, starting at level 1 and raising it one step at a time. The server first came up at level 4. That mode is for getting data out, not for running production, so the server stayed out of service while we worked.
innodb_force_recovery is for getting data out. It is not a repair.
An OOM kill mid-write. MariaDB was killed by the kernel’s OOM killer while under heavy write load. On restart, InnoDB’s own crash recovery could not complete: the pages it needed to replay the redo log against were damaged, and initialization aborted. A kill mid-write is normally something InnoDB recovers from on its own; this time it did not, and the server needed manual recovery.
MariaDB would not start; InnoDB aborted during initialization and every dependent application was down.
Preserve the datadir, start at the lowest working recovery level, dump everything, rebuild InnoDB clean, restore through SQL.
Back in production with no data loss identified across 13 validation checks, and a fresh full backup.
Extract, rebuild, restore through SQL. Each database was dumped separately, with tables, triggers, stored routines and events, so a problem with one could not invalidate the others. Every dump was checked for completion, and database lists, table counts and key row counts were compared against the recovery instance. The damaged databases and the InnoDB system files (system tablespace and redo logs) were then moved outside the datadir, preserved rather than deleted. With force-recovery removed, MariaDB started normally on a clean InnoDB environment, with innodb_file_per_table=1 confirmed. We recreated each database with its original character set and collation and imported the verified dumps, so InnoDB rebuilt every table and index instead of reusing damaged files. Users and grants were checked separately.
Back in production, no data loss identified. Every expected database and table was present, counts matched the recovery instance, mysqlcheck came back clean, and MariaDB started without force-recovery and with no new InnoDB errors in the log. The applications were tested directly, with reads and writes under their own database users. A fresh full backup was taken, and the original files were kept until data integrity and application behaviour were confirmed.
When InnoDB cannot initialize, every blind restart and quick fix puts the data at risk. We preserve first, extract what is recoverable and rebuild clean.
