ZFS recovery and stability

[mrgreiner]

Hi,

I need help with a ZFS partition with stability problems. It runs in a 7TB Hardware-RAID 5 disk. The problem I have is that when the server is turned off without a proper server shutdown (ie: power fail), the metadata of the partition gets corrupted and I'm forced to run a zpool import with '-fFX' parameters. With the size of the partition (~50TB raw data, ~5TB compressed/dedup), it's been running for ten days and the recovery still hasn't finished. So, I would like help to:

• Know how to speed up this recovery.
• What should I modify to improve reliability and avoid the corruption of the metadata in the first place.

Tks,

Roberto

PS: Installation information:
Ubuntu 22.04 server using ZFS packages from the repository
OpenZFS version 2.1.5
Partition has a raw capacity of 7.2TB, about 50TB of raw data, about 5TB compressed/dedup.
The only use of this partition is to store Bacula backup files, using the Aligned-Disk driver from Bacula.

[amotin]

Does your "Hardware-RAID 5 disk" properly handle cache flush requests? If it cheats with that to improve RAID5 read-modify-write performance, then ZFS is doomed there. Check what write caching policy you have configured there and whether backup battery on the raid is still alive. Or even better, replace the garbage with simple HBA and let ZFS do the job.

[mschilli87]

Also consider investing in some backup-power solution. You don't need much to complete a proper shutdown upon main power failure. I would not spend much effort on optimizing the speed of disaster recovery (i.e. something that ideally should never be needed in the first place) as long as I still have options to prevent such situations to begin with.