zpool stalls and is unkillable after attempting to import pool of unallocated volume

[Wormsweat]

System information

Type	Version/Name
Distribution Name	Windows 10
Distribution Version	22H2 (build 19045.7417)
Kernel Version	10.0.19041.7417
Architecture	x64
OpenZFS Version	zfswin-2.4.1rc12

Describe the problem you're observing

A thread in zpool.exe stalled on I/O. Its thread was at 0% CPU at the time and there were no openzfs.sys or openzvol.sys or any other kernel threads consistently at 100% of a CPU core at the time. I was unable to debug it, because I attempted to end the process before attempting to debug it. I attempted to cancel I/O through System Informer, but System Informer also stalled and became unkillable.

Describe how to reproduce the problem

I mounted a pool from a USB drive then deleted all the volumes through Disk Management while it's mounted disregarding any warnings then unmounted the pool then unplugged and plugged the drive in again, mounted it again, then zpool got stuck when importing it.

Include any warning/errors/backtraces from the system logs

This is a stack I got of the stalled zpool thread captured by System Informer:

0, ntdll.dll!NtDeviceIoControlFile+0x14
1, bdhkm64.dll+0x1ded
2, bdhkm64.dll!BdhkmIsApiInDllImports+0xb06a
3, bdhkm64.dll!BdhkmIsApiInDllImports+0x5ef2
4, 0x7ff43e0b10ce
5, KernelBase.dll!DeviceIoControl+0x6b
6, kernel32.dll!DeviceIoControlImplementation+0x81
7, zpool.exe+0xb5944
8, zpool.exe+0xb5821
9, zpool.exe+0xb07e1
10, zpool.exe+0x860a8
11, zpool.exe+0x718d7
12, zpool.exe+0x75c8b
13, zpool.exe+0x71dbd
14, zpool.exe+0x2f2cc
15, zpool.exe+0x268b3
16, zpool.exe+0xd8b909
17, zpool.exe+0xd8b7b2
18, zpool.exe+0xd8b66e
19, zpool.exe+0xd8b99e
20, kernel32.dll!BaseThreadInitThunk+0x14
21, ntdll.dll!RtlUserThreadStart+0x21

System Informer also printed this when analyzing the thread: "Unable to determine why the thread is waiting."

[sskras]

@Wormsweat, does the importing process recover after the Windows restart?

[Wormsweat]

@Wormsweat, does the importing process recover after the Windows restart?

My PC was able to restart and didn't stall during the restart. When importing again after the restart, it printed this: "cannot import 'zfs-tank': no such pool available" and zpool didn't stall. I was able to import it again once I made a RAW partition that covered the entire disk.

[guenther-alka]

Had you unmounted or exported the usb pool prior disconnect.
(zpool export is mandatory prior disconnect, this is general with ZFS otherwie you need a reboot to regain access to this pool)

[sskras]

That's a very relevant question.

If the pool got just its' datasets unmounted, but not exported, this would be #450 (at least partially).
If the pool was exported in addition, this would be different issue – a very interesting one.

[Wormsweat]

That's a very relevant question.

If the pool got just its' datasets unmounted, but not exported, this would be #450 (at least partially). If the pool was exported in addition, this would be different issue – a very interesting one.

I tried testing it again but it got stuck while exporting the pool instead. I couldn't replicate it. For the time I got zpool stuck on importing the pool, I remembered successfully exporting it previously before it got stuck importing by either running zpool export zfs-tank or by using the zfs_tray utility. I guess it's best to prioritize #450 first before this issue. It was a single USB drive with 1 volume. For when it was stuck on zpool export, zpool also got stuck at zpool status and zpool clear so it's either another issue or behavior than #450 or it updated to have these parts of zpool affected.

[lundman]

7, zpool.exe+0xb5944
8, zpool.exe+0xb5821
9, zpool.exe+0xb07e1
10, zpool.exe+0x860a8

Awesome you produced the stack, but also point it to the symbols C:/Program Files/OpenZFS on Windows/symbols so that we get function names, then we can totally see what is going on.

[guenther-alka]

I also have had problems with a USB disk as basic backup pool, mainly due sudden unclear disconnects. This produces unfinished io that blocks the pool with a needed reboot to regain access. This behviour is identical to other ZFS platforms.

When I switched from USB2 to USB3 on my Lenovo, I had much less problems. The number of possible USB chipset, cable and driver problems is what makes USB disks unreliable.

[lundman]

  zpool import
    → zpool_iter          (iterating existing pools to check for conflicts)
      → zpool_open_silent
        → zpool_refresh_stats (libzfs_config.c:235)
          → ZFS_IOC_POOL_STATS ioctl
            → spa_get_stats → spa_open_common
              → STUCK

but yeah blowing away the partition disk form under it is pretty random.