Re: [PATCH] ZFS: Ignore replacing-* devices

[Richard Laager]

On 2/6/19 5:39 AM, Daniel Kiper wrote:
> On Sat, Feb 02, 2019 at 05:06:39PM -0600, Richard Laager wrote:
>> I submitted this patch to Ubuntu some time ago, but apparently never
>> submitted it directly to the GRUB project.
>>
>> I have copied the information from my Ubuntu bug report (including
>> attachments) into this email. If you wish to see the original bug
>> report, it is available here:
>> https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1639209:
>>
>> Replace a drive in a pool.
>> zpool replace grape1 sdg1 sdf1
>>
>> Note that this creates a replacing-0 line in the zpool status output:
>>
>> address@hidden:~# zpool status
>>   pool: grape1
>>  state: ONLINE
>> status: One or more devices is currently being resilvered. The pool will
>>  continue to function, possibly in a degraded state.
>> action: Wait for the resilver to complete.
>>   scan: resilver in progress since Fri Nov 4 04:58:55 2016
>>     37.4G scanned out of 4.06T at 17.6M/s, 66h42m to go
>>     5.26G resilvered, 0.90% done
>> config:
>>
>>  NAME STATE READ WRITE CKSUM
>>  grape1 ONLINE 0 0 0
>>    raidz3-0 ONLINE 0 0 0
>>      replacing-0 ONLINE 0 0 0
>>        sdg1 ONLINE 0 0 0
>>        sdf1 ONLINE 0 0 0 (resilvering)
>>      sdd1 ONLINE 0 0 0
>>      sdc1 ONLINE 0 0 0
>>      sdb1 ONLINE 0 0 0
>>      sdh1 ONLINE 0 0 0
>>      sde1 ONLINE 0 0 0
>>      sda1 ONLINE 0 0 0
>>
>> errors: No known data errors
>>
>> Then try to install GRUB to that drive:
>>
>> address@hidden:~# grub-install /dev/sdf
>> Installing for i386-pc platform.
>> grub-install: error: failed to get canonical path of `/dev/replacing-0'.
>>
>> The attached patch fixes this.
>>
>> Unfortunately, then I get this:
>>
>> address@hidden:~# grub-install /dev/sdf
>> Installing for i386-pc platform.
>> grub-install: error: unknown filesystem.
> 
> So, AIUI this patch does not solve the problem or just does it partially.
> Should not we update the docs instead and say that grub-install should
> be called when resilvering is finished?

In any event, the replacing-0 line is not a block device (just as raidz
lines are not a block device) and needs to be excluded, so the patch is
fully correct in that sense.

How much more is necessary is another question. I haven't re-tested this
recently, as it would require me to setup a pool and get it into a
resilvering state.

One option would be to exclude devices that are involved in the
replacement operation. That would be those devices indented under it:
sdg1 and sdf1 in my example. In the case where the pool has redundancy,
this is a great option. It skips both the disk that has failed / is
failing and the new disk. However, if you have a single-disk pool,
that's not going to be a good thing.

However, my notes in the Ubuntu bug report say that the grub-install
still failed even with this patch. The extended debug logs say, "zfs
detection failed", but it's not at all clear why.

It certainly would be safer and easier to simply declare that you should
not grub-install while the pool is rebuilding. That's not great if the
pool takes a non-trivial amount of time to rebuild. That said, for the
separate reason that GRUB's ZFS support does not support the latest ZFS
feature flags in ZFS-on-Linux, I expect I'm going to start suggesting
(in the Ubuntu & Debian root-on-ZFS HOWTOs) that users make a separate,
small pool for /boot. That would make rebuild times for that pool trivial.

-- 
Richard