|
Replies:
9
-
Last Post:
Oct 27, 2008 5:00 PM
by: nwsmith
|
|
|
Posts:
2
From:
RU
Registered:
6/19/05
|
|
|
|
My 500-gig ZFS is gone: insufficient replicas, corrupted data
Posted:
Oct 19, 2008 12:02 AM
To: Communities » zfs » discuss
|
|
Hi,
I'm running FreeBSD 7.1-PRERELEASE with a 500-gig ZFS drive. Recently I've encountered a FreeBSD problem (PR kern/128083) and decided about updating the motherboard BIOS. It looked like the update went right but after that I was shocked to see my ZFS destroyed! Rolling the BIOS back did not help.
Now it looks like that:
# zpool status
pool: tank
state: UNAVAIL
status: One or more devices could not be used because the label is missing
or invalid. There are insufficient replicas for the pool to continue
functioning.
action: Destroy and re-create the pool from a backup source.
see: http://www.sun.com/msg/ZFS-8000-5E
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank UNAVAIL 0 0 0 insufficient replicas
ad4 UNAVAIL 0 0 0 corrupted data
# zdb -l /dev/ad4
--------------------------------------------
LABEL 0
--------------------------------------------
version=6
name='tank'
state=0
txg=4
pool_guid=12069359268725642778
hostid=2719189110
hostname='home.gladchenko.ru'
top_guid=5515037892630596686
guid=5515037892630596686
vdev_tree
type='disk'
id=0
guid=5515037892630596686
path='/dev/ad4'
devid='ad:5QM0WF9G'
whole_disk=0
metaslab_array=14
metaslab_shift=32
ashift=9
asize=500103118848
--------------------------------------------
LABEL 1
--------------------------------------------
version=6
name='tank'
state=0
txg=4
pool_guid=12069359268725642778
hostid=2719189110
hostname='home.gladchenko.ru'
top_guid=5515037892630596686
guid=5515037892630596686
vdev_tree
type='disk'
id=0
guid=5515037892630596686
path='/dev/ad4'
devid='ad:5QM0WF9G'
whole_disk=0
metaslab_array=14
metaslab_shift=32
ashift=9
asize=500103118848
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
#
I've tried to import the problem pool into OpenSolaris 2008.05 with no success:
# zpool import
pool: tank
id: 12069359268725642778
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: http://www.sun.com/msg/ZFS-8000-EY
config:
tank UNAVAIL 0 0 0 insufficient replicas
c3d0s2 UNAVAIL 0 0 0 corrupted data
#
Is there a way to recover my files from this broken pool? Maybe at least some of them? The drive was 4/5 full. :(
I would appreciate any help.
p.s. I already bought another drive of the same size yesterday. My next ZFS experience definitely will be a mirrored one.
|
|
|
Posts:
426
From:
GB
Registered:
3/21/06
|
|
|
|
Re: My 500-gig ZFS is gone: insufficient replicas, corrupted data
Posted:
Oct 19, 2008 5:29 AM
in response to: eg15
To: Communities » zfs » discuss
|
|
With ZFS, that are 4 identical labels on each
physical vdev, in this case a single hard drive.
L0/L1 at the start of the vdev, and
L2/L3 at the end of the vdev.
As I understand it, part of the reason for having
four identical labels is to make it difficult
to completely loose the information in the labels.
In this case, labels L0 & L1 look ok, but
labels L2 & L3 'failed to unpack'.
And status says "..the label is missing or invalid."
Ok, my theory is that some setting in the bios
has got confused about the size of the hard drive,
and thinks it's smaller that it was originally.
Maybe it thinks the geometry is changed.
If it thinks the size of the hard drive has reduced,
then maybe that is why it cannot read the labels at
the end of the vdev.
And maybe it think the two readable labels
are invalid because now the 'asize' does
not match what the bios is currently reporting.
I would switch back to you original bios and try
looking at settings for the hard drive geometry.
(BTW, this is the sort of situation, where it
would have been good to have noted the reported size
of the hard drive BEFORE the update.)
(And if the above theory is right, having a
mirrored pair of identical hard drives would not help,
as the bios update may cause an identical problem
with each drive.)
Good Luck
Nigel Smith
|
|
|
|
|
Posts:
778
From:
Registered:
2/14/06
|
|
|
|
Re: My 500-gig ZFS is gone: insufficient replicas, corrupted data
Posted:
Oct 19, 2008 6:52 AM
in response to: nwsmith
To: Communities » zfs » discuss
|
|
would it help to insert the raid into another computer and import it there?
|
|
|
|
|
Posts:
1,858
From:
US
Registered:
6/17/05
|
|
|
|
Re: [zfs-discuss] My 500-gig ZFS is gone: insufficient replicas,
corrupted data
Posted:
Oct 20, 2008 10:38 AM
in response to: eg15
|
|
Eugene Gladchenko wrote:
> Hi,
>
> I'm running FreeBSD 7.1-PRERELEASE with a 500-gig ZFS drive. Recently I've encountered a FreeBSD problem (PR kern/128083) and decided about updating the motherboard BIOS. It looked like the update went right but after that I was shocked to see my ZFS destroyed! Rolling the BIOS back did not help.
>
> Now it looks like that:
>
> # zpool status
> pool: tank
> state: UNAVAIL
> status: One or more devices could not be used because the label is missing
> or invalid. There are insufficient replicas for the pool to continue
> functioning.
> action: Destroy and re-create the pool from a backup source.
> see: http://www.sun.com/msg/ZFS-8000-5E
> scrub: none requested
> config:
>
> NAME STATE READ WRITE CKSUM
> tank UNAVAIL 0 0 0 insufficient replicas
> ad4 UNAVAIL 0 0 0 corrupted data
> # zdb -l /dev/ad4
> --------------------------------------------
> LABEL 0
> --------------------------------------------
> version=6
> name='tank'
> state=0
> txg=4
> pool_guid=12069359268725642778
> hostid=2719189110
> hostname='home.gladchenko.ru'
> top_guid=5515037892630596686
> guid=5515037892630596686
> vdev_tree
> type='disk'
> id=0
> guid=5515037892630596686
> path='/dev/ad4'
> devid='ad:5QM0WF9G'
> whole_disk=0
> metaslab_array=14
> metaslab_shift=32
> ashift=9
> asize=500103118848
> --------------------------------------------
> LABEL 1
> --------------------------------------------
> version=6
> name='tank'
> state=0
> txg=4
> pool_guid=12069359268725642778
> hostid=2719189110
> hostname='home.gladchenko.ru'
> top_guid=5515037892630596686
> guid=5515037892630596686
> vdev_tree
> type='disk'
> id=0
> guid=5515037892630596686
> path='/dev/ad4'
> devid='ad:5QM0WF9G'
> whole_disk=0
> metaslab_array=14
> metaslab_shift=32
> ashift=9
> asize=500103118848
> --------------------------------------------
> LABEL 2
> --------------------------------------------
> failed to unpack label 2
> --------------------------------------------
> LABEL 3
> --------------------------------------------
> failed to unpack label 3
>
This would occur if the beginning of the partition was intact,
but the end is not. Causes for the latter include:
1. partition table changes (or vtoc for SMI labels)
2. something overwrote data at the end
If the cause is #1, then restoring the partion should work. If
the cause is #2, then the data may be gone.
Note: ZFS can import a pool with one working label, but if
more of the data is actually unavailable or overwritten, then
it may not be able to get to a consistent state.
-- richard
> #
>
> I've tried to import the problem pool into OpenSolaris 2008.05 with no success:
>
> # zpool import
> pool: tank
> id: 12069359268725642778
> state: UNAVAIL
> status: The pool was last accessed by another system.
> action: The pool cannot be imported due to damaged devices or data.
> see: http://www.sun.com/msg/ZFS-8000-EY
> config:
>
> tank UNAVAIL 0 0 0 insufficient replicas
> c3d0s2 UNAVAIL 0 0 0 corrupted data
> #
>
> Is there a way to recover my files from this broken pool? Maybe at least some of them? The drive was 4/5 full. :(
>
> I would appreciate any help.
>
> p.s. I already bought another drive of the same size yesterday. My next ZFS experience definitely will be a mirrored one.
> --
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris dot org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
2
From:
RU
Registered:
6/19/05
|
|
|
|
Re: My 500-gig ZFS is gone: insufficient replicas, corrupted data
Posted:
Oct 27, 2008 8:00 AM
in response to: eg15
To: Communities » zfs » discuss
|
|
Hi everyone, sorry for the late reply.
First of all, I've got all my files back. Cheers! :-)
Next, I'd like to tnank Nigel Smith, you are the best!
And, if anyone is interested, here is the end of the story and I'll try not to make it too long.
As one can see at http://www.freebsd.org/cgi/query-pr.cgi?pr=128083 the size of /dev/ad4 was 476940MB before the trouble arose.
And as Nigel said I really didn't notice the size had changed to 476938MB. 2MB got stolen.
The one to blame was HPA, or Host Protected Area, see http://en.wikipedia.org/wiki/Host_Protected_Area for details.
A damned new motherboard BIOS silently cut 2 megabytes down from the drive so that ZFS went insane.
All I had to do was to reset the HPA back to the native capacity and pray that BIOS did not overwrite any ZFS vital data.
With the new 7200.11 Seagates the resetting procedure was a bit complicated because I had to power-cycle the drive while feeding it the new HPA setting.
After making my data come back I made a copy of them, let the BIOS cut the drive again and re-created the pool with the new HDD size. That's all.
Thank you all again. Take care.
Eugene Gladchenko
|
|
|
|
|
Posts:
3,398
From:
Registered:
3/9/05
|
|
|
|
Re: [zfs-discuss] My 500-gig ZFS is gone: insufficient replicas,
corrupted data
Posted:
Oct 27, 2008 8:09 AM
in response to: eg15
|
|
>A damned new motherboard BIOS silently cut 2 megabytes down from
>the drive so that ZFS went insane
.
Can you tell us which BIOS/Motherboard we should avoid?
Casper
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
426
From:
GB
Registered:
3/21/06
|
|
|
|
|
|
Posts:
426
From:
GB
Registered:
3/21/06
|
|
|
|
Re: My 500-gig ZFS is gone: insufficient replicas, corrupted data
Posted:
Oct 27, 2008 10:24 AM
in response to: eg15
To: Communities » zfs » discuss
|
|
Hi Eugene
I'm delighted to hear you got your files back!
I've seen a few posts to this forum where people have
done some change to the hardware, and then found
that the ZFS pool have gone. And often you never
hear any more from them, so you assume they could
not recover it.
Thanks for reporting back your interesting story.
I wonder how many other people have been caught out
with this 'Host Protected Area' (HPA) and never
worked out that this was the cause...
Maybe one moral of this story is to make a note of
your hard drive and partitions sizes now, while
you have a working system.
If your using Solaris, maybe try 'prtvtoc'.
http://docs.sun.com/app/docs/doc/819-2240/prtvtoc-1m?a=view
(Unless someone knows a better way?)
Thanks
Nigel Smith
# prtvtoc /dev/rdsk/c1t1d0
* /dev/rdsk/c1t1d0 partition map
*
* Dimensions:
* 512 bytes/sector
* 1465149168 sectors
* 1465149101 accessible sectors
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 34 222 255
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 4 00 256 1465132495 1465132750
8 11 00 1465132751 16384 1465149134
|
|
|
|
|
Miles Nordin
carton@Ivy.NET
|
|
|
|
Re: [zfs-discuss] My 500-gig ZFS is gone: insufficient replicas,
corrupted data
Posted:
Oct 27, 2008 12:13 PM
in response to: nwsmith
|
|
|
|
>>>>> "ns" == Nigel Smith <nwsmith at wilusa dot freeserve dot co dot uk> writes:
ns> make a note of your hard drive and partitions sizes now, while
ns> you have a working system.
keeping a human-readable backup of all your disklabels somewhere safe
has helped me a few times. For me it was mostly moving disks among
architectures (i386, sparc, alpha), but even if it weren't for this
new and broken idea of boot firmware feeling entitled to ``write'' to
disks, there are already many label formats and many label readers and
writers just on a single x86 system which motivated that one-pager:
the Linux label writer becomes incompatible with the Solaris reader
because Linux disables the HPA, and the Linux reader is more forgiving
than the Solaris reader.
The other thing I don't like is that it's hard to tell under Solaris
the difference between a physically defective disk, and a disk where
Solaris ``doesn't like'' the label. There's no uniform Solaris
equivalent to Linux's unpartitioned device interface. And the solaris
label tools like fmthard, rmformat, format, prtvtoc, have a variety of
quiet and ambiguously reported reasons for refusing to operate with a
disk, and a bunch of undocumented command line flags and secret
environment variables.
Lastly shouldn't these bugs or ARC's or whatever:
http://opensolaris.org/os/community/arc/caselog/2007/660/onepager/
http://bugs.opensolaris.org/view_bug.do?bug_id=5044205
cover SATA framework drives and drives appearing with SCSI emulation
through mpt or mega_sas? Covering PATA only isn't much help. A
uniform ATA layer would be good for everyone---for keeping this HPA
work well-factored as well as making tools like hdparm, smartctl,
cdrecord work consistently.
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
426
From:
GB
Registered:
3/21/06
|
|
|
|
Re: [zfs-discuss] My 500-gig ZFS is gone: insufficient replicas, corrupted data
Posted:
Oct 27, 2008 5:00 PM
in response to: Miles Nordin
To: Communities » zfs » discuss
|
|
Hi Miles
I think you make some very good points in your comments.
It would be nice to get some positive feedback on these from Sun.
And my thought also on (quickly) looking at that bug & ARC case was
does not this also need to be factored into the SATA framework.
I really miss not having 'smartctl' (fully) working with PATA and
SATA drives on x86 Solaris.
I've done a quick search on PSARC 2007/660 and it was
"closed approved fast-track 11/28/2007".
I did a quick search, but I could not find any code that had been
committed to 'onnv-gate' that references this case.
Regards
Nigel Smith
|
|
|
|
|
