|
Replies:
43
-
Last Post:
Dec 8, 2006 11:02 AM
by: relling
|
|
|
Elizabeth Schwa...
betsy.schwartz@gmail...
|
|
|
|
Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 12:02 PM
|
|
So I rebuilt my production mail server as Solaris 10 06/06 with zfs, it
ran for three months, and it's had no hardware errors. But my zfs file
system seems to have died a quiet death. Sun engineering response was
to point to the FMRI, which says to throw out the zfs partition and
start over. I'm real reluctant to do that, since it'll take hours to do
a tape restore, and we don't know what's wrong. I'm seriously
wondering if I should just toss zfs. Again, this is Solaris 10 06/06,
not some beta version. It's an older server, a 280R with an older SCSI
RaidKing
The sun engineer is escalating but thanks for any clues or
thoughts. I don't want to erase the evidence until we get some idea
what's wrong. zpool status -v pool: mailstore state: ONLINE status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see:
http://www.sun.com/msg/ZFS-8000-8A scrub: scrub completed with 1 errors on Tue Nov 28 12:29:18 2006 config: NAME STATE READ WRITE CKSUM mailstore ONLINE 0 0 250
c1t4d0 ONLINE 0 0 250 errors: The following persistent errors have been detected: DATASET OBJECT RANGE mailstore/imap 4539004 0-8192
|
|
|
Posts:
279
From:
US
Registered:
7/7/06
|
|
|
|
Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 1:41 PM
in response to: Elizabeth Schwa...
|
|
On 11/28/06, Elizabeth Schwartz <betsy dot schwartz at gmail dot com> wrote:
> So I rebuilt my production mail server as Solaris 10 06/06 with zfs, it ran
> for three months, and it's had no hardware errors. But my zfs file system
> seems to have died a quiet death. Sun engineering response was to point to
> the FMRI, which says to throw out the zfs partition and start over. I'm real
> reluctant to do that, since it'll take hours to do a tape restore, and we
> don't know what's wrong. I'm seriously wondering if I should just toss zfs.
> Again, this is Solaris 10 06/06, not some beta version. It's an older
> server, a 280R with an older SCSI RaidKing
Looks to me like another example of ZFS noticing and reporting an
error that would go quietly by on any other filesystem. And if you're
concerned with the integrity of the data, why not use some ZFS
redundancy? (I'm guessing you're applying the redundancy further
downstream; but, as this situation demonstrates, separating it too far
from the checksum verification makes it less useful.)
--
David Dyer-Bennet, <mailto:dd-b@dd-b.net>, <http://
RKBA: <http://
Pics: <http://
Dragaera/Steven Brust: <http://
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Elizabeth Schwa...
betsy.schwartz@gmail...
|
|
|
|
Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 1:50 PM
in response to: dd-b
|
|
On 11/28/06, David Dyer-Bennet < dd-b@dd-b.net> wrote:
Looks to me like another example of ZFS noticing and reporting an
error that would go quietly by on any other filesystem. And if you're
concerned with the integrity of the data, why not use some ZFS
redundancy? (I'm guessing you're applying the redundancy further
downstream; but, as this situation demonstrates, separating it too far
from the checksum verification makes it less useful.)
Well, this error meant that two files on the file system were inaccessible, and one user was completely unable to use IMAP, so I don't know about unnoticeable.
How would I use more redundancy?
thanks Betsy
|
|
|
|
|
Posts:
413
From:
US
Registered:
3/9/05
|
|
|
|
Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 2:04 PM
in response to: Elizabeth Schwa...
|
|
Elizabeth Schwartz wrote:
> On 11/28/06, *David Dyer-Bennet* <dd-b@dd-b.net <mailto:dd-b@dd-b.net>>
> wrote:
>
> Looks to me like another example of ZFS noticing and reporting an
> error that would go quietly by on any other filesystem. And if you're
> concerned with the integrity of the data, why not use some ZFS
> redundancy? (I'm guessing you're applying the redundancy further
> downstream; but, as this situation demonstrates, separating it too far
> from the checksum verification makes it less useful.)
>
>
> Well, this error meant that two files on the file system were
> inaccessible, and one user was completely unable to use IMAP, so I don't
> know about unnoticeable.
David said, "[the error] would go quietly by on any other filesystem".
The point is that ZFS detected and reported the fact that your hardware
corrupted the data. A different filesystem would have simply given your
application the incorrect data.
> How would I use more redundancy?
By creating a zpool with some redundancy, eg. 'zpool create poolname
mirror disk1 disk2'.
--matt
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
1,858
From:
US
Registered:
6/17/05
|
|
|
|
Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 3:02 PM
in response to: ahrens
|
|
Matthew Ahrens wrote:
> Elizabeth Schwartz wrote:
>> How would I use more redundancy?
>
> By creating a zpool with some redundancy, eg. 'zpool create poolname
> mirror disk1 disk2'.
after the fact, you can add a mirror using 'zpool attach'
-- richard
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
802
From:
Plano, TX
Registered:
4/27/05
|
|
|
|
Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 6:17 PM
in response to: ahrens
|
|
On Tue, 28 Nov 2006, Matthew Ahrens wrote:
> Elizabeth Schwartz wrote:
> > On 11/28/06, *David Dyer-Bennet* <dd-b@dd-b.net <mailto:dd-b@dd-b.net>>
> > wrote:
> >
> > Looks to me like another example of ZFS noticing and reporting an
> > error that would go quietly by on any other filesystem. And if you're
> > concerned with the integrity of the data, why not use some ZFS
> > redundancy? (I'm guessing you're applying the redundancy further
> > downstream; but, as this situation demonstrates, separating it too far
> > from the checksum verification makes it less useful.)
> >
> >
> > Well, this error meant that two files on the file system were
> > inaccessible, and one user was completely unable to use IMAP, so I don't
> > know about unnoticeable.
>
> David said, "[the error] would go quietly by on any other filesystem".
> The point is that ZFS detected and reported the fact that your hardware
> corrupted the data. A different filesystem would have simply given your
> application the incorrect data.
>
> > How would I use more redundancy?
>
> By creating a zpool with some redundancy, eg. 'zpool create poolname
> mirror disk1 disk2'.
Or a 3-way mirror.... As others have said, it's likely that your hardware
RAID box has hardware/firmware bugs, that up to now, have gone by
unnoticed. It might also be time to upgrade the h/w RAID firmware and/or
the disk drive firmware for the drives that are installed in the h/w RAID
box.
In any case, ZFS will keep your h/w RAID supplier honest! Or ... it's
quite possible that you hardware RAID provider does not know that there is
a (nasty) bug(s) present. Think about that for a second.... Not a
pleasant thought for a production system!
Regards,
Al Hopper Logical Approach Inc, Plano, TX. al@logical-approach.com
Voice: 972.379.2133 Fax: 972.379.2134 Timezone: US CDT
OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005
OpenSolaris Governing Board (OGB) Member - Feb 2006
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
279
From:
US
Registered:
7/7/06
|
|
|
|
Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 2:24 PM
in response to: Elizabeth Schwa...
|
|
On 11/28/06, Elizabeth Schwartz <betsy dot schwartz at gmail dot com> wrote:
> On 11/28/06, David Dyer-Bennet <dd-b@dd-b.net> wrote:
> > Looks to me like another example of ZFS noticing and reporting an
> > error that would go quietly by on any other filesystem. And if you're
> > concerned with the integrity of the data, why not use some ZFS
> > redundancy? (I'm guessing you're applying the redundancy further
> > downstream; but, as this situation demonstrates, separating it too far
> > from the checksum verification makes it less useful.)
>
> Well, this error meant that two files on the file system were inaccessible,
> and one user was completely unable to use IMAP, so I don't know about
> unnoticeable.
Is that only because ZFS knows the file is corrupt, and refuses to
allow access? Possibly another filesystem would happily serve
whatever bits happened to come back from the RaidKing, not knowing any
better.
> How would I use more redundancy?
Create a RAIDZ or mirror vdev and put that in the pool, instead of
putting a single device in the pool. You're probably counting on your
RaidKing to handle the RAID, and then running ZFS on top of that, but
this looks like a situation where the RaidKing had some kind of
unrecoverable (and possibly undetected) error. Either it did, or else
ZFS corrupted the data all by itself. One or the other, anyway, has
let you down seriously. (I'm inclined to have considerable faith in
ZFS at this point, and am not familiar with RaidKing; what's needed is
some kind of actual evidence of which side failed, not guesses.)
Stacking them that way, running the redundancy in a separate layer
from ZFS, means you're not getting the full benefit from the
integration of filesystem, volume management, and redundancy handling
all into ZFS. On the other hand, running on old hardware that's
heavily set up for offloading RAID activities from the CPU, that might
be the best available tradeoff; I'm not very familiar with such
hardware.
--
David Dyer-Bennet, <mailto:dd-b@dd-b.net>, <http://
RKBA: <http://
Pics: <http://
Dragaera/Steven Brust: <http://
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Jason J. W. Wil...
jasonjwwilliams@gmai...
|
|
|
|
Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 1:54 PM
in response to: dd-b
|
|
Do both RAID-Z and Mirror redundancy use checksums on ZFS? Or just RAID-Z?
Thanks in advance,
J
On 11/28/06, David Dyer-Bennet <dd-b@dd-b.net> wrote:
> On 11/28/06, Elizabeth Schwartz <betsy dot schwartz at gmail dot com> wrote:
> > So I rebuilt my production mail server as Solaris 10 06/06 with zfs, it ran
> > for three months, and it's had no hardware errors. But my zfs file system
> > seems to have died a quiet death. Sun engineering response was to point to
> > the FMRI, which says to throw out the zfs partition and start over. I'm real
> > reluctant to do that, since it'll take hours to do a tape restore, and we
> > don't know what's wrong. I'm seriously wondering if I should just toss zfs.
> > Again, this is Solaris 10 06/06, not some beta version. It's an older
> > server, a 280R with an older SCSI RaidKing
>
> Looks to me like another example of ZFS noticing and reporting an
> error that would go quietly by on any other filesystem. And if you're
> concerned with the integrity of the data, why not use some ZFS
> redundancy? (I'm guessing you're applying the redundancy further
> downstream; but, as this situation demonstrates, separating it too far
> from the checksum verification makes it less useful.)
> --
> David Dyer-Bennet, <mailto:dd-b@dd-b.net>, <http://
> RKBA: <http://
> Pics: <http://
> Dragaera/Steven Brust: <http://
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris dot org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
91
From:
Menlo Park, CA
Registered:
3/9/05
|
|
|
|
Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 1:59 PM
in response to: Jason J. W. Wil...
|
|
They both use checksums and can provide self-healing data.
--Bill
On Tue, Nov 28, 2006 at 02:54:56PM -0700, Jason J. W. Williams wrote:
> Do both RAID-Z and Mirror redundancy use checksums on ZFS? Or just RAID-Z?
>
> Thanks in advance,
> J
>
> On 11/28/06, David Dyer-Bennet <dd-b@dd-b.net> wrote:
> >On 11/28/06, Elizabeth Schwartz <betsy dot schwartz at gmail dot com> wrote:
> >> So I rebuilt my production mail server as Solaris 10 06/06 with zfs, it
> >ran
> >> for three months, and it's had no hardware errors. But my zfs file system
> >> seems to have died a quiet death. Sun engineering response was to point
> >to
> >> the FMRI, which says to throw out the zfs partition and start over. I'm
> >real
> >> reluctant to do that, since it'll take hours to do a tape restore, and we
> >> don't know what's wrong. I'm seriously wondering if I should just toss
> >zfs.
> >> Again, this is Solaris 10 06/06, not some beta version. It's an older
> >> server, a 280R with an older SCSI RaidKing
> >
> >Looks to me like another example of ZFS noticing and reporting an
> >error that would go quietly by on any other filesystem. And if you're
> >concerned with the integrity of the data, why not use some ZFS
> >redundancy? (I'm guessing you're applying the redundancy further
> >downstream; but, as this situation demonstrates, separating it too far
> >from the checksum verification makes it less useful.)
> >--
> >David Dyer-Bennet, <mailto:dd-b@dd-b.net>, <http://
> >RKBA: <http://
> >Pics: <http://
> >Dragaera/Steven Brust: <http://
> >_______________________________________________
> >zfs-discuss mailing list
> >zfs-discuss at opensolaris dot org
> >http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
> >
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris dot org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
3,793
From:
GB
Registered:
3/9/05
|
|
|
|
Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 2:02 PM
in response to: Jason J. W. Wil...
|
|
Jason J. W. Williams wrote:
> Do both RAID-Z and Mirror redundancy use checksums on ZFS? Or just RAID-Z?
Both do not only that but even if you have no redundancy in the pool you
still use checksuming. That is what has actually happened in this case.
The checksuming in ZFS has detected errors but due to the lack of
redundancy at that ZFS layer it isn't able to correct them. In a raidz
or mirroring configuration correction is possible.
--
Darren J Moffat
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
3,422
From:
Austin, TX, USA
Registered:
6/15/05
|
|
|
|
Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 1:53 PM
in response to: Elizabeth Schwa...
|
|
On Tue, Nov 28, 2006 at 03:02:59PM -0500, Elizabeth Schwartz wrote:
> So I rebuilt my production mail server as Solaris 10 06/06 with zfs, it ran
> for three months, and it's had no hardware errors. But my zfs file system
> seems to have died a quiet death. Sun engineering response was to point to
> the FMRI, which says to throw out the zfs partition and start over. I'm real
> reluctant to do that, since it'll take hours to do a tape restore, and we
> don't know what's wrong. I'm seriously wondering if I should just toss zfs.
> Again, this is Solaris 10 06/06, not some beta version. It's an older
> server, a 280R with an older SCSI RaidKing
So you have a one device pool and that device is a RAID device of some
sort, and ZFS is getting errors from that device. From ZFS' point of
view this is disastrous. From your point of view this shouldn't happen
because your RAID device ought to save your bacon from single disk
failures (depending on how it's configured).
RAID devices aren't magical -- they can't detect certain kinds of errors
that ZFS can. But ZFS can only recover from those errors -- provided it
itself is in charge of the RAIDing and there are enough remaining good
devices to reconstruct the correct data. (Then there's ditto blocks,
but I don't recall the status of that, which would let you add some
degree of redundancy without having to add devices.)
The main reason for wanting to use hardware RAID with ZFS is to
significantly reduce the amount of I/O that ZFS has to do (for a 5 disk
RAID-5 we're talking about 5 times less I/O for the host to do), but
because this means that ZFS can't do combinatorial reconstruction of bad
disk data you want to then add mirroring (which also adds I/Os) so that
ZFS can cope with bad data from the RAID device.
How is your RAID device configured? Does it have any diagnostics?
Or is your RAID device silently corrupting data? If so ZFS saved you by
detecting that, but because you did not have enough redundancy (from
ZFS' point of view) ZFS can't actually reconstruct the correct data, and
so you lose.
Nico
--
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
59
From:
Registered:
4/10/06
|
|
|
|
Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 6:38 PM
in response to: Elizabeth Schwa...
To: Communities » zfs » discuss
|
|
Oh my, one day after I posted my horror story another one strikes. This is validation of the design objectives of ZFS, looks like this type of stuff happens more often than not. In the past we'd have just attributed this type of problem to some application induced corruption, now ZFS is pinning this problem squarely on the storage sub-system.
If you didn't do any ZFS redundancy then your data is DONE as the support person indicated. Make sure you follow the instructions in the ZFS FAQ otherwise your server will end up in an endless 'panic-reboot cycle'.
Don't shoot the messenger (ZFS), consider running diags on your storage sub-system. Good luck.
|
|
|
|
|
Elizabeth Schwa...
betsy.schwartz@gmail...
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 7:01 PM
in response to: asrirama
|
|
Well, I fixed the HW but I had one bad file, and the problem was that ZFS was saying "delete the pool and restore from tape" when, it turns out, the answer is just find the file with the bad inode, delete it, clear the device and scrub. Maybe more of a documentation problme, but it sure is disconcerting to have a file system threatening to give up the game over one bad file (and the real irony: it was a file in someone's TRASH!)
Anyway I'm back in business without a restore (and with a rebuilt RAID) but yeesh, it sure took a lot of escalating to get to the point where someone knew to tell me to do a find -inum.
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Toby Thain
toby@smartgames.ca
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 7:19 PM
in response to: Elizabeth Schwa...
|
|
On 28-Nov-06, at 10:01 PM, Elizabeth Schwartz wrote:
> Well, I fixed the HW but I had one bad file, and the problem was
> that ZFS was saying "delete the pool and restore from tape" when,
> it turns out, the answer is just find the file with the bad inode,
> delete it, clear the device and scrub. Maybe more of a
> documentation problme, but it sure is disconcerting to have a file
> system threatening to give up the game over one bad file (and the
> real irony: it was a file in someone's TRASH!)
>
> Anyway I'm back in business without a restore (and with a rebuilt
> RAID) but yeesh, it sure took a lot of escalating to get to the
> point where someone knew to tell me to do a find -inum.
Do you now have a redundant ZFS configuration, to prevent future data
loss/inconvenience?
--T
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris dot org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
279
From:
US
Registered:
7/7/06
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 7:29 PM
in response to: Elizabeth Schwa...
|
|
On 11/28/06, Elizabeth Schwartz <betsy dot schwartz at gmail dot com> wrote:
> Well, I fixed the HW but I had one bad file, and the problem was that ZFS
> was saying "delete the pool and restore from tape" when, it turns out, the
> answer is just find the file with the bad inode, delete it, clear the device
> and scrub. Maybe more of a documentation problme, but it sure is
> disconcerting to have a file system threatening to give up the game over one
> bad file (and the real irony: it was a file in someone's TRASH!)
The ZFS documentation was assuming you wanted to recover the data, not
abandon it. Which, realistically, isn't always what people want; when
you know a small number of files are trashed, it's often easier to
delete those files and either just go on, or restore only those files,
compared to a full restore. So yeah, perhaps the documentation could
be more helpful in that situation.
> Anyway I'm back in business without a restore (and with a rebuilt RAID) but
> yeesh, it sure took a lot of escalating to get to the point where someone
> knew to tell me to do a find -inum.
Ah, if people here had realized that's what you needed to know, many
of us could have told you I'm sure. I, at least, hadn't realized that
was one of the problem points. (Probably too focused on the ZFS
content to think about the general issues enough!)
Very glad you're back in service, anyway!
--
David Dyer-Bennet, <mailto:dd-b@dd-b.net>, <http://
RKBA: <http://
Pics: <http://
Dragaera/Steven Brust: <http://
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
59
From:
Registered:
4/10/06
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Nov 28, 2006 9:16 PM
in response to: Elizabeth Schwa...
To: Communities » zfs » discuss
|
|
Glad it worked for you. I suspect in your case the corruption happened way down in the tree and you could get around it by pruning the tree (rm the file) below the point of corruption. I suspect this could be due to a very localized corruption like Alpha particle problem where a bit was flipped on the platter or in the cache of the storage sub-system before destaging to disk. In our case the problem was pervasive due to the problems affecting our data path (FC).
[b]You do raise a very very valid point.[/b] It'd be nice if ZFS provided better diagnostics; namely identify where exactly in the tree it found corruption. At that point we can determine if the remedy is to contain our damage (similar to fsck discarding all suspect inodes) and continue.
For example I've very high regard for the space management in the Oracle DB. When it finds a bad block(s) it prints out the address of the block and marks it corrupt. [b]It doesn't put the whole file/tablespace/table/index in 'suspect' mode like ZFS[/b]. DBA can then either drop the table/index that contains the bad block or extract data from the table minus the bad block. Oracle DB handles it very gracefully giving the user/DBA a chance to recover the known good data.
For ZFS to achieve wide acceptance we [b]must[/b] have the ability to pin point the problem area and take remedial action (rm for example) not simply give up. Yes there are times when the corruption could affect a block high up in the chain making the situation hopeless, in such a case we'd have to discard and restart. ZFS now has solved one part of the problem, namely identifying bad data and doing it reliably. It provides resiliency in the form form of Raid-Z(2) and Raid-1. For it to realize its full potential it must also provide tools to discard corrupt parts (branches) of the tree and give us a chance to save the remaining data. We won't always have the luxury of rebuilding the pool and restoring in a production environment.
Easier said than done, me thinks.
Good night.
|
|
|
|
|
Posts:
404
From:
US
Registered:
11/2/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Nov 29, 2006 7:43 AM
in response to: Elizabeth Schwa...
|
|
Hi Betsy,
Yes, part of this is a documentation problem.
I recently documented the find -inum scenario in the community version
of the admin guide. Please see page 156, (well, for next time) here:
http://opensolaris.org/os/community/zfs/docs/
We're working on the larger issue as well.
Cindy
Elizabeth Schwartz wrote:
> Well, I fixed the HW but I had one bad file, and the problem was that
> ZFS was saying "delete the pool and restore from tape" when, it turns
> out, the answer is just find the file with the bad inode, delete it,
> clear the device and scrub. Maybe more of a documentation problme, but
> it sure is disconcerting to have a file system threatening to give up
> the game over one bad file (and the real irony: it was a file in
> someone's TRASH!)
>
> Anyway I'm back in business without a restore (and with a rebuilt RAID)
> but yeesh, it sure took a lot of escalating to get to the point where
> someone knew to tell me to do a find -inum.
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris dot org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
802
From:
Plano, TX
Registered:
4/27/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 1, 2006 8:50 AM
in response to: Elizabeth Schwa...
|
|
On Tue, 28 Nov 2006, Elizabeth Schwartz wrote:
> Well, I fixed the HW but I had one bad file, and the problem was that ZFS
Hi Elizabeth,
Followup: When you say you "fixed the HW", I'm curious as to what you
found and if this experience with ZFS convinced you that your trusted RAID
H/W did, in fact, have issues?
Do you think that it's likely that there are others running production
systems on RAID systems that they trust, but don't realize may have bugs
(causing data corruption) that have yet to be discovered?
> was saying "delete the pool and restore from tape" when, it turns out, the
> answer is just find the file with the bad inode, delete it, clear the device
> and scrub. Maybe more of a documentation problme, but it sure is
> disconcerting to have a file system threatening to give up the game over one
> bad file (and the real irony: it was a file in someone's TRASH!)
>
> Anyway I'm back in business without a restore (and with a rebuilt RAID) but
> yeesh, it sure took a lot of escalating to get to the point where someone
> knew to tell me to do a find -inum.
>
Al Hopper Logical Approach Inc, Plano, TX. al@logical-approach.com
Voice: 972.379.2133 Fax: 972.379.2134 Timezone: US CDT
OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005
OpenSolaris Governing Board (OGB) Member - Feb 2006
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
44
From:
Utah
Registered:
9/25/06
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 1, 2006 3:29 PM
in response to: alhopper
|
|
|
|
On Dec 1, 2006, at 9:50 AM, Al Hopper wrote:
> Followup: When you say you "fixed the HW", I'm curious as to what you
> found and if this experience with ZFS convinced you that your
> trusted RAID
> H/W did, in fact, have issues?
>
> Do you think that it's likely that there are others running production
> systems on RAID systems that they trust, but don't realize may have
> bugs
> (causing data corruption) that have yet to be discovered?
And this is different from any other storage system, how? (ie, JBOD
controllers and disks can also have subtle bugs that corrupt data)
Chad
---
Chad Leigh -- Shire.Net LLC
Your Web App and Email hosting provider
chad at shire.net
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
164
From:
Registered:
3/9/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 1, 2006 3:34 PM
in response to: chadbag
|
|
Chad Leigh -- Shire.Net LLC wrote:
>
> On Dec 1, 2006, at 9:50 AM, Al Hopper wrote:
>
>> Followup: When you say you "fixed the HW", I'm curious as to what you
>> found and if this experience with**** convinced you that your trusted
>> RAID
>> H/W did, in fact, have issues?
>>
>> Do you think that it's likely that there are others running production
>> systems on RAID systems that they trust, but don't realize may have bugs
>> (causing data corruption) that have yet to be discovered?
>
> And this is different from any other storage system, how? (ie, JBOD
> controllers and disks can also have subtle bugs that corrupt data)
Of course, but there isn't the expectation of data reliability with a
JBOD that there is with some RAID configurations.
****
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
44
From:
Utah
Registered:
9/25/06
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 1, 2006 3:36 PM
in response to: myers
|
|
|
|
On Dec 1, 2006, at 4:34 PM, **** H. Myers wrote:
> Chad Leigh -- Shire.Net LLC wrote:
>>
>> On Dec 1, 2006, at 9:50 AM, Al Hopper wrote:
>>
>>> Followup: When you say you "fixed the HW", I'm curious as to what
>>> you
>>> found and if this experience with ZFS convinced you that your
>>> trusted
>>> RAID
>>> H/W did, in fact, have issues?
>>>
>>> Do you think that it's likely that there are others running
>>> production
>>> systems on RAID systems that they trust, but don't realize may
>>> have bugs
>>> (causing data corruption) that have yet to be discovered?
>>
>> And this is different from any other storage system, how? (ie, JBOD
>> controllers and disks can also have subtle bugs that corrupt data)
>
> Of course, but there isn't the expectation of data reliability with a
> JBOD that there is with some RAID configurations.
>
There is not? People buy disk drives and expect them to corrupt
their data? I expect the drives I buy to work fine (knowing that
there could be bugs etc in them, the same as with my RAID systems).
Chad
---
Chad Leigh -- Shire.Net LLC
Your Web App and Email hosting provider
chad at shire.net
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
164
From:
Registered:
3/9/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 1, 2006 4:38 PM
in response to: chadbag
|
|
Chad Leigh -- Shire.Net LLC wrote:
>
> On Dec 1, 2006, at 4:34 PM, **** H. Myers wrote:
>
>> Chad Leigh -- Shire.Net LLC w****:
>>>
>>> On Dec 1, 2006, at 9:50 AM, Al Hopper wrote:
>>>
>>>> Followup: When you say you "fixed the HW", I'm curious as to what you
>>>> found and if this experience with ZFS convinced you that your trusted
>>>> RAID
>>>> H/W did, in fact, have issues?
>>>>
>>>> Do you think that it's likely that there are others running production
>>>> systems on RAID systems that they trust, but don't realize may have
>>>> bugs
>>>> (causing data corruption) that have yet to be discovered?
>>>
>>> And this is different from any other storage system, how? (ie, JBOD
>>> controllers and disks can also have subtle bugs that corrupt data)
>>
>> Of course, but there isn't the expectation of data reliability with a
>> JBOD that there is with some RAID configurations.
>>
>
> There is not? People buy disk drives and expect them to corrupt their
> data? I expect the drives I buy to work fine (knowing that there could
> be bugs etc in them, the same as with my RAID systems).
So, what do you think reliable RAID configurations are for?
****
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 1, 2006 9:17 PM
in response to: chadbag
|
|
Chad Leigh -- Shire.Net LLC wrote:
>
> On Dec 1, 2006, at 4:34 PM, **** H. Myers wrote:
>
>> Chad Leigh -- Shire.Net LLC wrote:
>>
>>>
>>> And this is different from any other storage system, how? (ie, JBOD
>>> controllers and disks can also have subtle bugs that corrupt data)
>>
>>
>> Of course, but there isn't the expectation of data reliability with a
>> JBOD that there is with some RAID configurations.
>>
>
> There is not? People buy disk drives and expect them to corrupt
> their data? I expect the drives I buy to work fine (knowing that
> there could be bugs etc in them, the same as with my RAID systems).
>
So you trust your important data to a single drive? I doubt it. But I
bet you do trust your data to a hardware RAID array.
Ian
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
44
From:
Utah
Registered:
9/25/06
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 1, 2006 9:48 PM
in response to: ian
|
|
|
|
On Dec 1, 2006, at 10:17 PM, Ian Collins wrote:
> Chad Leigh -- Shire.Net LLC wrote:
>
>>
>> On Dec 1, 2006, at 4:34 PM, **** H. Myers wrote:
>>
>>> Chad Leigh -- Shire.Net LLC wrote:
>>>
>>>>
>>>> And this is different from any other storage system, how? (ie,
>>>> JBOD
>>>> controllers and disks can also have subtle bugs that corrupt data)
>>>
>>>
>>> Of course, but there isn't the expectation of data reliability
>>> with a
>>> JBOD that there is with some RAID configurations.
>>>
>>
>> There is not? People buy disk drives and expect them to corrupt
>> their data? I expect the drives I buy to work fine (knowing that
>> there could be bugs etc in them, the same as with my RAID systems).
>>
> So you trust your important data to a single drive? I doubt it.
> But I
> bet you do trust your data to a hardware RAID array.
Yes, but not because I expect a single drive to be more error prone
(versus total failure). Total drive failure on a single disk loses
all your data. But we are not talking total failure, we are talking
errors that corrupt data. I buy individual drives with the
expectation that they are designed to be error free and are error
free for the most part and I do not expect a RAID array to be more
robust in this regard (after all, the RAID is made up of a bunch of
single drives).
Some people on this list think that the RAID arrays are more likely
to corrupt your data than JBOD (both with ZFS on top, for example, a
ZFS mirror of 2 raid arrays or a JBOD mirror or raidz). There is no
proof of this or even reasonable hypothetical explanation for this
that I have seen presented.
Chad
>
> Ian
---
Chad Leigh -- Shire.Net LLC
Your Web App and Email hosting provider
chad at shire.net
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 1, 2006 11:06 PM
in response to: chadbag
|
|
Chad Leigh -- Shire.Net LLC wrote:
>
> On Dec 1, 2006, at 10:17 PM, Ian Collins wrote:
>
>> Chad Leigh -- Shire.Net LLC wrote:
>>
>>> There is not? People buy disk drives and expect them to corrupt
>>> their data? I expect the drives I buy to work fine (knowing that
>>> there could be bugs etc in them, the same as with my RAID systems).
>>>
>> So you trust your important data to a single drive? I doubt it. But I
>> bet you do trust your data to a hardware RAID array.
>
>
> Yes, but not because I expect a single drive to be more error prone
> (versus total failure). Total drive failure on a single disk loses
> all your data. But we are not talking total failure, we are talking
> errors that corrupt data. I buy individual drives with the
> expectation that they are designed to be error free and are error
> free for the most part and I do not expect a RAID array to be more
> robust in this regard (after all, the RAID is made up of a bunch of
> single drives).
>
But people expect RAID to protect them from the corruption caused by a
partial failure, say a bad block, which is a common failure mode. The
worst system failure I experienced was caused by one half of a mirror
experiencing bad blocks and the corrupt data being nicely mirrored on
the other drive. ZFS would have saved this system from failure.
> Some people on this list think that the RAID arrays are more likely
> to corrupt your data than JBOD (both with ZFS on top, for example, a
> ZFS mirror of 2 raid arrays or a JBOD mirror or raidz). There is no
> proof of this or even reasonable hypothetical explanation for this
> that I have seen presented.
>
I don't think that the issue here, it's more one of perceived data
integrity. People who have been happily using a single RAID 5 are now
finding that the array has been silently corrupting their data. People
expect errors form single drives, so they put them in a RAID knowing the
firmware will protect them from drive errors. They often fail to
recognise that the RAID firmware may not be perfect.
ZFS looks to be the perfect tool for mirroring hardware RAID arrays,
with the advantage over other schemes of knowing which side of the
mirror has an error. Thus ZFS can be used as a tool to compliment,
rather than replace hardware RAID.
Ian
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
44
From:
Utah
Registered:
9/25/06
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 1, 2006 11:15 PM
in response to: ian
|
|
|
|
On Dec 2, 2006, at 12:06 AM, Ian Collins wrote:
> Chad Leigh -- Shire.Net LLC wrote:
>
>>
>> On Dec 1, 2006, at 10:17 PM, Ian Collins wrote:
>>
>>> Chad Leigh -- Shire.Net LLC wrote:
>>>
>>>> There is not? People buy disk drives and expect them to corrupt
>>>> their data? I expect the drives I buy to work fine (knowing that
>>>> there could be bugs etc in them, the same as with my RAID systems).
>>>>
>>> So you trust your important data to a single drive? I doubt
>>> it. But I
>>> bet you do trust your data to a hardware RAID array.
>>
>>
>> Yes, but not because I expect a single drive to be more error prone
>> (versus total failure). Total drive failure on a single disk loses
>> all your data. But we are not talking total failure, we are talking
>> errors that corrupt data. I buy individual drives with the
>> expectation that they are designed to be error free and are error
>> free for the most part and I do not expect a RAID array to be more
>> robust in this regard (after all, the RAID is made up of a bunch of
>> single drives).
>>
> But people expect RAID to protect them from the corruption caused by a
> partial failure, say a bad block, which is a common failure mode.
They do? I must admit no experience with the big standalone raid
array storage units, just (expensive) HW raid cards, but I have never
expected an array to protect me against data corruption. Bad blocks
can be detected and remapped, and maybe the array can recalculate the
block from parity etc, but that is a known disk error, and not the
subtle kinds of errors created by the RAID array that are being
claimed here.
> The
> worst system failure I experienced was caused by one half of a mirror
> experiencing bad blocks and the corrupt data being nicely mirrored on
> the other drive. ZFS would have saved this system from failure.
None of my comments are meant to denigrate ZFS. I am implementing it
myself.
>
>> Some people on this list think that the RAID arrays are more likely
>> to corrupt your data than JBOD (both with ZFS on top, for example, a
>> ZFS mirror of 2 raid arrays or a JBOD mirror or raidz). There is no
>> proof of this or even reasonable hypothetical explanation for this
>> that I have seen presented.
>>
> I don't think that the issue here, it's more one of perceived data
> integrity. People who have been happily using a single RAID 5 are now
> finding that the array has been silently corrupting their data.
They are? They are being told that the problems they are having is
due to that but there is no proof. It could be a bad driver for
example.
> People
> expect errors form single drives,
They do? The tech specs show very low failure rates for single
drives in terms of bit errors.
> so they put them in a RAID knowing the
> firmware will protect them from drive errors.
The RAID firmware will not protect them from bit errors on block
reads unless the disk detects that the whole block is bad. I admit
not knowing how much the disk itself can detect bit errors with CRC
or similar sorts of things.
> They often fail to
> recognise that the RAID firmware may not be perfect.
ZFS, JBOS disk controllers, drivers for said disk controllers, etc
may not be perfect either.
>
> ZFS looks to be the perfect tool for mirroring hardware RAID arrays,
> with the advantage over other schemes of knowing which side of the
> mirror has an error. Thus ZFS can be used as a tool to compliment,
> rather than replace hardware RAID.
I agree. That is what I am doing :-)
Chad
>
> Ian
>
---
Chad Leigh -- Shire.Net LLC
Your Web App and Email hosting provider
chad at shire.net
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
164
From:
Registered:
3/9/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 1, 2006 11:32 PM
in response to: chadbag
|
|
Chad Leigh -- Shire.Net LLC wrote:
>
> On Dec 1, 2006, at 10:17 PM, Ian Collins wrote:
>
>> Chad Leigh -- Shire.Net LLC wrote:
>>
>>>
>>> On Dec 1, 2006, at 4:34 PM, **** H. Myers wrote:
>>>
>>>> Chad Leigh -- Shire.Net LLC wrote:
>>>>
>>>>>
>>>>> And this is different from any other storage system, how? (ie, JBOD
>>>>> controllers and disks can also have subtle bugs that corrupt data)
>>>>
>>>>
>>>> Of course, but there isn't the expectation of data reliability with a
>>>> JBOD that there is with some RAID configurations.
>>>>
>>>
>>> There is not? People buy disk drives and expect them to corrupt
>>> their data? I expect the drives I buy to work fine (knowing that
>>> there could be bugs etc in them, the same as with my RAID systems).
>>>
>> So you trust your important data to a single drive? I doubt it. But I
>> bet you do trust your data to a hardware RAID array.
>
> Yes, but not because I expect a single drive to be more error prone
> (versus total failure).
Ah. I was guessing you were thinking this. You believe that a single
disk is no more prone to a soft data err****han a reliable RAID configuration.
This is arguably incorrect. Soft errors and hard failures are pretty
similar in that they both result in data corruption - where they differ
is in the recovery.
While I can not quote a specific figure, disk drives are not immune
from soft errors. Normally, the drive is able to detect the error
and correct it transparently. As a result, the apparent soft error
rate is quite low for a typical drive. However, there are limits to the
soft errors that a drive can detect and correct; it is possible for
an error to slip through the drive's controller without correction if it
is more than some number of bits. I don't actually know current estimates
for undetected soft errors, but it's small, like 1 in 10^30 bits or more. My
guess could be wildly wrong - it doesn't change the outcome. Perhaps
one of the actual disk experts can help here :-).
Thus, undetected soft errors don't happen very often but they do happen.
Further, the drive is attached to the computer via a cable and electronics
which themselves are not immune to errors. Again, the probability of an
error is very small, so small that we take for granted the reliability
of the data.
A single disk can have an undetected error - if the OS doesn't check, it'll
never notice either. Since this happens rarely, when it does happen, it
may not even be noticed, or have a lasting impact. If a record is read from
a database with an error but not modified, the error won't be written-back
to the disk and may not occur again. An application may crash, and simply be
restarted. Since programming errors occur far more frequently than soft read
errors (they do), it's probable that buggy software will be blamed rather
than a soft error from a disk.
> Total drive failure on a single disk loses all
> your data. But we are not talking total failure, we are talking errors
> that corrupt data. I buy individual drives with the expectation that
> they are designed to be error free
That's not a reasonable expectation, but, for the above reasons, you may
retire and never have seen something that you believed was a soft error.
The very nature of soft errors tends to hide them, while the nature of
a hard failure exacerbates them.
> and are error free for the most part
For the most part? They're not always error-free? Remember, for every
so many detected errors, there's probably an undetected error.
> and I do not expect a RAID array to be more robust in this regard (after
> all, the RAID is made up of a bunch of single drives).
Some RAID configurations are not more robust than a single drive - a simple
stripe/concat for example. It doesn't matter what kind of error is encountered,
a soft error or a hard failure, there's no redundancy of the data. A hard
error can't be ignored, but a soft error probably is.
Reliable RAID configurations can tolerate at least one error - be it soft
or hard. The real difference is the recovery protocol.
> Some people on this list think that the RAID arrays are more likely to
> corrupt your data than JBOD (both with ZFS on top, for example, a ZFS
> mirror of 2 raid arrays or a JBOD mirror or raidz). There is no proof
> of this or even reasonable hypothetical explanation for this that I have
> seen presented.
We've had an example presented just last week in the field where a RAID
array running in a reliable configuration returned corrupt data as a result
of a faulty interconnection. The error was detected by ZFS, but recovery
of the data was not possible because the RAID array was trusted to maintain
data integrity and not ZFS. The RAID array could only insure the integrity
of the data inside the RAID array but can not detect or correct errors
occurring in the interconnect.
If ZFS had been managing the disks in that array as JBOD, and the disks
were in a reliable configuration, the interconnect errors would have most
likely have been detected and corrected transparently, since ZFS is managing
the data at the very top level in the computer, on top of all interconnection
and disks.
So it doesn't matter how often soft errors occur in a single disk; a reliable
configuration will reduce the rate, and a reliable ZFS configuration will
protect against errors in the entire path to the disks.
****
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
802
From:
Plano, TX
Registered:
4/27/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 2, 2006 9:29 AM
in response to: chadbag
|
|
On Fri, 1 Dec 2006, Chad Leigh -- Shire.Net LLC wrote:
>
> On Dec 1, 2006, at 10:17 PM, Ian Collins wrote:
>
> > Chad Leigh -- Shire.Net LLC wrote:
> >
> >>
> >> On Dec 1, 2006, at 4:34 PM, **** H. Myers wrote:
> >>
> >>> Chad Leigh -- Shire.Net LLC wrote:
> >>>
> >>>>
> >>>> And this is different from any other storage system, how? (ie,
> >>>> JBOD
> >>>> controllers and disks can also have subtle bugs that corrupt data)
> >>>
> >>>
> >>> Of course, but there isn't the expectation of data reliability
> >>> with a
> >>> JBOD that there is with some RAID configurations.
> >>>
> >>
> >> There is not? People buy disk drives and expect them to corrupt
> >> their data? I expect the drives I buy to work fine (knowing that
> >> there could be bugs etc in them, the same as with my RAID systems).
> >>
> > So you trust your important data to a single drive? I doubt it.
> > But I
> > bet you do trust your data to a hardware RAID array.
>
> Yes, but not because I expect a single drive to be more error prone
> (versus total failure). Total drive failure on a single disk loses
> all your data. But we are not talking total failure, we are talking
> errors that corrupt data. I buy individual drives with the
> expectation that they are designed to be error free and are error
> free for the most part and I do not expect a RAID array to be more
> robust in this regard (after all, the RAID is made up of a bunch of
> single drives).
>
> Some people on this list think that the RAID arrays are more likely
> to corrupt your data than JBOD (both with ZFS on top, for example, a
> ZFS mirror of 2 raid arrays or a JBOD mirror or raidz). There is no
Can you present a cut/paste where that assertion was made?
> proof of this or even reasonable hypothetical explanation for this
> that I have seen presented.
>
> Chad
>
> >
> > Ian
>
> ---
> Chad Leigh -- Shire.Net LLC
> Your Web App and Email hosting provider
> chad at shire.net
>
>
Al Hopper Logical Approach Inc, Plano, TX. al@logical-approach.com
Voice: 972.379.2133 Fax: 972.379.2134 Timezone: US CDT
OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005
OpenSolaris Governing Board (OGB) Member - Feb 2006
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
1,091
From:
CA
Registered:
4/27/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 2, 2006 11:43 AM
in response to: alhopper
|
|
On Sat, 2 Dec 2006, Al Hopper wrote:
> > Some people on this list think that the RAID arrays are more likely
> > to corrupt your data than JBOD (both with ZFS on top, for example, a
> > ZFS mirror of 2 raid arrays or a JBOD mirror or raidz). There is no
>
> Can you present a cut/paste where that assertion was made?
I don't want to put words in Chad's mouth, but I think he might be
misunderstanding representations that people make here about ZFS vs
HW RAID. I don't think that people have asserted that "RAID arrays
are more likely to corrupt data than a JBOD"; what I think people ARE
asserting is that corruption is more likely to go undetected in a HW
RAID than in a JBOD with ZFS. (A subtle, but important, difference.)
The reason for this is understandable: if you write some data to a
HW RAID device, you assume that unless otherwise notified, your data
is safe. The HW RAID, but its very nature, is a black box that we
assume is OK. With ZFS+JBOD, ZFS' built in end-to-end error checking
will catch any silent errors created in the JBOD, when they happen,
and can correct them (or at least notify you) right away.
--
Rich Teer, SCSA, SCNA, SCSECA, OpenSolaris CAB member
President,
Rite Online Inc.
Voice: +1 (250) 979-1638
URL: http://www.rite-group.com/rich
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Toby Thain
toby@smartgames.ca
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 1, 2006 9:42 PM
in response to: chadbag
|
|
On 1-Dec-06, at 6:36 PM, Chad Leigh -- Shire.Net LLC wrote:
>
> On Dec 1, 2006, at 4:34 PM, **** H. Myers wrote:
>
>> Chad Leigh -- Shire.Net LLC wrote:
>>>
>>> On Dec 1, 2006, at 9:50 AM, Al Hopper wrote:
>>>
>>>> Followup: When you say you "fixed the HW", I'm curious as to
>>>> what you
>>>> found and if this experience with ZFS convinced you that your
>>>> trusted
>>>> RAID
>>>> H/W did, in fact, have issues?
>>>>
>>>> Do you think that it's likely that there are others running
>>>> production
>>>> systems on RAID systems that they trust, but don't realize may
>>>> have bugs
>>>> (causing data corruption) that have yet to be discovered?
>>>
>>> And this is different from any other storage system, how? (ie, JBOD
>>> controllers and disks can also have subtle bugs that corrupt data)
>>
>> Of course, but there isn't the expectation of data reliability with a
>> JBOD that there is with some RAID configurations.
>>
>
> There is not? People buy disk drives and expect them to corrupt
> their data? I expect the drives I buy to work fine (knowing that
> there could be bugs etc in them, the same as with my RAID systems).
Yes, but in either case, ZFS will tell you. Other filesystems in
general cannot.
--Toby
>
> Chad
>
> ---
> Chad Leigh -- Shire.Net LLC
> Your Web App and Email hosting provider
> chad at shire.net
>
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris dot org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
44
From:
Utah
Registered:
9/25/06
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 1, 2006 9:49 PM
in response to: Toby Thain
|
|
|
|
On Dec 1, 2006, at 10:42 PM, Toby Thain wrote:
>
> On 1-Dec-06, at 6:36 PM, Chad Leigh -- Shire.Net LLC wrote:
>
>>
>> On Dec 1, 2006, at 4:34 PM, **** H. Myers wrote:
>>
>>> Chad Leigh -- Shire.Net LLC wrote:
>>>>
>>>> On Dec 1, 2006, at 9:50 AM, Al Hopper wrote:
>>>>
>>>>> Followup: When you say you "fixed the HW", I'm curious as to
>>>>> what you
>>>>> found and if this experience with ZFS convinced you that your
>>>>> trusted
>>>>> RAID
>>>>> H/W did, in fact, have issues?
>>>>>
>>>>> Do you think that it's likely that there are others running
>>>>> production
>>>>> systems on RAID systems that they trust, but don't realize may
>>>>> have bugs
>>>>> (causing data corruption) that have yet to be discovered?
>>>>
>>>> And this is different from any other storage system, how? (ie,
>>>> JBOD
>>>> controllers and disks can also have subtle bugs that corrupt data)
>>>
>>> Of course, but there isn't the expectation of data reliability
>>> with a
>>> JBOD that there is with some RAID configurations.
>>>
>>
>> There is not? People buy disk drives and expect them to corrupt
>> their data? I expect the drives I buy to work fine (knowing that
>> there could be bugs etc in them, the same as with my RAID systems).
>
> Yes, but in either case, ZFS will tell you.
And then kill your whole pool :-)
> Other filesystems in general cannot.
>
While other file systems, when they become corrupt, allow you to
salvage data :-)
Chad
> --Toby
>
>>
>> Chad
>>
>> ---
>> Chad Leigh -- Shire.Net LLC
>> Your Web App and Email hosting provider
>> chad at shire.net
>>
>>
>>
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at opensolaris dot org
>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
---
Chad Leigh -- Shire.Net LLC
Your Web App and Email hosting provider
chad at shire.net
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
3,398
From:
Registered:
3/9/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 2, 2006 5:01 AM
in response to: chadbag
|
|
>While other file systems, when they become corrupt, allow you to
>salvage data :-)
They allow you to salvage what you *think* is your data.
But in reality, you have no clue what the disks are giving you.
Casper
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
44
From:
Utah
Registered:
9/25/06
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 2, 2006 8:50 AM
in response to: casper
|
|
|
|
On Dec 2, 2006, at 6:01 AM, Casper.***@Sun.COM wrote:
>
>> While other file systems, when they become corrupt, allow you to
>> salvage data :-)
>
>
> They allow you to salvage what you *think* is your data.
>
> But in reality, you have no clue what the disks are giving you.
I stand by what I said. If you have a massive disk failure, yes.
You are right.
When you have subtle corruption, some of the data and meta data is
bad but not all. In that case you can recover (and verify the data
if you have the means to do so) t he parts that did not get
corrupted. My ZFS experience so far is that it basically said the
whole 20GB pool was dead and I seriously doubt all 20GB was corrupted.
Chad
>
> Casper
---
Chad Leigh -- Shire.Net LLC
Your Web App and Email hosting provider
chad at shire.net
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
802
From:
Plano, TX
Registered:
4/27/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 2, 2006 9:56 AM
in response to: chadbag
|
|
On Sat, 2 Dec 2006, Chad Leigh -- Shire.Net LLC wrote:
>
> On Dec 2, 2006, at 6:01 AM, Casper.***@Sun.COM wrote:
>
> >
> >> While other file systems, when they become corrupt, allow you to
> >> salvage data :-)
> >
> >
> > They allow you to salvage what you *think* is your data.
> >
> > But in reality, you have no clue what the disks are giving you.
>
> I stand by what I said. If you have a massive disk failure, yes.
> You are right.
>
> When you have subtle corruption, some of the data and meta data is
> bad but not all. In that case you can recover (and verify the data
> if you have the means to do so) t he parts that did not get
> corrupted. My ZFS experience so far is that it basically said the
> whole 20GB pool was dead and I seriously doubt all 20GB was corrupted.
That was because you built a pool with no redundancy. In the case where
ZFS does not have a redundant config from which to try to reconstruct the
data (today) it simply says: sorry charlie - you pool is corrupt.
Regards,
Al Hopper Logical Approach Inc, Plano, TX. al@logical-approach.com
Voice: 972.379.2133 Fax: 972.379.2134 Timezone: US CDT
OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005
OpenSolaris Governing Board (OGB) Member - Feb 2006
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Toby Thain
toby@smartgames.ca
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 2, 2006 10:23 AM
in response to: alhopper
|
|
On 2-Dec-06, at 12:56 PM, Al Hopper wrote:
> On Sat, 2 Dec 2006, Chad Leigh -- Shire.Net LLC wrote:
>
>>
>> On Dec 2, 2006, at 6:01 AM, Casper.***@Sun.COM wrote:
>>
>>>
>>>> While other file systems, when they become corrupt, allow you to
>>>> salvage data :-)
>>>
>>>
>>> They allow you to salvage what you *think* is your data.
>>>
>>> But in reality, you have no clue what the disks are giving you.
>>
>> I stand by what I said. If you have a massive disk failure, yes.
>> You are right.
>>
>> When you have subtle corruption, some of the data and meta data is
>> bad but not all. In that case you can recover (and verify the data
>> if you have the means to do so) t he parts that did not get
>> corrupted. My ZFS experience so far is that it basically said the
>> whole 20GB pool was dead and I seriously doubt all 20GB was
>> corrupted.
>
> That was because you built a pool with no redundancy. In the case
> where
> ZFS does not have a redundant config from which to try to
> reconstruct the
> data (today) it simply says: sorry charlie - you pool is corrupt.
Is that the whole story though? Even without redundancy, isn't there
a lot of resilience against corruption (redundant metadata, etc)?
--Toby
>
> Regards,
>
> Al Hopper Logical Approach Inc, Plano, TX. al@logical-approach.com
> Voice: 972.379.2133 Fax: 972.379.2134 Timezone: US CDT
> OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005
> OpenSolaris Governing Board (OGB) Member - Feb 2006
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris dot org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
44
From:
Utah
Registered:
9/25/06
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 2, 2006 11:22 AM
in response to: alhopper
|
|
|
|
On Dec 2, 2006, at 10:56 AM, Al Hopper wrote:
> On Sat, 2 Dec 2006, Chad Leigh -- Shire.Net LLC wrote:
>
>>
>> On Dec 2, 2006, at 6:01 AM, Casper.***@Sun.COM wrote:
>>
>>>
>>>> While other file systems, when they become corrupt, allow you to
>>>> salvage data :-)
>>>
>>>
>>> They allow you to salvage what you *think* is your data.
>>>
>>> But in reality, you have no clue what the disks are giving you.
>>
>> I stand by what I said. If you have a massive disk failure, yes.
>> You are right.
>>
>> When you have subtle corruption, some of the data and meta data is
>> bad but not all. In that case you can recover (and verify the data
>> if you have the means to do so) t he parts that did not get
>> corrupted. My ZFS experience so far is that it basically said the
>> whole 20GB pool was dead and I seriously doubt all 20GB was
>> corrupted.
>
> That was because you built a pool with no redundancy. In the case
> where
> ZFS does not have a redundant config from which to try to
> reconstruct the
> data (today) it simply says: sorry charlie - you pool is corrupt.
Where a RAID system would still be salvageable.
Chad
---
Chad Leigh -- Shire.Net LLC
Your Web App and Email hosting provider
chad at shire.net
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
409
From:
Registered:
6/16/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 2, 2006 11:29 AM
in response to: chadbag
|
|
Chad Leigh -- Shire.Net LLC wrote:
>
> On Dec 2, 2006, at 10:56 AM, Al Hopper wrote:
>
>> On Sat, 2 Dec 2006, Chad Leigh -- Shire.Net LLC wrote:
>>
>>>
>>> On Dec 2, 2006, at 6:01 AM, Casper.***@Sun.COM wrote:
>>>
>>>>
>>>>> While other file systems, when they become corrupt, allow you to
>>>>> salvage data :-)
>>>>
>>>> They allow you to salvage what you *think* is your data.
>>>>
>>>> But in reality, you have no clue what the disks are giving you.
>>>
>>>
>>> I stand by what I said. If you have a massive disk failure, yes.
>>> You are right.
>>>
>>> When you have subtle corruption, some of the data and meta data is
>>> bad but not all. In that case you can recover (and verify the data
>>> if you have the means to do so) t he parts that did not get
>>> corrupted. My ZFS experience so far is that it basically said the
>>> whole 20GB pool was dead and I seriously doubt all 20GB was corrupted.
>>
>> That was because you built a pool with no redundancy. In the case where
>> ZFS does not have a redundant config from which to try to reconstruct the
>> data (today) it simply says: sorry charlie - you pool is corrupt.
>
> Where a RAID system would still be salvageable.
That is a comparison of apples to oranges. The RAID system has Redundancy. If
the ZFS pool had been configured with redundancy, it would have fared at least as
well as the RAID system.
Without redundancy, neither of them can magically reconstruct data. The RAID
system would simply be an AID system.
--------------------------------------------------------------------------
Jeff VICTOR Sun Microsystems jeff.victor @ sun.com
OS Ambassador Sr. Technical Specialist
Solaris 10 Zones FAQ: http://www.opensolaris.org/os/community/zones/faq
--------------------------------------------------------------------------
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
44
From:
Utah
Registered:
9/25/06
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 2, 2006 11:34 AM
in response to: jeffv
|
|
|
|
On Dec 2, 2006, at 12:29 PM, Jeff Victor wrote:
> Chad Leigh -- Shire.Net LLC wrote:
>> On Dec 2, 2006, at 10:56 AM, Al Hopper wrote:
>>> On Sat, 2 Dec 2006, Chad Leigh -- Shire.Net LLC wrote:
>>>
>>>>
>>>> On Dec 2, 2006, at 6:01 AM, Casper.***@Sun.COM wrote:
>>>>
>>>>>
>>>>>> While other file systems, when they become corrupt, allow you to
>>>>>> salvage data :-)
>>>>>
>>>>> They allow you to salvage what you *think* is your data.
>>>>>
>>>>> But in reality, you have no clue what the disks are giving you.
>>>>
>>>>
>>>> I stand by what I said. If you have a massive disk failure, yes.
>>>> You are right.
>>>>
>>>> When you have subtle corruption, some of the data and meta data is
>>>> bad but not all. In that case you can recover (and verify the data
>>>> if you have the means to do so) t he parts that did not get
>>>> corrupted. My ZFS experience so far is that it basically said the
>>>> whole 20GB pool was dead and I seriously doubt all 20GB was
>>>> corrupted.
>>>
>>> That was because you built a pool with no redundancy. In the
>>> case where
>>> ZFS does not have a redundant config from which to try to
>>> reconstruct the
>>> data (today) it simply says: sorry charlie - you pool is corrupt.
>> Where a RAID system would still be salvageable.
>
> That is a comparison of apples to oranges. The RAID system has
> Redundancy. If the ZFS pool had been configured with redundancy,
> it would have fared at least as well as the RAID system.
>
> Without redundancy, neither of them can magically reconstruct
> data. The RAID system would simply be an AID system.
That is not the question. Assuming the error came OUT of the RAID
system (which it did in this case as there was a bug in the driver
and the cache did not get flushed in a certain shutdown situation),
another FS would have been salvageable as the whole 20GB of the pool
was not corrupt.
Chad
---
Chad Leigh -- Shire.Net LLC
Your Web App and Email hosting provider
chad at shire.net
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
262
From:
Cardiff, Wales, United Kingdom
Registered:
5/2/06
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 2, 2006 11:35 AM
in response to: chadbag
|
|
On 02/12/06, Chad Leigh -- Shire.Net LLC <chad at shire dot net> wrote:
>
> On Dec 2, 2006, at 10:56 AM, Al Hopper wrote:
>
> > On Sat, 2 Dec 2006, Chad Leigh -- Shire.Net LLC wrote:
> >> On Dec 2, 2006, at 6:01 AM, Casper.***@Sun.COM wrote:
> >> When you have subtle corruption, some of the data and meta data is
> >> bad but not all. In that case you can recover (and verify the data
> >> if you have the means to do so) t he parts that did not get
> >> corrupted. My ZFS experience so far is that it basically said the
> >> whole 20GB pool was dead and I seriously doubt all 20GB was
> >> corrupted.
> > That was because you built a pool with no redundancy. In the case
> > where
> > ZFS does not have a redundant config from which to try to
> > reconstruct the
> > data (today) it simply says: sorry charlie - you pool is corrupt.
> Where a RAID system would still be salvageable.
RAID level what? How is anything salvagable if you lose your only copy?
ZFS does store multiple copies of metadata in a single vdev, so I
assume we're talking about data here.
--
Rasputin :: Jack of All Trades - Master of Nuns
http://number9.hellooperator.net/
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
294
From:
Mendota Heights, Minnesota, USA
Registered:
3/9/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 3, 2006 8:54 PM
in response to: number9
To: Communities » zfs » discuss
|
|
> RAID level what? How is anything salvagable if you
> lose your only copy? [ ... ]
>
> ZFS does store multiple copies of metadata in a
> single vdev, so I
> assume we're talking about data here.
I believe we're talking about metadata, as that is the case where ZFS reports that the pool (as opposed to a dataset) is corrupt.
On a UFS file system, if the superblock is bad, the entire file system would be lost; but the fsck utility can use an alternate copy. On most other file systems, there is sufficient redundancy within the file system that metadata damage results in the loss of only a fraction of data.
ZFS, in fact, also has sufficient redundancy to allow this, I believe. However, there is not currently a file system checker which exploits this to alow a damaged pool to be recovered.
ZFS does keep two or three copies of its metadata, and it seems highly unlikely that both or all three of these would be damaged without easily noticeable error rates (and data being affected much more quickly, simply because there are more data blocks than metadata). It seems likely that a corrupted pool, then, is *not* likely the result of hardware problems, or at least not hardware problems which affect all reads or writes independently.
|
|
|
|
|
Posts:
44
From:
Utah
Registered:
9/25/06
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 6, 2006 10:29 PM
in response to: number9
|
|
|
|
On Dec 2, 2006, at 12:35 PM, **** Davies wrote:
> On 02/12/06, Chad Leigh -- Shire.Net LLC <chad at shire dot net> wrote:
>>
>> On Dec 2, 2006, at 10:56 AM, Al Hopper wrote:
>>
>> > On Sat, 2 Dec 2006, Chad Leigh -- Shire.Net LLC wrote:
>
>> >> On Dec 2, 2006, at 6:01 AM, Casper.***@Sun.COM wrote:
>
>> >> When you have subtle corruption, some of the data and meta data is
>> >> bad but not all. In that case you can recover (and verify the
>> data
>> >> if you have the means to do so) t he parts that did not get
>> >> corrupted. My ZFS experience so far is that it basically said the
>> >> whole 20GB pool was dead and I seriously doubt all 20GB was
>> >> corrupted.
>
>> > That was because you built a pool with no redundancy. In the case
>> > where
>> > ZFS does not have a redundant config from which to try to
>> > reconstruct the
>> > data (today) it simply says: sorry charlie - you pool is corrupt.
>
>> Where a RAID system would still be salvageable.
>
> RAID level what? How is anything salvagable if you lose your only
> copy?
>
The whole raid does not fail -- we are talking about corruption
here. If you lose some inodes your whole partition is not gone.
My ZFS pool would not salvage -- poof, whole thing was gone (granted
it was a test one and not a raidz or mirror yet). But still, for
what happened, I cannot believe that 20G of data got messed up
because a 1GB cache was not correctly flushed.
Sorry for "bailing" on this topic. I did not mean to. Was on
babysitting duty on the weekend and the new week hit with a vengeance
on Monday. I will try and post a last post or two on the subject and
let it die.
Chad
> ZFS does store multiple copies of metadata in a single vdev, so I
> assume we're talking about data here.
---
Chad Leigh -- Shire.Net LLC
Your Web App and Email hosting provider
chad at shire.net
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
80
From:
Registered:
4/27/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 7, 2006 12:12 AM
in response to: chadbag
|
|
> The whole raid does not fail -- we are talking about corruption
> here. If you lose some inodes your whole partition is not gone.
>
> My ZFS pool would not salvage -- poof, whole thing was gone (granted
> it was a test one and not a raidz or mirror yet). But still, for
> what happened, I cannot believe that 20G of data got messed up
> because a 1GB cache was not correctly flushed.
Chad, I think what you're saying is for a zpool to allow you to
salvage whatever remaining data that passes it's checksums.
--
Regards,
Jeremy
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Posts:
1,858
From:
US
Registered:
6/17/05
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 8, 2006 11:02 AM
in response to: nerant
|
|
Jeremy Teo wrote:
>> The whole raid does not fail -- we are talking about corruption
>> here. If you lose some inodes your whole partition is not gone.
>>
>> My ZFS pool would not salvage -- poof, whole thing was gone (granted
>> it was a test one and not a raidz or mirror yet). But still, for
>> what happened, I cannot believe that 20G of data got messed up
>> because a 1GB cache was not correctly flushed.
If you mess up 4 critical blocks (all copies of the same uberblock
metadata) then this is possible. It is expected that this would be a
highly unlikely event since the blocks are spread across the device.
Inserting a cache between what ZFS thinks is a device and the device
complicates this a bit, but it is not clear to me what alternative
would be better for all possible failure modes.
> Chad, I think what you're saying is for a zpool to allow you to
> salvage whatever remaining data that passes it's checksums.
That is the way I read this thread. Perhaps a job for zdb (I speculate
because I've never used zdb)? Perhaps a zdb expert can chime in...
-- richard
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
Toby Thain
toby@smartgames.ca
|
|
|
|
Re: Re: Production ZFS Server Death (06/06)
Posted:
Dec 1, 2006 9:40 PM
in response to: chadbag
|
|
On 1-Dec-06, at 6:29 PM, Chad Leigh -- Shire.Net LLC wrote:
>
> On Dec 1, 2006, at 9:50 AM, Al Hopper wrote:
>
>> Followup: When you say you "fixed the HW", I'm curious as to what you
>> found and if this experience with ZFS convinced you that your
>> trusted RAID
>> H/W did, in fact, have issues?
>>
>> Do you think that it's likely that there are others running
>> production
>> systems on RAID systems that they trust, but don't realize may
>> have bugs
>> (causing data corruption) that have yet to be discovered?
>
> And this is different from any other storage system, how? (ie,
> JBOD controllers and disks can also have subtle bugs that corrupt
> data)
I think Al probably means, "running production systems on [any
trusted storage system where errors can remain
undetected]" (contrasted with ZFS).
--Toby
>
> Chad
>
>
> ---
> Chad Leigh -- Shire.Net LLC
> Your Web App and Email hosting provider
> chad at shire.net
>
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris dot org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris dot org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
|
|
|
|
|
