I'd like to second everything Bill Todd said. And add a few things.
RAID has two functions: It enhances the reliability of your data (the
data is likely to be restorable even after a failure of part of the
system), and it enhances the availability of your data (the data can
be read at all times, even right after a fault, as long as the fault
is of the kind you're protected against). We'll have to look at
likely fault scenarios, and see what it would take to provide
reliability and availability at a level you need. You seem to say
that reliability is absolutely required, but availability is
secondary.
The highest probability of data loss in such a system is not hardware
failure, but brain failure. You cd into the archival file system, and
say "rm *". Or you're formatting some other filesystem, and pick the
wrong /dev/ entry, formatting your multi-terabyte array instead of the
USB stick. Big reliability problem. To guard against that, you need
to backup from your archive to some other file system. That other
file system should be either a dedicated backup solution (which can't
be written to), or disconnected most of the time (so it doesn't fall
prey to similar mistakes).
The next biggest problem is not actual drive failure, but failure of
your RAID firmware or software. This is particularly true if your
RAID array is consumer-grade (although enterprise-grade RAID arrays
have seen their share of data loss events too). RAID data losses are
particularly common ruding rebuild. For that reason, I would keep
rebuild simple (RAID 10), instead of RAID 6. Particularly true for
low-end RAID solutions that don't have NVRAM: for complicated reasons,
it's difficult to do parity-based RAID (RAID 5, 6 etc.) without NVRAM,
which more often than not causes low-end parity-based RAID systems to
be buggy. So you mjight suddenly find your whole array to be
scrambled. Again, backup is your friend.
Remember, it's not a backup until you have successfully restored it.
Part of such a storage solution must be exercising your emergency
procedures. Pull a drive from the RAID array, and watch it go through
rebuild, then reinsert the drive. Use "write long" to cause a parity
error, and make sure the array scrubs it out pretty fast (leaving
lingering errors unscrubbed is a recipe for disaster). Pretend that
your local array has died, and try to restore onto a spare array, and
make your restore procedures actually work.
Speaking of hardware failures: As has been mentioned already, a
combination of a failure of a while drive with a failure of a single
block is dangerously probably on 2TB-class disks. Recommending
single-fault-tolerant RAID 5 arrays this size would be irresponsible.
But much more likely than any of these faults are systemic faults
which take the whole array out, with all the nice redundancy. Ten
years ago, a 10TB array would have been the size of a few
refrigerators, would have been in a computer room with a raised floor,
with a fire suppression system, and its own Liebert power conditioner.
Management was done by a team of highly skilled storage admins, who
have gone to training classes at places like EMC, IBM, or StorageTek.
The array had no single point of failure (redundant power and cooling,
multiple power connections to multiple grids, batteries to power
through short outages). Today it fits on a desktop, people put their
bottles of beer on top of it, it's exposed to all manner of
environmental dangers (beginning with dust clogging the fans, up to
local fires), and it's likely to get fried by "Pakistani Gas and
Electric" (our local power company here in Silicon Valley). It has
probably just one power supply and one cooling fan - and those two
components are actually much less reliable than the disks themselves.
So more likely than a disk or block error is a complete failure of the
whole array, likely caused by an environmental problem (like local
fire in your office, or sprinklers going off by mistake). Again,
backup is your friend, and the backup better be in a different
building, far away (as was clearly demonstrated a few years ago,
putting your backup data center in the other tower of the World Trace
Center is insufficient). On the other hand, some of these systemic
failures (like fan or power supply failure) only leave your array
temporarily disabled (availability), and don't necessarily induce data
loss (reliability). Still, restoring from remote backup might be
faster than repairing your array.
Once you are protected against brain cramp, firmware faults, local
disaster, the protection against disk failure and disk data loss
becomes secondary. Let's get back to the distinction between
reliability and availability. Backup takes care of reliability. Do
you actually need continuous availability? If your power supply in
the RAID array fails, would it bother you if you have to wait 3 days
until the replacement has been shipped? Even assuming that 3 years
from now you can still get spare parts for your disk enclosure (only
likely if you bought the enclosure from a name-brand vendor, Dell,
Sun, IBM, HP, EMC, NetApp ...). If your office catches fire, would it
bother you if it took 3 days to purchase a replacement disk array and
restore it from the remote backup? If you can handle a few days of
downtime, then I would suggest not wasting your money on RAID, and
instead invest it in better remote backup, and more bandwidth. And if
you want to invest in RAID, pick RAID 10 - inefficient, but reliable
and simple.
--
Ralph Becker-Szendy ***@lr_dot_los-gatos_dot_ca_dot_us
735 Sunset Ridge Road; Los Gatos, CA 95033