Four urban legends on hard disk

As a result of using 100,000 hard disks by Google,Hard disk fails regardless of temperature or frequency of useFollowing the awesome report that concluded that, Bianca Schroeder of CMU's Parallel Data Lab said "Failure of hard disk in the real world: What does mean meaningful mean time of 1 million hours?"We issued a report saying" The following common sense concerning the hard disk is the content that it is unfounded.

1. Expensive SCSI / FC drives are more reliable than inexpensive SATA drives
2. The reason that RAID 5 is safe is that the probability of two drives failing at the same time is very low
3. Increasing the trust that it will not break until the service life is reached if it exceeds the early failure breakdown period
4. MTBF (Mean Time Between Failure) as hard disk manufacturer is a useful measure for comparing hard disk drives

I do not know what to believe anymore, but the above conclusion seems to be that the reliability is very low if we base the autocorrelation function in statistics as the basis. Also, as this report is based on exact statistics, it is content that is one step ahead of the report issued by Google in terms of reliability.

Details are as follows.
StorageMojo >> Everything You Know About About Disks Is Wrong

Schroeder who gave this report got these conclusions from 100,000 hard disk drives including HPC (High Performance Computing) clusters such as Los Alamos and the Pittsburgh Super Computer Center and anonymous Internet service providers. Of course these hard disks are not uniform like Google in the definition of work loads, faults etc. However, in order to absorb these differences, Mr. Schroeder seems to use the statistical method considerably carefully.

In this research, there was not so clear difference between expensive high-end SCSI / FC hard disk drives for business and consumer SATA hard disk drives. In this regard, it seems that Google is using a hard disk for civilian use for its server in particular is not surprising.

Regarding the initial failure rate, it is said that the failure rate itself is not constant depending on the age of use, and whether wear or deterioration starts earlier than the initial failure rate is more significant. In other words, it does not matter whether you pass a long driving test that is done before shipping, as is often the case with high end hard disk drives for servers, but that is not related to the failure rate later on.

So what is MTBF (Mean Time Between Failure) that Hard Disk Maker says? For example, "MTBF is 10 years" means "to be broken once every ten years". Similarly, "AFR (annual average failure rate) of 50%" means that it will fail once every two years. In the case of a hard disk drive whose manufacturer's announced AFR falls within the range of 0.58% to 0.88%, the actual observation result was that it ranged from 0.5% up to 13.5% in the actual observation result. On average, the AFR is about 15% of the manufacturer's announced value, about 3%.

That means that what actually MTBF is like is that "MTBF is one million hours" is actually "MTBF becomes about 300,000 hours" in fact. Speaking of the relation between theoretical value and actual numerical value, since both theoretical values ​​and actual measured values ​​exist for USB, wireless LAN, and net, even if the same thing happens to the hard disk drive, it is not worth surprising I do not know, until now no one knew.

The security of RAID 5 (the probability of two units failing at the same time) is changed according to the probability of when the second hard disk failure occurs. In other words, the data is completely lost in RAID 5 if another one is broken while replacing the failed hard disk and rebuilding it. Probabilistically, it is still very low, but in fact it seems that the probability of breaking the other within 4 hours of the theoretical value within 1 hour during replacement. For the sake of simplicity, the fact that one drive constituting RAID 5 is broken means that the probability that another drive will be broken is higher.

Anyhow, if the hard disk drives are attached a lot, it is an illusion that somehow it gets more reliable, but in fact it is not so high, rather it is rather hard to consume more hard disks, so as a result, I conclude that the manufacturer is only profitable.

As you go, it seems correct that that there are three copies of the fundamental mechanism that supports Google's huge search index, as it is certainly more reliable. This mechanism is called "Google File System", and there is a Japanese translation of detailed explanation below.

Ariel Area - Document - Google File System (GFS) Technical Notes

Each chunk is duplicated and synchronized on multiple chunk servers. The default number of chunk replicas is 3.

The full text of this paper is below.

5th USENIX Conference on File and Storage Technologies - Paper

For hard disk drive manufacturers, we would like to ask hard disks that seriously price and durable life are reliably proportional to ... ... not only to aim for large capacity.