Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

A real benefit would be file system support for Reed-Solomon error correction. Some archival tools support such a feature. You would spent 1%-10% of the disk space for error correction.


Sector failures on a single drive are not independent random events. So doing this on a single drive is not perceived to be a good idea.

The only choices, imo, are multiple disks, or automatic backup to offsite (dropbox, box, one drive, etc).


Sorry, a better answer is that it's already done in the hardware/firmware layer. e.g. from Wikipedia[1]:

""" Modern HDDs present a consistent interface to the rest of the computer, no matter what data encoding scheme is used internally. Typically a DSP in the electronics inside the HDD takes the raw analog voltages from the read head and uses PRML and Reed–Solomon error correction[144] to decode the sector boundaries and sector data, then sends that data out the standard interface. That DSP also watches the error rate detected by error detection and correction, and performs bad sector remapping, data collection for Self-Monitoring, Analysis, and Reporting Technology, and other internal tasks. """

With the answer I gave above, and the fact that it's already being done in hardware, I don't think adding another layer of EC will be fruitful.

[1] https://en.wikipedia.org/wiki/Hard_disk_drive#Access_and_int...


If this is the case, then why does bit-rot still occur?


It's not magic. If you have enough bit flips for the same data then it's not recoverable. Sometimes it could even flip and resemble correct data. It might not happen to you, but if 500 people reading this thread each have access to 50T of data, then sure bit rot can happen in some of that 25 petabytes.


Do you think that there is a need for some kind of ECC at the OS level? There are surely some applications where even 1 bit flip in 25 petabytes is bad. Amazon's 2008 site outage comes to mind: http://status.aws.amazon.com/s3-20080720.html


Yes, that's why you have erasure coding across drives either using RAID groups or using an object store. Just not at the single drive file system level where blocks going bad within a disk tend to be correlated (i.e. the drive is getting old and will die).

Also, EC at the block level would probably spread the blocks around the drive. This means any read would need to seek all over the dang place trying to reassemble files and that's a bad access pattern for spinning disks. Real bad. Like, the worst. It might even reduce the effective lifetime of the drive. So it would be not only correlated with device failure, it could precipitate it.

Maybe it would be ok on an SSD.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: