Skip to content

Comment on HPE Drive fail at 32,768 hours without firmware update

Comments

Those who forget history are doomed to repeat it. Just seven years ago Crucial sold tens of thousands of their "M4" SSDs with a firmware bug that made them fail after 5184 hours: https://www.anandtech.com/show/5424/crucial-provides-a-firmw...

Do they still not test these things with artificially incremented counters?

Boeing didn't even test their 787 aircraft for integer overflows, and that's in a safety-critical environment, so I'm not sure I'd expect SSD vendors to be any better.

https://www.engadget.com/2015/05/01/boeing-787-dreamliner-so...

Not that throwing an exception on integer overflow is any better, unless you catch the exception. The classic example here is the Ariane 5 failure:

http://sunnyday.mit.edu/accidents/Ariane5accidentreport.html

This same problem also led to the loss of the Deep Impact spacecraft on its extended mission:

"On September 20, 2013, NASA abandoned further attempts to contact the craft.[77] According to chief scientist A'Hearn,[78] the reason for the software malfunction was a Y2K-like problem. August 11, 2013, 00:38:49, was 232 tenth-seconds from January 1, 2000, leading to speculation that a system on the craft tracked time in one-tenth second increments since January 1, 2000, and stored it in an unsigned 32-bit integer, which then overflowed at this time, similar to the Year 2038 problem"

https://en.wikipedia.org/wiki/Deep_Impact_(spacecraft)#Conta...

Your superscript got eaten. 2^32

The 787 case is most fascinating in that while the bug is dead simple, a fix is not.

Most importantly, the company's already working on an update that will patch the software vulnerability -- though there's no word on when its jets will receive it.

My search of DDG turned up nothing about a resolution. Anyone know?

I know what I would recommend, but marketing would not like it ;-)

This exact thing happened to me. I went crazy for a week straight testing every other component of my PC. I was convinced it was the graphics cards drawing too much power. Then it was clear that my OS was corrupted and needed to be reinstalled.

Finally, I found an obscure forum post telling me about a firmware bug happening at ~5K hours of disk usage. I updated the firmware and haven't had an issue since.

I thought the coding/design pattern was to set the initial value of any counter 1 minute (or whatever eon makes sense in your application) from the roll-over so you'd see it 'right away' if it was badly handled. It's like you should use specific types with default values...

Some early Intel SSD's did the same thing, prior to M4's... haha

Do you have a link for the old Intel bug? Here's one for a new Intel bug after just 1700 power-on hours on some enterprise-class SSDs that are still being sold today: https://www.intel.com/content/www/us/en/support/articles/000...

That's just 71 days of uptime and they hang. There are tens of thousands of these drives deployed as well.

Oh boy. We had somerhing like 5 our of 8 drives fail all at the same time. All of them were affected models bought at begining of summer and failed a couple months later.

It was a pretty maddening thing to debug and figure out where the issue was (servers, rack, drives, RAID do controllers). 2 different machines 2 and 3 drives. Week later we found the Intel bulletin about the issue.

Thank God for pgbackrest backups.

Why are they allowed to still sell these broken products? That's a scam as far as I'm concerned.

They released a fix for existing devices, replaced affected devices that had been bricked, and included the fix as part of the manufacturing process for new devices being built.

It was certainly very inconvenient having to reboot systems while we waited for a fix to exist, fortunately we didn't lose too many disks before the fault was identified, which reduced the man hours involved in the DCs.

I'm not sure if any part of this is a scam. A bug, certainly.

New drives already had the updated firmware back in May when I bought some.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.