Something about these numbers doesn't quite make sense. The reason cited for dismantling the machine is that "it isn't energy-efficient enough to make the power bill worth it." But the supercomputer uses 2345 kilowatts, which at US prices of around 15 cents per kWh would cost $352 / hour to run in energy costs. By comparison, the $120 million cost of building roadrunner, amortized over the four years it's been running, comes out to $3400 / hour. The article makes it sound like the power bill is costing them a fortune, at $3 million a year, it isn't that much at all next to the $120 million price tag.
Disclaimer: while I worked for a DOE supercomputing lab, it wasn't one of the NNSA (nuke) labs, so I don't have all the info. (And even if I did, I certainly wouldn't be allowed to comment on it.)
You're a bit right in that it's more complicated than they say in the article. The power of keeping it up and running is definitely costly, but there's more to it than that.
* It takes people to run it. They're expensive, and you don't want your people wasting a lot of time on the old super while your new one is being installed and accepted. I don't recall having seen any plans for a new super at LANL, but there's probably something new showing up soon.
* The architecture of roadrunner is outdated and obsolete. The NNSA isn't buying any more, and you can bet that with the new Blue Gene/Q at Lawrence Livermore (sequoia), all their new code is being written for different non-roadrunner architectures. There is considerable cost to porting new code to legacy hardware.
* Kind of in line with the last point, sequoia is >17x faster than roadrunner at only 3.5x the power. There's not a lot of point in paying such a premium in flops/watt by running on roadrunner since the new one is already online.
* 2.3MW is only what the computer itself uses. There's a ton of other supporting equipment needed to use it (chillers, disk, control nodes, etc.)
* This computer isn't being manufactured anymore, so maintenance is going to cost a fortune. When they bought it, they either paid for a hardware support contract or bought a ton of spares and did self-maintenance. If they did the former, IBM is either charging through the nose for replacements or refusing to support it outright. If they did the latter, they're probably running low on spares.
The tri-labs have a rotating schedule for acquiring new machines, that include capacity and capability classes (the later being RR). The last time I was in the computing center at LANL the computing floor was quite packed. So, regardless of the claims made in the article for disposal reasons, there are other issues such as total machine weight, cooling requirements (which dominate), making room for new machines, etc...
I do not know if RR will be physically shredded, but I'm hoping that it is retired to the brand-new PRObE (http://nmc-probe.org/) computing center where RR would be available for use by academic researchers.
PRObE will probably get a chunk of it, but they also have power limitations. Basically every piece of the machine has already been given to some researcher or another for some purpose -- there's a bit of a waiting list at the moment.
The hard drives will probably be sent to a military base to be melted down, I don't think they risk shredding them nowadays. The hardware that doesn't store data will probably be kept intact for quite some time.
Since Roadrunner takes up about 1/3 of the Metropolis datacenter, getting rid of it leaves some room for the 2015 "Trinity" machine (assuming LANL hosts it).
There's one thing you're forgetting here: Cell architecture. Noone writes software for that anymore, since it's been discontinued (either officially or inofficially). Most new supercomputers at that scale are powered by a mix of NVIDIA Tesla (with Intel MIC now coming as well) and Opterons/Xeons. This dictates the software stack of present and future research[1]. There's no reason to keep a computer running when it's severely underused.
The cost of building the thing has already been paid. It is a sunk cost. If it costs $352/hour to run the roadrunner and they can only get $150/hour in income to operate it, it don't matter how much it cost to build it.
If it costs $352/hour to run the roadrunner and they can only get $150/hour in income to operate it, it don't matter how much it cost to build it.
Assuming they already paid for the replacement super-computer too etc. Otherwise, sure it matters. Instead of finding $xx-$xxx million to build a more efficient one, you might be better off to pay the electricity bill difference. This is still in the top 25 computers in the world, so it can be used for a lot of things, but we don't know all the details.
It's a distributed batch-job type system, not a single computer with a petaflop core. I don't think it is technically accurate to call it a cluster, but it isn't one machine either.
Unless you've got a big HPC job to run, it wouldn't be very exciting. You log in to a "login node" which is just an ordinary linux server and you get a bash prompt. Nothing very remarkable. To actually do anything you would create a script and submit your job to the workload manager, then come back later for your results.
The 2.5 megawatts Road Runner consumes are not available for other purposes, such as running more efficient computers. In other words there is an opportunity cost associated with a power budget.
Retiring Road Runner may be significantly cheaper than adding additional megawatts of capacity, particularly when considering redundant power services and backup power supplies.
And how much to build it to mil. spec? Probably x10 that number at a minimum. Sure, it's not a military establishment, but I am sure their security and 50 year event survivability is higher than any commercial DC facility.
Add to that the fact that utilization in most DCs is terrible. A super computer will throw out significantly more heat than your "traditional" DC.
Let's not even consider the op. ex. of moving the staff working who would work on the new facilities.
Edit: And LOL at those figures being accurate. That's just the building. What about the costs for the fibre, all that copper, power infrastructure, cooling, false floors, air filtering, special construction needs such as sunken well floors (to contain any leaks from the cooling systems, for example).
You'd be surprised how many locations are inconvenient. You need to find the right nexus of preexisting fiber trenches (digging new ones is super expensive!), cheap power, ideally some environmental factor to make cooling cheap (eg, a river nearby), and acceptable local zoning and laws.
The $120m is sunk costs from a previous building budget, it's not relevant. The more relevant piece is more recent supercomputer (with architectures which are not dead ends) being 4 times as power efficient.
Actually if you compare the Roadrunner against what it would cost to get the equivalent horsepower on Amazon (hypothetically, of course), that price tag is quite a deal.
An economic decision only takes into account marginal expenses (and income for that matter).
At this point the 120M investment is a sunk cost. And the investment was so bad that the operational income can not even cover just the operational expenses, so shutting it down is the wise thing to do.
Comments
Something about these numbers doesn't quite make sense. The reason cited for dismantling the machine is that "it isn't energy-efficient enough to make the power bill worth it." But the supercomputer uses 2345 kilowatts, which at US prices of around 15 cents per kWh would cost $352 / hour to run in energy costs. By comparison, the $120 million cost of building roadrunner, amortized over the four years it's been running, comes out to $3400 / hour. The article makes it sound like the power bill is costing them a fortune, at $3 million a year, it isn't that much at all next to the $120 million price tag.
Disclaimer: while I worked for a DOE supercomputing lab, it wasn't one of the NNSA (nuke) labs, so I don't have all the info. (And even if I did, I certainly wouldn't be allowed to comment on it.)
You're a bit right in that it's more complicated than they say in the article. The power of keeping it up and running is definitely costly, but there's more to it than that.
* It takes people to run it. They're expensive, and you don't want your people wasting a lot of time on the old super while your new one is being installed and accepted. I don't recall having seen any plans for a new super at LANL, but there's probably something new showing up soon.
* The architecture of roadrunner is outdated and obsolete. The NNSA isn't buying any more, and you can bet that with the new Blue Gene/Q at Lawrence Livermore (sequoia), all their new code is being written for different non-roadrunner architectures. There is considerable cost to porting new code to legacy hardware.
* Kind of in line with the last point, sequoia is >17x faster than roadrunner at only 3.5x the power. There's not a lot of point in paying such a premium in flops/watt by running on roadrunner since the new one is already online.
* 2.3MW is only what the computer itself uses. There's a ton of other supporting equipment needed to use it (chillers, disk, control nodes, etc.)
* This computer isn't being manufactured anymore, so maintenance is going to cost a fortune. When they bought it, they either paid for a hardware support contract or bought a ton of spares and did self-maintenance. If they did the former, IBM is either charging through the nose for replacements or refusing to support it outright. If they did the latter, they're probably running low on spares.
The tri-labs have a rotating schedule for acquiring new machines, that include capacity and capability classes (the later being RR). The last time I was in the computing center at LANL the computing floor was quite packed. So, regardless of the claims made in the article for disposal reasons, there are other issues such as total machine weight, cooling requirements (which dominate), making room for new machines, etc...
I do not know if RR will be physically shredded, but I'm hoping that it is retired to the brand-new PRObE (http://nmc-probe.org/) computing center where RR would be available for use by academic researchers.
PRObE will probably get a chunk of it, but they also have power limitations. Basically every piece of the machine has already been given to some researcher or another for some purpose -- there's a bit of a waiting list at the moment.
The hard drives will probably be sent to a military base to be melted down, I don't think they risk shredding them nowadays. The hardware that doesn't store data will probably be kept intact for quite some time.
Since Roadrunner takes up about 1/3 of the Metropolis datacenter, getting rid of it leaves some room for the 2015 "Trinity" machine (assuming LANL hosts it).
There's one thing you're forgetting here: Cell architecture. Noone writes software for that anymore, since it's been discontinued (either officially or inofficially). Most new supercomputers at that scale are powered by a mix of NVIDIA Tesla (with Intel MIC now coming as well) and Opterons/Xeons. This dictates the software stack of present and future research[1]. There's no reason to keep a computer running when it's severely underused.
[1] Plug: That's also what I'm working on currently: https://github.com/muellermichel/Hybrid-Fortran
Interesting project. Do you have a blog?
Thanks. Not yet, but soon at http://typhooncomputing.com.
The cost of building the thing has already been paid. It is a sunk cost. If it costs $352/hour to run the roadrunner and they can only get $150/hour in income to operate it, it don't matter how much it cost to build it.
If it costs $352/hour to run the roadrunner and they can only get $150/hour in income to operate it, it don't matter how much it cost to build it.
Assuming they already paid for the replacement super-computer too etc. Otherwise, sure it matters. Instead of finding $xx-$xxx million to build a more efficient one, you might be better off to pay the electricity bill difference. This is still in the top 25 computers in the world, so it can be used for a lot of things, but we don't know all the details.
even then, I'd rent a day's worth of time for a prompt with a petaflop processor :)
It's a distributed batch-job type system, not a single computer with a petaflop core. I don't think it is technically accurate to call it a cluster, but it isn't one machine either.
You'd still be able to mine bitcoins on it.
Unless you've got a big HPC job to run, it wouldn't be very exciting. You log in to a "login node" which is just an ordinary linux server and you get a bash prompt. Nothing very remarkable. To actually do anything you would create a script and submit your job to the workload manager, then come back later for your results.
If there were a GUI for that, it could show decks of FORTRAN cards and a shelf with printouts... ;-)
The 2.5 megawatts Road Runner consumes are not available for other purposes, such as running more efficient computers. In other words there is an opportunity cost associated with a power budget.
Retiring Road Runner may be significantly cheaper than adding additional megawatts of capacity, particularly when considering redundant power services and backup power supplies.
This is the "sunk cost fallacy" -- see http://en.wikipedia.org/wiki/Sunk_costs#Loss_aversion_and_th...
If it's cheaper to build (or rent) and run a new supercomputer, then it makes sense to dismantle the old one.
They probably need the space for a faster one...
A 22,000 square foot data center costs about $5mm to build [1].
And these data centers can be located wherever is convenient. And clearly, forward planning is not an issue when we are talking $100mm+ investments.
[1] http://www.reedconstructiondata.com/rsmeans/models/data-cent...
And how much to build it to mil. spec? Probably x10 that number at a minimum. Sure, it's not a military establishment, but I am sure their security and 50 year event survivability is higher than any commercial DC facility.
Add to that the fact that utilization in most DCs is terrible. A super computer will throw out significantly more heat than your "traditional" DC.
Let's not even consider the op. ex. of moving the staff working who would work on the new facilities.
Edit: And LOL at those figures being accurate. That's just the building. What about the costs for the fibre, all that copper, power infrastructure, cooling, false floors, air filtering, special construction needs such as sunken well floors (to contain any leaks from the cooling systems, for example).
You'd be surprised how many locations are inconvenient. You need to find the right nexus of preexisting fiber trenches (digging new ones is super expensive!), cheap power, ideally some environmental factor to make cooling cheap (eg, a river nearby), and acceptable local zoning and laws.
The $120m is sunk costs from a previous building budget, it's not relevant. The more relevant piece is more recent supercomputer (with architectures which are not dead ends) being 4 times as power efficient.
Actually if you compare the Roadrunner against what it would cost to get the equivalent horsepower on Amazon (hypothetically, of course), that price tag is quite a deal.
An economic decision only takes into account marginal expenses (and income for that matter).
At this point the 120M investment is a sunk cost. And the investment was so bad that the operational income can not even cover just the operational expenses, so shutting it down is the wise thing to do.
They already have the Titan, four times more efficient, with 17 pflop consuming only 8000 kwh.
Titan is for open science, not classified research as roadrunner was. Sequoia is the new classified nuclear supercomputer.
But at what price amortized over the same length of time?