I've done some cost analyses between our AWS and DC infrastructure.
To come up with our on-prem compute costs, we baked in the cost of power, real estate, staff, taxes, network infrastructure, servers (both in-use and in reserve), etc. On the AWS side, we used 3 year RIs and Savings Plan. After all that, there was around a 30% cost advantage on-prem. That's non-trivial, but not as big as one might think.
Outbound networking, however, is ludicrously cheaper on-prem. It's about 85% cheaper on-prem than in AWS. Bandwidth is not expensive outside the public cloud.
In fact, egress volume is the #1 cost driver for us moving a service on-prem or building it there to begin with. Some of the AWS managed services are also very pricey, but nowhere near the egregious markup of egress bandwidth.
Typical cloud bandwidth pricing is known as "roach motel pricing" after the old roach motel pest control slogan of "roaches check in but they never check out." The idea is to make ingress free but egress expensive to make it easy to move all your data in but costly and hard to move it out.
Meh cpus/ram most likely have lighter margins for cloud vendors compared to storage and bandwidth (which is just ridiculous) but they have to keep a lot of spare capacity for scaling so if you don’t scale by much you can do way better on metal even on cpus
not sure about pricing here, but graviton on aws has generally offered more performance at a lower price point, which is likely linked to lower power usage and perhaps lower cost of custom silicon vs intel.
the notion, "the cloud is expensive", ignores the fact that the cloud is not just rented hardware, but staff, facilities, planning, management, etc..
there are businesses where it makes more sense to own hardware and employ your own staff, but if you just want generic compute and storage, you're unlikely to do it as well for less.
also you cannot easily source arm hardware commercially, there is the honeycomb lx2, and its' lead time is months for a single unit. if you want hundreds of nodes, you're gonna use a cloud provider who manufactures their own silicon.
You can (in theory) run Linux straight on Apple Silicon. It's not locked like the iPad and iPhone. The M1 Ultra would be a pretty solid high volume server if you can manage to plug a 10-40gbps LAN dongle into it. I believe USB-C and/or thunderbolt connectors like that exist.
In theory so far because I'm not sure if there are mature installers and such yet.
I've been passively curious about this for a homelab--does linux virtualize on apple silicon today? I was of the impression this didn't work when they announced m1 in 2020, but I could be misremembering.
Absolutely. It worked from day one. There are issues with emulating x86 Linux, but that’s a different story. For ARM Linux — just ordinary qemu works fine.
Comments
Pricing is in line with the cloud: expensive. That's it, they made it: now Arm servers are mainstream.
I always found the CPU pricing of cloud to be relatively reasonable. It's everything else that's expensive, and egress bandwidth just ridiculous.
I've done some cost analyses between our AWS and DC infrastructure.
To come up with our on-prem compute costs, we baked in the cost of power, real estate, staff, taxes, network infrastructure, servers (both in-use and in reserve), etc. On the AWS side, we used 3 year RIs and Savings Plan. After all that, there was around a 30% cost advantage on-prem. That's non-trivial, but not as big as one might think.
Outbound networking, however, is ludicrously cheaper on-prem. It's about 85% cheaper on-prem than in AWS. Bandwidth is not expensive outside the public cloud.
In fact, egress volume is the #1 cost driver for us moving a service on-prem or building it there to begin with. Some of the AWS managed services are also very pricey, but nowhere near the egregious markup of egress bandwidth.
85% cheaper seems little. In my case (collocation) bandwidth is 95% cheaper (i.e AWS is 20x as expensive) than AWS.
A quick question.
Have you also included:
in the above price comparison?Would you have some insights there? Thanks.
Typical cloud bandwidth pricing is known as "roach motel pricing" after the old roach motel pest control slogan of "roaches check in but they never check out." The idea is to make ingress free but egress expensive to make it easy to move all your data in but costly and hard to move it out.
Meh cpus/ram most likely have lighter margins for cloud vendors compared to storage and bandwidth (which is just ridiculous) but they have to keep a lot of spare capacity for scaling so if you don’t scale by much you can do way better on metal even on cpus
I am very ignorant of the current ARM cloud offerings. Is it similar prices but less power usage? Or have they had to ramp up the watts to compete?
not sure about pricing here, but graviton on aws has generally offered more performance at a lower price point, which is likely linked to lower power usage and perhaps lower cost of custom silicon vs intel.
the notion, "the cloud is expensive", ignores the fact that the cloud is not just rented hardware, but staff, facilities, planning, management, etc..
there are businesses where it makes more sense to own hardware and employ your own staff, but if you just want generic compute and storage, you're unlikely to do it as well for less.
also you cannot easily source arm hardware commercially, there is the honeycomb lx2, and its' lead time is months for a single unit. if you want hundreds of nodes, you're gonna use a cloud provider who manufactures their own silicon.
Buy Mac Minis and run virtual linux inside?
You can (in theory) run Linux straight on Apple Silicon. It's not locked like the iPad and iPhone. The M1 Ultra would be a pretty solid high volume server if you can manage to plug a 10-40gbps LAN dongle into it. I believe USB-C and/or thunderbolt connectors like that exist.
In theory so far because I'm not sure if there are mature installers and such yet.
The Asahi Linux installer worked flawlessly on my MacBook Pro M1 Max, though it asks you to do some low level things like repartitioning the drive.
I've been passively curious about this for a homelab--does linux virtualize on apple silicon today? I was of the impression this didn't work when they announced m1 in 2020, but I could be misremembering.
Absolutely. It worked from day one. There are issues with emulating x86 Linux, but that’s a different story. For ARM Linux — just ordinary qemu works fine.
Oh, interesting. That's great to hear!
Not at all. Any organization that runs its datacenters can calculate a TCO.
It's really business 101.
Dell and HP are still nowhere to be seen in the on-prem Arm server market. Is there not enough of a cost advantage for there to be demand?
https://www.hpe.com/us/en/newsroom/press-release/2022/06/hew...