B2 is interesting, and the price for object storage is pretty competitive. I wrote my stuff against S3's API, particularly for DO Spaces, but DO's fairly epic pants-crapping this week has me nervous.
My use case is download-heavy, though, and a little bit bursty (but with a long tail, a CDN doesn't really address my needs); what sort of bandwidth speeds can I expect from downloading B2 when a decent number of clients, let's say 100-500, are hitting a single object concurrently?
Sysadmin at Backblaze here. Unless it's closer to 500+ and they all have 100Mbps+ connections, the clients will definitely be the bottleneck. Even in that case with smallish objects, it's unlikely that the TCP window will ramp up enough to put a dent in things. That number will of course continue going up with time as we expand and improve the infrastructure.
That's pretty awesome. Depending on how my DO Spaces tests go, I very much might try it out; it's a bummer that you guys don't support S3API, but I can deal with that. (I know Minio has a layer for B2, but I'd rather not go that way.)
Yev from Backblaze -> We've found that the majority of folks hit bottlenecks on their ends or latency issues before they hit any performance issues on our end!
They blew up a Ceph cluster yesterday that was serving block storage (and they use Ceph for object storage, too, hence my concern). It was down for over thirty hours.
I’d argue that Backblaze has significantly more experience dealing with large amounts of data, given their large backup service. Digital Ocean has been pretty amatureish for .. basically ever.
That was a strange oversight, sorry! Here is an explanation of our architecture regarding downloads:
Backblaze files are served up from a set of load balanced "download servers". Think of the download servers as "caches". The FIRST time you request a URL, the download server has to go fetch the file from the vault. The file must be reconstructed since it is striped across 20 separate pods (as soon as any 17 of these pods respond, it is enough to reconstruct the file).
The download servers have 10 Gbit networking and a fast SSD. The file is cached on the local download server's fast SSD. So the SECOND time you fetch the URL from one download server it can go really fast.
Ok, so the download servers don't talk to each other and there isn't any IP address affinity. So take the example where we have 20 download servers serving files from the area your one file comes from. If there is a lot of "hot" activity on one file (like 500 threads fetching the same one file in the same minute) then each download server fetches it ONCE from the vault, which means out of the 500 threads, about 20 early threads will get "lower performance" as the vault reconstructs the file, so maybe 2 - 5 Mbits/sec for the first file fetch on each download server? Then 480 threads will be fetching the file directly from the 10 Gbit/sec fast SSD cache servers. If the threads are spread evenly across the 20 download servers, so each download server serves up around 24 threads, which is easy.
Now some disclaimers: Backblaze has a TON of experience storing and writing files into our datacenter. However, we are still learning about serving files OUT of our datacenter. Recently we realized the SSDs in the download servers were a little small and files weren't getting cache hits enough. In other words, we could deploy fewer download servers if we gave them more local SSD space. So we're fixing that.
We also like to listen to customers and work with them. So if a knowledgeable customer experiences slow downs or they see an anomaly we can take a look and figure out how to improve.
Comments
B2 is interesting, and the price for object storage is pretty competitive. I wrote my stuff against S3's API, particularly for DO Spaces, but DO's fairly epic pants-crapping this week has me nervous.
My use case is download-heavy, though, and a little bit bursty (but with a long tail, a CDN doesn't really address my needs); what sort of bandwidth speeds can I expect from downloading B2 when a decent number of clients, let's say 100-500, are hitting a single object concurrently?
Sysadmin at Backblaze here. Unless it's closer to 500+ and they all have 100Mbps+ connections, the clients will definitely be the bottleneck. Even in that case with smallish objects, it's unlikely that the TCP window will ramp up enough to put a dent in things. That number will of course continue going up with time as we expand and improve the infrastructure.
That's pretty awesome. Depending on how my DO Spaces tests go, I very much might try it out; it's a bummer that you guys don't support S3API, but I can deal with that. (I know Minio has a layer for B2, but I'd rather not go that way.)
Anyway, thanks a lot!
Yev from Backblaze -> We've found that the majority of folks hit bottlenecks on their ends or latency issues before they hit any performance issues on our end!
What happened with DO?
They blew up a Ceph cluster yesterday that was serving block storage (and they use Ceph for object storage, too, hence my concern). It was down for over thirty hours.
https://status.digitalocean.com/incidents/8sk3mbgp6jgl
I’d argue that Backblaze has significantly more experience dealing with large amounts of data, given their large backup service. Digital Ocean has been pretty amatureish for .. basically ever.
I agree. I'm not worried about their reliability. Just asking about bandwidth.
Sounds like a question you might be better off asking their sales team.
You realize their team is in this thread, right?
Yep, and they didn't answer. It's a question that sales would be able to answer. No need to downvote me for a legitimate response, right?
That was a strange oversight, sorry! Here is an explanation of our architecture regarding downloads:
Backblaze files are served up from a set of load balanced "download servers". Think of the download servers as "caches". The FIRST time you request a URL, the download server has to go fetch the file from the vault. The file must be reconstructed since it is striped across 20 separate pods (as soon as any 17 of these pods respond, it is enough to reconstruct the file).
The download servers have 10 Gbit networking and a fast SSD. The file is cached on the local download server's fast SSD. So the SECOND time you fetch the URL from one download server it can go really fast.
Ok, so the download servers don't talk to each other and there isn't any IP address affinity. So take the example where we have 20 download servers serving files from the area your one file comes from. If there is a lot of "hot" activity on one file (like 500 threads fetching the same one file in the same minute) then each download server fetches it ONCE from the vault, which means out of the 500 threads, about 20 early threads will get "lower performance" as the vault reconstructs the file, so maybe 2 - 5 Mbits/sec for the first file fetch on each download server? Then 480 threads will be fetching the file directly from the 10 Gbit/sec fast SSD cache servers. If the threads are spread evenly across the 20 download servers, so each download server serves up around 24 threads, which is easy.
Now some disclaimers: Backblaze has a TON of experience storing and writing files into our datacenter. However, we are still learning about serving files OUT of our datacenter. Recently we realized the SSDs in the download servers were a little small and files weren't getting cache hits enough. In other words, we could deploy fewer download servers if we gave them more local SSD space. So we're fixing that.
We also like to listen to customers and work with them. So if a knowledgeable customer experiences slow downs or they see an anomaly we can take a look and figure out how to improve.