I'm sure there are enough skilled C hackers out there; but for them to bring in a new face to re-architect the system now (which is to say, after all the scalability problems have started) seems risky.
i am really surprised how facebook, so quikly found the skillset to build a 10,000 server site and how come, twitter didnt! just why ...
its the skillset, but again why was facebook successful in aquiring the skillset when twitter wasnt! ... why is twitter considering a rewrite and facebook didnt ... they just grew !!
According to various scalability articles on FaceBook, they continually rewrite their software. It's probably been rewritten far more than Twitter has, you just never hear about it because TechCrunch doesn't have a source inside.
In my experience, that's really the only way to do software. It's completely unrealistic to think you'll get it right the first time, and yet completely destructive to go and start from scratch. So you pick your worst problem, rewrite it, then hook it back into the working system without touching the rest. Then pick your next-worst problem, rewrite it, etc.
For me, Facebook often seems very sluggish and slow - I have a pretty useful 4Meg connection and others sites are fine - maybe 10,000 servers is part of the problem - it seems like a ridiculously high number. I wonder how many transactions/pasge views per second they get ...
The bottleneck/stress-point is almost certainly the unique IO patterns and latency expectations. I doubt CPU is the issue or that C is necessary; instead they need to figure out the right fanning/batching algorithms, and buy gobs of RAM and use it efficiently.
With their recent fundraising, they might even be able to run a couple parallel fixup projects in competition, and let the best win.
Comments
Twitter always struck me as a little lower on the complexity scale and a little higher on the throughput scale than where RoR's sweet spot lies.
Since their requirements seem like they're pretty static now, I would think writing a backend in C wouldn't be out of the question.
The question is, do they have the skillset in-house for that sort of undertaking?
Even if they don't, how difficult will it be for them to bring in a great hacker who can?
I'm sure there are enough skilled C hackers out there; but for them to bring in a new face to re-architect the system now (which is to say, after all the scalability problems have started) seems risky.
For them to not do so is riskier.
skillset is a really good word
i am really surprised how facebook, so quikly found the skillset to build a 10,000 server site and how come, twitter didnt! just why ...
its the skillset, but again why was facebook successful in aquiring the skillset when twitter wasnt! ... why is twitter considering a rewrite and facebook didnt ... they just grew !!
According to various scalability articles on FaceBook, they continually rewrite their software. It's probably been rewritten far more than Twitter has, you just never hear about it because TechCrunch doesn't have a source inside.
In my experience, that's really the only way to do software. It's completely unrealistic to think you'll get it right the first time, and yet completely destructive to go and start from scratch. So you pick your worst problem, rewrite it, then hook it back into the working system without touching the rest. Then pick your next-worst problem, rewrite it, etc.
You've got a point regarding the inside scoop however, Facebook doesn't suffer a global outage on an almost weekly basis either.
Having 10,000 servers doesn't say 'skillful' to me.
Facebook's system is also 1000x times more complicated than twitter's simple message passing system!
For me, Facebook often seems very sluggish and slow - I have a pretty useful 4Meg connection and others sites are fine - maybe 10,000 servers is part of the problem - it seems like a ridiculously high number. I wonder how many transactions/pasge views per second they get ...
Employing people with the skillset to build 10,000 servers and having the money to do so are two separate things.
Twitter has millions of dollars.
The bottleneck/stress-point is almost certainly the unique IO patterns and latency expectations. I doubt CPU is the issue or that C is necessary; instead they need to figure out the right fanning/batching algorithms, and buy gobs of RAM and use it efficiently.
With their recent fundraising, they might even be able to run a couple parallel fixup projects in competition, and let the best win.