And as I was trying to emphasise in my post, there's not a "right way" to do things, some problems just make some approaches more right than others in those instances.
Just with Bentley's Programming Pearls book, he underlines again and again that knowing your problem is more important than knowing the best algorithms, the one that will work for you depends on your problem and only you know that.
Highscalability.com and shared slidedecks act as a community generated set of architectural patterns, but no-one should implement them without knowing what their problems will be.
What I don't understand - and perhaps will be incapable of understanding until I face these issues myself (and hopefully I will in the future) - is why this view that scaling is in a sense un-documentable...
What is it about scaling an application that can't be reduced to an algorithm - at some level of abstraction at least, so newbies can at least get an idea of how they should start thinking about it.
To put the question in another way - could a framework like django ever come to provide scaling tools out of the box? Or is it just something that fundamentally can't be reduced. Might it be that there just haven't been enough people who have faced scaling problems that repeatable patterns haven't yet become obvious?
It's undocumentable because as strange as it seems, no two problems are the same. There is no one-size-fits-all approach, not even a one-size-fits-many. At best we have a series of one-size-may-fit-you-if-you're-lucky options. Disqus isn't using NoSQL or Nginx, something a good number of scaled web applications have switched to. Why? It doesn't solve their problem. Why is their problem different from others? That's a long and complex answer that revolves around almost every aspect of how their applications run, access data, what types of data they're accessing, and so on, and so forth.
Is the problem their algorithm? Does it spend a long amount of CPU time working away? Could it be written a different way?
Is it data access, is the lag caused by queries taking to long?
Is that down to badly formatted queries, inefficient schema, server problems or something else?
Is it even a single problem or a combination of a multitude of minor little niggles that combine into a big headache?
Do you do thousands of little queries, or smaller numbers of big ones? Are your tables narrow or wide? What size and types of data do you store in the fields?
A number of the things that Disqus have done to scale out aren't appropriate for other environments, by nature of the fundamentals of the app. All they can advise on is how to scale your python/Django/MySQL based commenting system, but even then your approach to writing one might be different to theirs.
Quite simply, no one can tell you how to scale your application and its infrastructure, because every application and infrastructure is unique by the very nature of every problem being unique, and every solution more so.
That's not to say there is no value in the information that Disqus has provided. Quite the contrary, there is every bit of value there, and I greatly appreciate them posting it. There is a good chance that whilst some of what they've done won't be of use to you, some of it may be. It may even be of use for other reasons that are entirely different from those that benefit Disqus.
Quick example. You want to load balance web traffic, what do you chose for it? Is software of hardware best? Do you do lots of SSL (would an SSL Accelerator be of use?) Do you want the servers to directly respond to the client, or respond through the load balancers?
You want to add in caching:
Varnish
Apache Traffic Server
Squid
Polipo
and so on! Each software package has its own particular strengths and weaknesses, and its as more a matter of gut instinct and intimate knowledge of the way your code and site works, than anything else that can help you find the right way to scale.
Comments
And as I was trying to emphasise in my post, there's not a "right way" to do things, some problems just make some approaches more right than others in those instances.
Just with Bentley's Programming Pearls book, he underlines again and again that knowing your problem is more important than knowing the best algorithms, the one that will work for you depends on your problem and only you know that.
Highscalability.com and shared slidedecks act as a community generated set of architectural patterns, but no-one should implement them without knowing what their problems will be.
What I don't understand - and perhaps will be incapable of understanding until I face these issues myself (and hopefully I will in the future) - is why this view that scaling is in a sense un-documentable...
What is it about scaling an application that can't be reduced to an algorithm - at some level of abstraction at least, so newbies can at least get an idea of how they should start thinking about it.
To put the question in another way - could a framework like django ever come to provide scaling tools out of the box? Or is it just something that fundamentally can't be reduced. Might it be that there just haven't been enough people who have faced scaling problems that repeatable patterns haven't yet become obvious?
It's undocumentable because as strange as it seems, no two problems are the same. There is no one-size-fits-all approach, not even a one-size-fits-many. At best we have a series of one-size-may-fit-you-if-you're-lucky options. Disqus isn't using NoSQL or Nginx, something a good number of scaled web applications have switched to. Why? It doesn't solve their problem. Why is their problem different from others? That's a long and complex answer that revolves around almost every aspect of how their applications run, access data, what types of data they're accessing, and so on, and so forth.
Is the problem their algorithm? Does it spend a long amount of CPU time working away? Could it be written a different way? Is it data access, is the lag caused by queries taking to long? Is that down to badly formatted queries, inefficient schema, server problems or something else? Is it even a single problem or a combination of a multitude of minor little niggles that combine into a big headache? Do you do thousands of little queries, or smaller numbers of big ones? Are your tables narrow or wide? What size and types of data do you store in the fields?
A number of the things that Disqus have done to scale out aren't appropriate for other environments, by nature of the fundamentals of the app. All they can advise on is how to scale your python/Django/MySQL based commenting system, but even then your approach to writing one might be different to theirs.
Quite simply, no one can tell you how to scale your application and its infrastructure, because every application and infrastructure is unique by the very nature of every problem being unique, and every solution more so.
That's not to say there is no value in the information that Disqus has provided. Quite the contrary, there is every bit of value there, and I greatly appreciate them posting it. There is a good chance that whilst some of what they've done won't be of use to you, some of it may be. It may even be of use for other reasons that are entirely different from those that benefit Disqus.
Quick example. You want to load balance web traffic, what do you chose for it? Is software of hardware best? Do you do lots of SSL (would an SSL Accelerator be of use?) Do you want the servers to directly respond to the client, or respond through the load balancers?
Apache httpd+mod_proxy Nginx HAProxy ldirectord lighttpd
You want to add in caching: Varnish Apache Traffic Server Squid Polipo
and so on! Each software package has its own particular strengths and weaknesses, and its as more a matter of gut instinct and intimate knowledge of the way your code and site works, than anything else that can help you find the right way to scale.
Thanks for that in depth reply... I get ya in the abstract... but yeah I guess I have to go through it myself to really see it.