Disqus is a classic case study when it comes to scalability.
If there is one thing I learned from Disqus it is the power of keeping a lightweight stack. Disqus keep it simple, and prove that all the myths that "Django/SQL/whatever doesn't scale" are obscene.
Even for an app with requests per second in the 5 digit range - they do pretty damn good with the basic Django stack with no more than some small tweaks.
I actually find it astonishing that they're using Apache for this. I have had a lot of problems with Apache behaving weirdly in the past, especially with mod_wsgi.
Yes, I'm aware of that. I've been using mod_wsgi since modpython was the recommended way for deploying with Django. The problem is that when you're running a high number of Django instances (via a large number of daemons) you can get all sorts of problems with Apache itself. Some of these issues have been due to mod_wsgi, and Graham has, in my past experiences, been very responsive about fixes. Other issues are simply due to Apache, and when you're running multiple instances in a memory-restricted environment you're left with a tough configuration job. I've even had a case where thread contention in a low-memory setup caused a complete Apache lock-up.
So, as you can see, for the normal case the recommended setup is fine, but for extreme cases you should use the combination with caution.
YMMV, of course, but if you're pushing your setup to the limit and don't have any options for extra servers, I would heartily recommend uwsgi+nginx or fcgi+lighttpd instead.
Comments
Disqus is a classic case study when it comes to scalability.
If there is one thing I learned from Disqus it is the power of keeping a lightweight stack. Disqus keep it simple, and prove that all the myths that "Django/SQL/whatever doesn't scale" are obscene.
Even for an app with requests per second in the 5 digit range - they do pretty damn good with the basic Django stack with no more than some small tweaks.
That was exactly my thought.
No NoSQL. And they use transactions for write queries.
They use apache (not even nginx!).
Only 25% of their servers are pure (no snapshot) caching servers (not 50% or 75%).
They prefer vertical partitioning over sharding (but they still use sharding).
I understand they are looking at redis for some of their features, but really, their main stack is traditional and proven. Very enlightening.
I actually find it astonishing that they're using Apache for this. I have had a lot of problems with Apache behaving weirdly in the past, especially with mod_wsgi.
You are aware the Apache + mod_wsgi is the recommended way for deploying Django apps, right?
Yes, I'm aware of that. I've been using mod_wsgi since modpython was the recommended way for deploying with Django. The problem is that when you're running a high number of Django instances (via a large number of daemons) you can get all sorts of problems with Apache itself. Some of these issues have been due to mod_wsgi, and Graham has, in my past experiences, been very responsive about fixes. Other issues are simply due to Apache, and when you're running multiple instances in a memory-restricted environment you're left with a tough configuration job. I've even had a case where thread contention in a low-memory setup caused a complete Apache lock-up.
So, as you can see, for the normal case the recommended setup is fine, but for extreme cases you should use the combination with caution.
YMMV, of course, but if you're pushing your setup to the limit and don't have any options for extra servers, I would heartily recommend uwsgi+nginx or fcgi+lighttpd instead.