Yes, if you want to participate in rooms with >10K users or >500 servers you need quite a large box (several GB of RAM) - although over the last few weeks we had several massive algorithmic performance breakthroughs which should help this a lot. these are currently being tested and implemented in Synapse (the python impl).