The OP message could have better said that 50TB databases are common these days when single metal or 24xl I3en or I4* instance on AWS can hold 60T raw.
it's more than big enough to cause big problems / risk days of downtime to change, yea. 50GB is not big. 50TB is at least touching big - you can do it on one physical machine if needed, but it's the sort of scale that benefits from bigger-system architecture. 50PB would be world-class big, hitting exciting new problems every time they do something.
Comments
50TB is big. Bigger is possible I'm sure, but I'd guess 99.something% of all PG databases are less than 50TB.
If someone here commented they had a 2PB database, I guarantee someone else here would be like "pfft, that's not big"...
The OP message could have better said that 50TB databases are common these days when single metal or 24xl I3en or I4* instance on AWS can hold 60T raw.
it's more than big enough to cause big problems / risk days of downtime to change, yea. 50GB is not big. 50TB is at least touching big - you can do it on one physical machine if needed, but it's the sort of scale that benefits from bigger-system architecture. 50PB would be world-class big, hitting exciting new problems every time they do something.
With 50TB, and if you were doing a full text search, wouldn't the entirety of the index have to be held in memory?
No. Full-text indexes exist.
You can also do an incremental/streaming search. Lots of ways to avoid loading it all into memory at once, yeah.