Perhaps you could simplify the terrain data at lower zoom? That's what we've done with route style geocorridor POIs to avoid thousands of points.

https://postgis.net/docs/ST_Simplify.html