it’s much less clear cut what the long-term limitations are.

Looking back on BERT I can name a handful of dealbreakers that haven't changed and have no solution on the horizon.