I think providers are making a mistake in simplifying prices at all costs, hiding the quadratic nature of attention. People can understand the pricing anyway, even if more complex, by having a tool that let them select a prompt and a reply length and see the cost, or fancy 3D graphs that capture the cost surface of different cases. People would start sending smaller prompts and less context when less is enough, and what they pay would be more related to the amount of GPU/TPU/... power they use.
Comments
I think providers are making a mistake in simplifying prices at all costs, hiding the quadratic nature of attention. People can understand the pricing anyway, even if more complex, by having a tool that let them select a prompt and a reply length and see the cost, or fancy 3D graphs that capture the cost surface of different cases. People would start sending smaller prompts and less context when less is enough, and what they pay would be more related to the amount of GPU/TPU/... power they use.