At least for python, I typically just add another RUN statement instead of changing a file used in a layer up the layer stack. That way the change is fast.

Then when I need to commit, I'll update the requirements file or whatever would cause all the layers to rebuild. And CI can rebuild the whole thing while I move to something else.

It is a bit of a pain, but the other benefits of containers are probably worth the trade off.