Get extensive code out of notebooks into testable libraries. Confirm it works before running a big computation.
We try to keep our notebooks slim and factor anything complex into a Python library. We can experiment with this libraries implementation, write unit tests on smaller data sets to confirm it works, before doing a bigger experiment. We can also deploy the library to prod and ensure users get the exact same thing we experimented with.
(There’s also the alternative nbdev that creates a full software engineering and experimentation setup in 100% notebooks. Not something I particularly like, but it might be another option.)
Comments
Get extensive code out of notebooks into testable libraries. Confirm it works before running a big computation.
We try to keep our notebooks slim and factor anything complex into a Python library. We can experiment with this libraries implementation, write unit tests on smaller data sets to confirm it works, before doing a bigger experiment. We can also deploy the library to prod and ensure users get the exact same thing we experimented with.
I would recommend the talk “I don’t like notebooks” https://m.youtube.com/watch?v=7jiPeIFXb6U
(There’s also the alternative nbdev that creates a full software engineering and experimentation setup in 100% notebooks. Not something I particularly like, but it might be another option.)