Firstly, congrats on the launch! Active learning is a super interesting space.
You say it's possible to download the data and use Humanloop for annotation only while still benefitting from active learning. I'm curious about your experience with how much active learning depends on the model. Are the examples that the online model selects for labelling generally also the most useful ones for a different model trained offline?
Cheers. It's a good thing to be wary of. Poor use of active learning will end up biasing the data according to the model it's trained on – so that data won't be the best X samples to train on a different model. Most of this issue comes from bad active learning selection methods. If you have well calibrated uncertainty estimates and sample for diversity and representiveness too, it's far less of a concern.
Comments
Firstly, congrats on the launch! Active learning is a super interesting space.
You say it's possible to download the data and use Humanloop for annotation only while still benefitting from active learning. I'm curious about your experience with how much active learning depends on the model. Are the examples that the online model selects for labelling generally also the most useful ones for a different model trained offline?
Cheers. It's a good thing to be wary of. Poor use of active learning will end up biasing the data according to the model it's trained on – so that data won't be the best X samples to train on a different model. Most of this issue comes from bad active learning selection methods. If you have well calibrated uncertainty estimates and sample for diversity and representiveness too, it's far less of a concern.