make the difference between successful and unsuccessful projects. Huggingface has nice (in a certain sense) tools for training and doing inference on small LLMs but is a train wreck when it comes to model selection and preprocessing. (To be fair a few years back I tried developing a general purpose trainer that worked for bigger models that scikit-learn would handle but did the model selection and preprocessing well and didn't like the answer I got)
Comments
You might want to learn to find your way around
https://scikit-learn.org/stable/
the algorithms it uses are not the most fashionable but the stuff in
https://scikit-learn.org/stable/model_selection.html
and
https://scikit-learn.org/stable/modules/preprocessing.html
make the difference between successful and unsuccessful projects. Huggingface has nice (in a certain sense) tools for training and doing inference on small LLMs but is a train wreck when it comes to model selection and preprocessing. (To be fair a few years back I tried developing a general purpose trainer that worked for bigger models that scikit-learn would handle but did the model selection and preprocessing well and didn't like the answer I got)