Nice writeup.
F1, balanced accuracy, etc. In truth it depends on your problem and what a practical "best" solution is, especially in imbalanced scenarios, but Matthews Correlation Coefficient (MCC) is probably the best comprehensive and balanced blind go-to metric, because it guarantees that more portions of the confusion matrix are good [0,1].
I made a quick interactive, graphical exploration to demonstrate this in python [2].
Really neat visualization! And thanks for the tip on MCC.
Out of curiosity I plugged it to the same visualization (performance vs. class weight when optimized with BCE) and it behaves similar to F1, i.e. best without weighting.
Comments
Nice writeup. F1, balanced accuracy, etc. In truth it depends on your problem and what a practical "best" solution is, especially in imbalanced scenarios, but Matthews Correlation Coefficient (MCC) is probably the best comprehensive and balanced blind go-to metric, because it guarantees that more portions of the confusion matrix are good [0,1].
I made a quick interactive, graphical exploration to demonstrate this in python [2].
[0]: https://biodatamining.biomedcentral.com/articles/10.1186/s13...
[1]: https://biodatamining.biomedcentral.com/articles/10.1186/s13...
[2]: https://www.glidergrid.xyz/post-archive/understanding-the-ro...
MCC also generalizes to multi-class well. I wish it had a better name though. It seems like F1 score has better marketing
Really neat visualization! And thanks for the tip on MCC.
Out of curiosity I plugged it to the same visualization (performance vs. class weight when optimized with BCE) and it behaves similar to F1, i.e. best without weighting.