Do you trust your Leaderboard Score?

Hi @michael_bordeleau thanks for your reply!

I completely agree with you. Shake-up or not, there will be a lot to learn from the winning solutions.

My main doubt is that the composition of the of the test set used for the public leaderboard does not reward well rounded models. The pre_alzheimer is the most difficult class to predict but probably the one that could have the higher impact from a social point of view. I’m focusing indeed on this aspect, but models that have better perforamnce on this class seems to perform worse on the public leaderboard.

How many models are just ignoring this class? If we look at the F1 scores of the top leaderboard they are all below 0.5. This is the main reason why I feel like the current leaderbord scores will not be reflective of the final scores.

View post 1

Do you trust your Leaderboard Score?

Hi all,

Do you trust your Leaderboard Score? This is a simple but fundamental question. I personally have not found a good correlation between my local cv score and the public leaderboard one. To be more precise, in my case I see a good correlation for models that scores in the high 0.6xx low 0.7xx, however when considering my best performing models (low 0.6xx, high 0.5xx on local cv) the correlation seems to be completely broken.

My gut feeling is to trust more my local cv and expect a huge shake-up for the final leaderboard, but I’m interested in your experience. It may very well be the case that I have just not found yet a good validation scheme.

View post 4