What we've learnt from trueskill #2: the curse of the GB triallist
Published on: May 5, 2018
One of the more amusing quirks that our ranking system has thrown up is what we call "the curse of the GB triallist". This is a bias where people who are actually good, significantly above average rowers, receive very poor rankings due to GB trials making up a large proportion of their race history. The reason it's a curse is because that person would have a better skill score in all likelihood if they hadn't turned up to trials in the first place.
What's going on here? Well, at the time of writing (end of April 2018) our richest data source on elite-level GB rowing data is the results of the various GB trials assessments. These happen 3-4 times a year and produce multiple results in men's and women's sculling and sweep rowing. We still lack a significant amount of Club-level data, although we now have 2018 WEHORR and a large back history of Marlow and Met results in the pipeline.
Of course, if you turn up to these trials, and don't actually do very well (e.g. come bottom of the pairs time trial), the computer simply thinks you are bad at rowing as you seem to be losing a lot of races. While the computer does adjust for the fact that you have lost to very good rowers, if GB trials make up a lot of your early results, this fixes the computers 'mind' against you.
What the computer fails to realise, of course - and unlike with most competitions, is that merely by being attending GB trials you are displaying a level of skill. Either you have been invited by the GB selectors (based on their assessment of your level of skill) or your club has nominated you as one of their top rowers. You may not be world-class (or else you would be winning trials) but certainly you are above-average and shouldn't be below the default score (i.e. with a skill level below 100).
Referencing our earlier article, this could be considered a form of 'bubble'.
So far, so amusing, but this isn't ideal, to say the least. You are probably wondering what's the solution. As we see it, the only solution is more data - we need to add more club-level data that should show the lower trialists beating 'average' rowers.
In other words, while we are having a quiet (if slightly embarrassed) chuckle at the sight of Henley champions and boat race winners having lower scores than random novices entered at Fours Head (we have specific examples but these may disappear as more data is added), this problem should sort itself out as more data is entered into our database.
But if Plan A - add more data - doesn't work, we could look at creating some sort of skill bonus for being entered at GB trials, like we do for international races. But we're reluctant to do so as we want the race results to speak for themselves with as little manipulation from us as possible.
Another change which would mitigate against this, which we are mulling over, is setting a minimum floor for a rower's score. So instead of starting at 100, and your score floating up or down based on successes, your score would start at 0 and only float up if you won races. If you didn't win races, your score would stay at 0 and couldn't go below this. This is more in line with classic attempts at assigning rowing points.