October 16, 2013 by Sean Childers in Livewire, News with 6 comments
Today, I co-authored a technical piece that looked at how well objective indicators like USA Ultimate regular season rankings or RRI ranks predicted team performance at Nationals. As I was writing that piece, I found myself wondering what the implications for the bid allocation at Nationals should be.
I think that discussion has to begin in the context that the current allocation system is knowingly “broken.” There is an incentive for an end of season rush scheduling of in-region games that magically produce the right score lines. We know the ranking system doesn’t work especially well in a small sample size. Nationals-worthy teams can have their bids sniped by veteran teams formed just for the series.
But I put the word broken in quotations for a reason: The system is only broken if you are looking for a perfect way to ensure that teams ranked 12th-16th do in fact qualify for Nationals. The numbers clearly support concluding that the objective rankings have predictive value, even if it is not perfectly predictive. This, combined with automatic bids, makes it extremely unlikely that the current bid system would principally cause a top 10 team to fail to qualify for Nationals, in any division.
Even if you wanted to incorporate a subjective ranking to select the final few bids or teams for Nationals, who could do it? Other sports, like college football and basketball, involve media polls and selection committees. But these are controversial as well. Ultiworld Editor Charlie Eisenhood has probably seen enough of the top teams to do a decent job – but maybe only in Open, certainly not in Mixed. Plus, one person doesn’t equal a poll or selection committee. There simply isn’t enough of a media presence in Ultimate yet to move strongly in this direction.
Trying to incorporate high-level teams in a polling process would unfortunately incentivize the same self-serving biases and gaming that the current allocation system faces. Crowdsourcing the selection to the Internet writ large could turn bids into a popularity contest.
So while the current system of computer ranking may not be the most “predictive”, it may be superior in other very important elements that matter to bid allocation. It’s announced well in advance, it is predictable, and there is no incentive for a computer to favor or be biased for any team or set of teams. The bigger concern I have is when important decisions are made on the thinnest of margins. Perhaps that is inevitable in any ranking system, but it seems especially suspect when we know that the computer rankings aren’t so great to begin with.
