Some reactions to the algorithm and Club Regionals bid allocation.
August 27, 2015 by Charlie Eisenhood in Analysis, Opinion with 30 comments
Another final set of USA Ultimate rankings, another gnashing of teeth. It’s like clockwork — every April and August, the bid allocation from Regionals to Nationals is determined for the College and Club Divisions, respectively. And that is invariably followed by outrage at perceived failures of the rankings algorithm.
Let’s take a look at this year’s controversies as we prepare for the start of the postseason this weekend at Sectional tournaments around the country.
Did the Santa Barbara Condors get a raw deal when they lost their bid to Kansas City Prairie Fire?
As you may have seen in our news piece, the Condors missed out on a second bid for the Southwest by a razor-thin 0.34 points to Prairie Fire, who earned a third (!) bid for the North Central.
The Condors finished the regular season with a 2-0 record v. Prairie Fire, which left many in Santa Barbara understandably upset about essentially missing a shot at Nationals (#1 ranked Revolver is expected to again dominate the Southwest) by less than a single point in the rankings.
Indeed, the cold nature of the algorithm makes it a tough pill to swallow. Still, the Condors lost two games this season to teams outside the top 16. They played some close games against mediocre competition. They traveled to the Select Flight Invite with a thin roster.
At this point, teams know what it takes to earn a bid. You can’t expect to make the cut when you drop games to average teams or take blowout losses to good ones. The Condors didn’t exactly knock it out of the park during the regular season.
Did Toronto GOAT get a raw deal by missing the bid cutoff?
Short answer: no.
While GOAT, a top four team from 2014, played a significantly tougher schedule than some teams that did ultimately earn bids, it’s hard to fathom GOAT’s argument for a top spot.
They sent a gutted roster to this past weekend’s Pro Flight Finale and were summarily stomped by much of the competition. While their rationale for doing so might be completely understandable (they had just wrapped up a long push to win gold at a Worlds-qualifying Canadian Championships), they also knew the risk they faced.
Imagine an NBA team complaining that they missed the playoffs because they sat their starters for the final 20 games of the regular season but arguing that their full-strength team totally deserves to be in. Crazy!
Are there problems with the rankings and algorithm?
There are certainly plenty of legitimate critiques. It is reasonable to argue, for example, that GOAT would have earned a bid, even with a short rostered team, had they played at lower level tournaments featuring worse teams that they could have dominated with their bench players on the field.
That opens up a whole new set of issues about the structure of the Triple Crown Tour. Because teams are tiered into flights and have limited crossover opportunities, rankings tend to pool within tiers as well since there is a lack of direct connectivity between teams.
An Ultiworld analysis earlier this year showed that the current algorithm does reward playing a relatively weaker schedule compared to your own team’s strength. That can mean that Nationals bubble teams that don’t face off much against Pro and Elite Flight teams (instead facing off against Select and Classic Flight teams) could have a scheduling advantage over stronger teams.
It is telling, perhaps, that three Pro Flight teams (in Men’s: GOAT and Atlanta Chain Lightning; in Mixed: San Francisco Blackbird) missed the bid cutoff this season.
There have been plenty of suggested improvements, including using probabilistic rankings, regressing to the mean, requiring more games (which increases connectivity and algorithm accuracy), and using a human ranking component.
Isn’t all this drama exciting?
Well, yes! The reality is that any system that you use to assign bids to Nationals is going to leave one or two teams feeling like they got robbed. It’s easy to imagine, for example, a selection committee giving a third bid to the Northeast by excusing GOAT’s performance at the windy Pro Flight Finale and recognizing that they were extremely competitive with the other elite teams at the US Open. But, then, of course, Prairie Fire and the Condors would be outraged.
There is always going to be a bubble team that doesn’t get in. There will always be a snub.
While incremental improvements to the algorithm and ranking system are certainly possible and should be strongly encouraged, there is no ‘quick fix’ that will give perfect answers.
Perhaps a possible solution lies in a different setup for the regular season and postseason…more on that in a future article.
