August 28, 2015 by Sean Childers in Livewire, Video with 11 comments
Reddit has an extensive discussion thread on the most recent Club rankings, and the top-voted reply chain centers on a popular proposed adjustment to the USAU ranking system: a “probabilistic” approach to determining the bids.
We’ve described the probabilistic approach before — it stems from an old RSD post :
When implemented, the probabilistic ranking would allocate bids based on “how likely” it is that there is a team within each region above the bid cutoff. A simplified example: consider a bid cutoff of 20, with a New England team ranked #20, a Southwest team ranked #21, a South Central team ranked #22, and another Southwest team ranked #23. It is more likely than not that the NE team is better than the SC team. And in isolation, it is more likely than not that the NE team is better than each of the SW teams. But the odds that the NE team is better than both of the SW teams is less than 50%.
One of the main advantages of the probabilistic system is that it would greatly reduce incentives for teams to “throw” intra-conference matchups.
Mitch Dengler, one of USAU’s best-known observers, weighed in with potential downsides of the probabilistic approach:
[The probabilistic system would have teams] get punished because other teams in your region aren’t good. If the region has the #1 and #15 team plus #50, 60, and #80, while another region has the #2, #16, #20, #25, then a bunch of lower teams, the #16, #20 and #25 get a huge advantage over the #15 team. Even if the #15 beats the #16, #20, and #25 teams, the odds of getting the bid are slim because other teams in the region aren’t good. The second team in a region shouldn’t be punished because the third team isn’t good, they should be rewarded for playing well themselves.
I totally get the problem that the algorithm isn’t accurate. Work on fixing that (more connectivity, more data, better math, etc) rather than just giving up and saying “you have to get the other teams in your region better rather than just playing better yourself.”
Kyle Weisbrod, an Ultiworld columnist and former UPA official, countered:
First, I think this is the most reasonable objection I’ve heard to the probabilistic bid concept.
That said, I don’t think it holds water. We know that the rankings are completely accurate and that’s what the point of the secondary level of prababilities accounts for. If this is truly a concern, we can adjust the sd [standard deviation] on the model to give greater weight to finishing with a high ranking . . . Your example leads readers to overweight those three games at the expense of the rest of the season’s games.
15 isn’t doing everything they need to on the field under this model. To do that, they would need to make clear that they are better and that means getting themselves a high probability score of being in the top 16.
My lay interpretation of Dengler’s argument (to the extent it’s necessary, as Dengler is pretty clear!) is that, for teams near the bid cut-off, you shouldn’t have to depend on other teams in your region in order to win a bid. If you are ranked #16 in the country, then we say that you — not any other team — is most likely to be the 16th best team in the country. You should get that bid. Introducing a probabilistic alteration would rob that team of their bid if the next-best team in the region was especially weak. And why should the #16th ranked team be penalized for having weak neighbors?
In a later reply, Dengler writes, “If I’m team A, and [I beat team B by >6 points, achieving the maximum ratings boost a single game win can give me], I’ve done my job on the field. I shouldn’t have to worry about not only beating B by the max margin, but also making sure team B is good.”
While Dengler makes a good point, I’m more inclined towards Weisbrod’s position. If the goal of the rankings is to ensure the top teams at Nationals, then I think you need to confront the inherent uncertainty of the rankings. A probablistic adjustment would more closely answer the question of which regions are most likely to have one (or more) of the top 16 teams? If Region A has the #16 team, but Region B has the #17 and #18 team, it is more likely that Region B has the #16th best team in the country given that combined probability.
I also think that, while Dengler’s critique is fair, it might overstate the likelihood of teams dropping out of the top 16. Without any real data to back up this total hunch, I suspect that it would be incredibly rare for any teams in the top #14 to lose a bid — they would almost always have more than a 50% chance of being a top 16 team. Even the #16 ranked team would, all things being equal, be favored to retain that last bid.
One thing under-discussed by the probabilistic advocates: team’s intentionally creating uncertain forecasts of themselves. When you play fewer games, the algorithm is less confident about your actual “true” rating. So, if you think the goal of the rankings system is to “reward” teams for playing competitive schedules and earning bids, I can see how you might be concerned that a probabilistic approach could incentivize intentional uncertainty (and few games played).
There may always be room at the edges to slightly improve these things: my reply on Reddit suggested limiting the number of teams that “qualify” for the probabilistic algorithm (trying to ensure it is more likely that your regional rivals can pull you up to a bid, but weak regional rivals can’t pull you out), and Jim Parinella has valuable contributions as well — including a comment on Ultiworld about how winning a tournament proves too little of a benefit.
