January 22, 2014 by Scott Schriner in Analysis, News with 24 comments
On Friday, USA Ultimate announced changes to the rankings algorithm they use to allocate bids in the College and Club division.
The changes come from an update in the game differential calculation, which is used to award points based on the result of a game. The weighting calculation was also updated.
For the math nerds out there, they’ve finally fixed the concavity issue. For those unfamiliar with the issue — in simple terms — the old system was set up in such a way so as to reward additional points in games that were already blowouts.
The new system makes additional points in close games more important than additional points in blowouts. The difference in points between a 13-12 victory and a 13-11 victory is now greater than the difference in points between a 13-8 victory and a 13-7 victory.
You still earn more points the greater the difference in score, and the maximum number of points you can earn for a single game is still capped at 600. The cap comes into effect when the winning team’s score is one higher than double the losing team’s score.
The following chart that shows the amount of game differential points awarded for a specific losing score in a game to 13. Points under the old system are in blue, while points in the new system are red.
The chart shows the value of the game differential calculation for a given losing score in a game to 13. The winner would have this value added to their rating, while the loser would have it subtracted. For example, X=12 represents a game won 13 to 12. The blue circle represents the old algorithm game differential value of ~180. The red X represents the new algorithm value of 125.
Implications
So what does this all mean? Because the new game differential calculation results in slightly higher values than the old calculation, teams who win games will receive more points in comparison to last year. Teams who lose games will lose more points.
The exception to the statement above comes when games are decided by 1 point. USA Ultimate has determined that every game with a margin of 1 point, regardless of final score, will receive the same score.
How drastic will the increase in points be? Consider the men of Northern Iowa. Last year, they had the largest average game differential of any team in the Open top 30. If you applied the new algorithm to their scores from last year, their game performance rating would increase by about 100 points.
The largest difference in the game differential calculation comes from games with mid-range score margins, such as 13-9 and 13-10.
Weighting
In addition to the concavity issue, USA Ultimate also introduced a change to the weighting system.
In previous years, the weights were calculated using a decay function on the time between the tournament and the end of the season. The equation used resulted in a minimum weight around 0.383.
The new minimum weight is 0.5. This means that early tournaments will have a larger effect on the overall rating for a team than in previous years, because they decay less, putting a more equal weight on tournaments throughout the season.
The weighting system allows for teams who perform poorly during the early part of the regular season to improve throughout the year and end up with a final rating that more closely resembles the body of work they completed towards the end of the year than the beginning of it.
However, teams who perform better at the beginning of the season than at the end of the season will see a final rating that is closer to their weaker performances.
I don’t particularly like the weighting element in the ratings calculation in general, so the fact that the weights are more even now is something I like. The weighting system is there to put more of an emphasis on the games played when teams are supposed to be at their peaks at the end of the season than at the start of the season.
But if you’re going to use the whole regular season to determine bids to Nationals, why weight part of it greater than another part? Teams’ development throughout the season will be captured by their performance in the Series.
Final ratings for teams are still a sum of the opponent’s rating and the game differential points awarded (or subtracted). Since there was no change to this element, the theory that it is more beneficial to play tougher opponents (rather than steamroll weaker ones) still holds true.
Across the board, I expect the upcoming year’s rankings to feature a wider spread – winning teams towards the top of the rankings will have a higher rating, and teams at the bottom of the rankings will have a lower rating.
All things considered, this change is an improvement over the previous algorithm. It lines up with the idea that beating an opponent by a larger score in a close game is worth more than beating an opponent by a larger score in a blowout.
Hopefully incremental changes like this will continue to come from USAU as the Ultimate community continues to place more emphasis on the systems, calculations, and algorithms used to determine bids to Nationals.
