Determining how best to approach your team's season in order to earn a strength bid in the USA Ultimate rankings is on many college captains' minds heading into the Spring. See a mathematical look at the issue from Scott Schriner.
December 2, 2013 by Scott Schriner in Analysis with 28 comments
Now that the fall college season is winding down, many teams are looking ahead to the spring to plan for the regular season. Unlike competition in the fall, every game played during a sanctioned event in the spring will be recorded and used to calculate team ratings. These ratings are then used to rank all of the teams across the nation and distribute bids to those who finish at the top.
In the chase for Nationals bids, maximizing your ranking becomes an important goal for many teams. While placing at the top of the rankings does not guarantee a trip to Nationals, earning an extra bid for your region certainly increases your chances of a Nationals berth.
Since there is no official schedule in the college division, teams are free to select the tournaments they attend. What decisions go into deciding those schedules? Many teams want to find the best way to maximize their rating. But how do they build their tournament schedules to do this?
Do teams seek out the last remaining bids to upper level tournaments in an effort to test their team and play the toughest opponents, or do they feed off of middle and lower-level tournaments in an effort to achieve the best game results?
One way to answer that question is by considering how each strategy affects your rating. The rankings algorithm awards points based on two things: the opponent you play and the score of the game. By separating each team’s overall rating into ‘opponent’ and ‘game result’ categories, a ‘strength of schedule’ and ‘game performance’ measurement can be used to compare season performances.
Knowing the merits of setting a schedule that matches you up with tough opponents versus setting a schedule that lets you walk away with numerous victories can help teams give themselves the best shot at maximizing their rating.
Below is a graphical representation of how each team in the top 30 of last year’s USAU end-of-regular season-rankings performed on average during individual games last season. The height of each bar represents the expected margin of victory for each team in a game to 15.
As you might expect, almost all of the teams had positive margins of victory. However, one team inside the top 20 – Stanford – actually had a negative average margin of victory.
So how did Stanford end up finishing at #18 and earning an extra bid for the Southwest? The other component of the rankings: strength of schedule. The combined average of opponent ratings can tell you the relative strength of a team’s schedule. In the next chart, you can see that Stanford played one of the toughest schedules of the teams who finished at the top of the rankings.
The rankings algorithm adds the two components (opponent rating and game result) and averages those over the course of the season. When you add the game performance metric and the strength of schedule metric (both of which account for weighted average used in the rankings algorithm) in the same way, you get the final composite rankings.
The chart below shows each team’s final rating, with their game performance highlighted in green.
Let’s look at another team in the top 30. Northern Iowa (#21) had the best game performance of the teams in the top 30. So why did UNI miss out on an additional Nationals bid for the North Central? When you dive a little deeper, you will find that Northern Iowa had a pretty weak strength of schedule – they had the 2nd lowest average opponent rating of all the teams in the top 30.
So if Stanford earned a bid because they played tough opponents (despite poor on-field results), and Northern Iowa missed out on a bid because they didn’t play tough enough opponents (despite great on-field results), the best way to maximize a rating is to play the strongest field you can, and the game results will seem insignificant, right?
Unfortunately, there isn’t a simple answer here. What we do know from the small sample size we have is that there has been more success increasing teams’ rankings by playing stronger opponents than by continuously steamrolling weaker ones.
It’s not always that easy, though. For example, Brown qualified to play the field at Easterns by placing high enough at the Easterns Qualifier. This helped bolster their relatively high strength of schedule. However, Brown had the only other negative game score in the top 30. Just being on the field with those teams wasn’t enough to buoy Brown into a spot for a bid.
Because the overall USAU rankings are calculated by adding the strength of schedule score to the game performance score, marginal increases in either component will lead to marginal increases in your overall ranking. Likewise, a big increase in either component will lead to a big increase in your overall ranking.
So the true method for achieving the highest rating remains the one that’s complicated in practice but simple in theory: play tough opponents AND beat them by the largest ratio. There have been many discussions about flaws and imperfections in the ranking system, but this conclusion actually does align with Ultimate conventional wisdom.
If that piece of advice was too obvious, keep in mind two other strategic considerations. First, given that increases in either your strength of schedule or your game performance score can increase your rankings, which one is actually easier for your team to increase? To increase your game performance score, you will want to win more games — but also win each game by as large of a margin as possible. This might include playing your starters longer than necessary, or loading up your D Line with an O Line handler even if you don’t need that break. Of course that decision carries with it other tradeoffs: you may increase injury risk or stall the development of younger players on your team.
Increasing your strength of schedule component isn’t easy, either, as the elite college tournaments are extremely selective about the teams they take.
But making the right tournament scheduling decision at the margin may be the easiest way to increase your overall ranking. Say you decide to attend Terminus instead of Grandma Jane’s Backyard Bash – your opponent rating will be higher just by going to Terminus than if you went to Grandma’s.
One possibility is that you perform the same at both tournaments; now your overall rating will be higher because of your decision1.
If you perform worse at Terminus, but the boost in your opponent rating is enough to offset the negative effect of your performance rating, your rating will also be higher because of the decision. Yes, you could probably improve your game performance by attending worse tournaments. However, like the third reality above, you would be balancing a change in your opponent rating versus a change in your game performance. My gut is that better selection of tournaments is the easier way to improve your rating, as it is more likely that increased opponent quality with offset worse results than for better game results to offset worse opponents.
A second, and perhaps more important strategic consideration, is that scheduling your regular season in such a fashion that you will end up wiping the floor with lesser opponents may not prepare you for the competition you will face Sunday at Regionals. On the otherhand, if your schedule consistently gives you the chance to face tough opponents, you will have the chance to prepare for those late-Sunday games at Regionals. How you do in those tough regular season games can give you a good indication of your preparation heading into Regionals.
Of course the rankings system rewards teams who set up their schedules to face those tougher opponents by giving them better strength of schedule components in their rating. If they perform well in those games, they will be rewarded with even higher ratings. But even if they don’t perform well, scheduling tougher opponents can still better prepare teams for the competition they will face at Regionals than scheduling weaker opponents.
And, of course, in the unlikely event that you perform better at Terminus than you would have at Backyard Bash, your overall rating will be much higher ↩
