In advance of the final ranking announcment, Ultiworld sent USA Ultimate questions about the algorithm and future of bid allocation. USAU spokesperson Andy Lee responded with an in-depth update on the thinking behind their system.
September 4, 2014 by Ultiworld in News with 23 comments
Ultiworld: Are they [the rankings] working from the USAU perspective?
Andy Lee: Yes, especially when taking into account the information that the rankings value, such as W-L, point differential and timing of competition. Generally speaking, the nature of rankings is such that there will always be teams on the outside looking in that believe they deserve to be at the next level. Naturally there will always be some debate and discussion about teams on the margin, as there is in every sport that utilizes a ranking system. However, we believe that our system – from the rankings to the bid allocation to the postseason format – goes to extraordinary lengths to be as fair as possible. At the end of the day the National Championships will feature top quality teams from around the country, Regionals will be highly competitive, and every team competing in the postseason will have their opportunity to advance on the field of play.
As an aside, it’s important to note that with WUCC in the mix this year the natural scheduling of the top Pro Flight teams (with regards to the number and timing of games played and the diversity of opponents) was altered in way that only occurs in those years. We worked diligently to encourage those teams to take advantage of that unique competitive opportunity while minimizing the impact on the Triple Crown Tour’s competitive structure and other teams as much as possible. The tradeoffs in 2014 due to Worlds was at least one fewer regular season event for those Pro Flight teams and having to wait until late in the season to get the level of connectivity that we’ve become accustomed to earlier in the season.
Are they primarily designed to reward in-season TCT performance or to put the best 16 teams on the field at Nationals?
Both, with the disclaimer that geography also plays a role in the composition of nationals and it being truly representative of a “national” event. The system is designed to qualify the best teams for nationals, while including a minimum of one team per geographic region. That qualification is driven by teams’ performance during the regular season of the Triple Crown Tour, which earns them opportunities to advance, and their performance in the Postseason Championship Series, which actually dictates who advances. The rankings aren’t only about the teams that qualify for nationals, they’re relevant to all 600 or so teams that are playing in the regular season and postseason as they’re being used to determine a fair mechanism for teams to advance from sectionals to regionals. Regular season performance informs that, but we’re still utilizing a structure that allows teams that don’t participate in the regular season to enter the postseason. That level of inclusiveness results in some competitive tradeoffs.
Do you think the club scene needs a different algorithm than the college season (where there are more games and more interconnectivity)?
Possibly. It’s something to consider as we continue to evaluate as the two seasons evolve. In most ways the college and TCT seasons are similar (with the exception of an event like WUCC), in that they are able to account for the competitive interaction between the teams in terms of level of play and geography. But we’ll continue to assess where there are important differences and adjust as needed. It’s important to be sure that the algorithm, which is designed to reflect strength based on performance in a mostly unstructured environment, continues to incentivize both playing competitive ultimate at the appropriate level and playing to win.
Would you ever consider adding a subjective component to bid allocation?
Not likely. It’s something that comes up in discussion periodically, but the same arguments continue to persist against the idea. There simply isn’t enough information out there, especially at the margins, to accurately rely on subjective input that is sufficiently informed and unbiased. While the amount of media attention and video coverage is growing, there are relatively few games, even among the top teams, available for subjective input, much less of the teams competing for bids at the margins. And there certainly aren’t any accountable, objective scouts out there watching these teams in person on a regular basis that can provide that kind of information.
Relatedly, in general, should the community expect to see “refinement” of the math/formula going forward, or are you thinking about potential bigger changes?
We’ll continue to refine as long as we maintain such a flexible and inclusive competitive structure. There are so many teams and results that need to be accounted for (500+ teams playing in 50+ events during the TCT Regular Season), and the algorithm does a good job of collecting and processing all of that information. We have a really solid working group of volunteers in the community that work alongside staff to review and refine the formula based on years of experience, input and ideas. They provide recommendations to our Competition Working Group, which makes decisions about competitive guidelines. As always, we welcome people with ideas and expertise to get involved with helping to make improvements along the way, and are always open to discussing ideas, both big and small.
