August 26, 2013 by Sean Childers in Livewire, Video with 2 comments
I’ve been lucky enough to play with or against Brett Matzuka many times. He’s an extremely talented player; a smart, outside-the-box thinker; and a loyal friend. I’m glad he decided to publish a piece questioning the usefulness of statistics in Ultimate.
Though Brett takes a decidedly skeptical tone towards statistics, I actually find his article to be relatively consistent with the broader “analytic” mode of thinking. It’s a player evaluation movement first popularized by (as Brett identifies) Billy Beane; I would argue it has now evolved into a journalistic movement that I personally (and Ultiworld generally) identifies with.
Like any broad tent or group, analytics isn’t a monolith and there are differences in opinion. But there are also important commonalities. One important commonality that bleeds through both Brett’s work and, I believe, our own stats work on Ultiworld is skepticism.
Brett is extremely skeptical that numbers like D’s or turnovers will give you very useful information on how well a player performed. The major limitation with these numbers, according to him, is that it unjustly credits single individuals for what was successful (often defensive) team play. And it may unfairly punish individual players who carry a large offensive burden for their team – a burden that may include a disproportionate share of the risk of mistakes, but a burden that someone on the team must nonetheless bear.
Our stat system responds to some but not all of these criticisms. For example, the defender yardage system is able to track defenders who deny their dumps any resets. But inevitably credit for any positive completion or block goes to one player rather than crediting a team for effective execution of a scheme.
But while Brett and I are both skeptical towards traditional “fantasy ultimate” stats (goals, assists, D’s, turnovers) and question legitimate quantification of team-wide tasks, his article highlights some areas of disagreement that I thought might be worth discussing.
First, Brett argues that Ultimate is uniquely team-oriented. I disagree. Team-based sports do present more difficult analytical challenges, but Ultimate isn’t special in this regard. In the NBA, it may take three defenders to execute pick-and-roll defense; it may take all five defenders to fully execute help rotations during the scheme. This isn’t dissimilar to an Ultimate defensive scheme that utilizes a tough mark, a last back, and some switching underneath. Both situations are extremely difficult to quantify; video analysis may be a better analytic tool than quantification. But the challenges are similar and the main difference is that major team sports (like basketball and soccer) have millions of dollars plus nearly infinite cameras and box score data to investigate these problems. The Ultiapps system, the only feasible system to-date, is mostly volunteer-driven.
Second, I think Brett is smart to note that the strength of any one defense may not always be the best strategy to defend against a given opposition’s offense. But I think he goes a bit far in this direction and setting up a simple number of throws to limit the other team’s offense is too simple.
Our spatial data, for example, backs up a long-existent camp of Ultimate thought that it may be optimal defense to force the other team’s offense to work from the sideline. Yes, it is true that each additional pass carries a residual chance of incompletion so increasing opponent throwing attempts should help in theory, but at the elite levels this rarely occurs without some type of additional pressure. Brett may have meant this as a just one potential example of utilizing team-wide metrics rather than specific optimal defensive strategy.
But Jeremy Weiss’ spatial work on probabilities, combined with our finding that (in existing elite men’s club data) even good teams turn it over a lot, suggest to me that we can get more specific. For example, maybe optimal defensive strategy includes sacrificing a decent defender for an excellent puller, pushing the other team to the sidelines, and baiting contested hucks if (and only if) the other team gets close to half field. We are just beginning this analytic journey, but I’m not sure we can’t one day (maybe even soon) reach that type of specificity.
