August 26, 2013 by Brett Matzuka in Analysis with 10 comments
I originally wrote this statistics primer for The Huddle, a now defunct online Ultimate magazine. Around that time, the movie Moneyball was introducing the sabermetric approach, of statistically understanding baseball, to the masses. In the movie, Billy Beane (Brad Pitt) asks, “What is the problem? How do you win given the rift [between rich teams and poor teams]?” The answer was to manufacture runs: runs win games. Beane looked for players whose statistics maximized on-base percentage, as this metric most directly translates to runs and, by extension, wins.
Unlike Ultimate, baseball is a static game where each at-bat is an independent event for all intents and purposes.
My main purpose in this article is to show that while we collect data much like baseball (assists, goals, Ds, turnovers, etc.). Unfortunately, these static, individual metrics are rather meaningless in understanding our sport. We need to move on to more meaningful statistical analysis and inference in order to understand what is important for increasing a team’s success. I plan to challenge each of you to start asking the right questions and then, subsequently, collect appropriate data and information to properly answer those questions.
To better understand, I pose the following anecdotes:
- Player X racks up two impressive defensive blocks in one game. Player X acquired these blocks through poaching that had little impact on the offense. The stat sheet suggests a good game and positive net defensive worth. However, player X also left their intended mark open over 75% of the time, giving the opposition many free resets and multiple quick goals. By pure individual statistics, this player would be considered a great defender despite their ability to play within the team’s system, but they may have done more harm than good with their defensive approach.
- Imagine a defensive scheme focused on fronting the offense’s cutters and having a last back. This last back defender, player Y, can be assumed to be a match up who is very dominant in the air. Under this scheme, if everyone succeeds in their individual jobs, player Y will accrue multiple Ds and again seem to be the best defender by pure individual statistics. But do we neglect to take into account the dump defender who always shuts down the dump when it mattered, leading the offense to take low percentage hucks? No intelligent team in Ultimate will choose to throw a contested dump over a contested huck. The dump defender will accrue no block stats, making their impact potentially unnoticeable.
- Our third hypothetical centers around fantasy ultimate. The best players to take in fantasy are generally defensive cutters and offensive receivers. Both positions have a high probability of a positive outcome while having a negligible probability of a negative impact in their fantasy score. On the contrary, offensive handlers are placed in a position to have very little positive impact in terms of fantasy while constantly dealing with the pressure and responsibility to keep the offense efficient, which inevitably leads to turnovers.
For example, looking over data collected from the 2007 UPA Open Club Championship final (Colorado’s Johnny Bravo lost to Seattle’s Sockeye 15-12) proves the inadequacy of the fantasy approach. Parker Krug, Bravo’s main handler, had seven turnovers and one assist, while Ben Wiggins, Sockeye’s main handler, had four turnovers and no assists. Using an individual framework, it would seem that you should either sit these players (if they are on your team) or allow them to get the disc as they are likely to turn it over (if they are on the opposing team). But watching that game on video, it seemed that both team’s defenses put a lot of focus on keeping the disc away from these players, completely against the individual or box score analysis.
Overall, these hypothetical situations illustrate the problem of tracking individual statistics to rate offensive or defensive effectiveness or to measure success within an offense or defense. We either need to find better ways to use this data or collect more important data. Since, as we have seen, you can constantly show that this data will only lead to confounding variables, it seems the latter is paramount.
I often hear coaches trying to come up with meaningful ways to better assess their team’s effectiveness, often by assigning an arbitrary metric that reinforces an ideal strategy. A coach assigns point values for players executing specific tasks and takes away a point for poor execution of a tactic. One issue with these systems is that they create what I would call a self-fulfilling prophecy or, in other words, a redundant positive feedback loop. For example, if I assign a positive value to players for dumping, reinforcing the importance of resetting the disc, we will soon find that everyone dumps the disc and no one is hucking (as turnovers are given negative values). This will not be efficient and does not reinforce effective ultimate. All in all, it leads players to focus on executing your arbitrary metric as opposed to playing together to achieve a defensive or offensive scheme. For specific purposes, this can be effective, but it only has limited meaning.
This leads to the very crux of the matter. All of the methods mentioned thus far have to do with rating individuals by some arbitrary set of actions or goals. However, ultimate is not an individual sport; it is far from it. Unlike any other major sport, there is no way for an individual to single-handedly score (neglecting a puller running down and catching a first pass Callahan). In baseball, a home run can earn a run. In basketball, the point guard can run the court and lay it in, dunk it, or pull up and shoot. In hockey, lacrosse, and soccer, the center can win the face-off/kick-off and take the puck or ball to the goal and score. Because of this very important fact, we should move away from individual statistics to establish the efficiency, effectiveness, or success of an offense or defense, as no individual can alone make an impact. We should move into analyses that are more appropriate to our sport.
For example, how do we maximize our team’s chance of winning? The most direct answer is that we win if we score more breaks than the opposition. So then the question becomes, “How do we get more breaks than the other team?” The answer is to minimize the opposition’s offensive efficiency.
The first thing to note is that this is not the same as maximizing our defense’s efficiency. For example, you may play on a team of very tall defenders and your most efficient defense, in general, is to force the opposition to huck deep and contest those with your downfield defenders. If the offense you are playing is most efficient hucking, you may be playing right into their hands. It is more efficient for your defense to adapt and force the offense to play less efficient strategies than to try to maximize your strength. Your team should probably make them throw many short unders and break marks in order to score.
But how do we know when we are minimizing the opposition’s efficiency? We first must make basic assumptions. Mainly, offenses are set up to score as efficiently as possible at all times. Secondly, teams will place their players in situations where they can use their strengths. If I have the fastest, tallest, highest jumping player on my team, I will have him downfield as opposed to handling. Understanding these assumptions, we can deduce a framework that offenses will always score in as few passes as possible given the opportunity. From here, we can establish a measuring system to determine the efficiency of an offense analytically.1
Basically, we are trying to assess the best way to determine the effectiveness of a defensive unit playing a specified defensive scheme. Since we have an understanding of how to define efficiency of the offense, we simply want to shift that efficiency to a worse value. If an offense typically scores in six passes, you want to decrease that efficiency and make them score in seven or more throws. Decreasing efficiency, based upon our assumptions, is done by making them throw more passes. However, we want to know that they are throwing more passes due to our defensive pressure and not pure chance, so a statistical inference is done by making sure this net difference is what we would call statistically significant.
This example is nothing revolutionary, and all top players and top coaches probably do these exact calculations in their head on the fly while the game is happening in front of their eyes. The point is that this is what is necessary to truly assess your team and it is not done properly or enough. My challenge to you is to stop putting value in individual statistics, but to start asking pointed questions that will affect your team and construct meaningful ways of evaluating those questions. This is a straightforward example of how to do this and this same idea can be applied to any area of our game. Assess your team as a whole and you will be more successful in understanding how to improve — no single player can win a game for you.
That paragraph can be considered a frequentist perspective approach to the problem. But the problem can also be understood using a Bayesian perspective. ↩
