Quantifying Player Value: Introducing ‘Expected Contribution’ And Ranking The 2013 NexGen Team

No one in Ultimate seriously believes simple statistics like goals, assists, or blocks fairly measure a player’s overall worth. But our new 'expected contribution' statistic - which summarizes each player’s involvement in one overall number -- is a first serious attempt to do exactly this.

August 13, 2013 by Sean Childers, Jeremy Weiss and Husayn Carnegie in Analysis

No one in Ultimate seriously believes simple statistics like goals, assists, or blocks fairly measure a player’s overall worth. Even more advanced metrics – completion percentage, number of touches, or plus/minus – fail to completely summarize a player’s performance.

But our new ‘expected contribution’ statistic, designed using data generated from the UltiApps statistical tracking system, is a first serious attempt to do exactly this. Similar to overall metrics like baseball’s Wins Above Replacement Player (WARP) or basketball’s Player Efficiency Rating (PER), expected contribution (EC) summarizes each player’s throwing, receiving, and defensive involvement in one overall number.

How is Expected Contribution Generated And What Does It Actually Measure?

The intuition behind EC is relatively simple. Each player’s EC score is calculated by measuring the change in team success that took place as a result of that player’s involvement. Catching a pass for a few yards advances your team up the field, so your EC score goes up a bit. You then throw a 40 yard completed huck; your EC goes up a lot. On the next point, you deny your man the disc for the first few throws before getting a layout block. At this point, your EC is sky-high. But if your next catch is a drop and your next throw a turf, then your EC is coming back down to earth. To be a bit more technical, the EC number measures any change in the probability that your team would win the point from the moment “before” you arrived on the scene compared to the moment after your involvement.

Our first step is to generate an overall field probability map that shows a team’s likelihood of winning the point conditional on where the disc is located on the field.¹ We’re able to generate a scoring chart like this for any team that tracks data using the UltiApps system.

But in order to compare players across teams, we need to generate an “average” scoring chart. The scoring chart below is used to generate Open EC scores and is the result of over 10,000 tracked throws from five teams (Doublewide, Ironside, NexGen 2013, Ring and Revolver), mostly gathered during the US Open.

As you can see in the graph, even elite men’s offenses have to work to win points when possessing the disc around their own end zone – the teams have about a 60% chance of winning the point from that location. By the time they move the disc to midfield, the teams have an 80% chance of winning the point. In order to understand how these numbers ultimately relate back to EC, just keep in mind that the 20% change (from their own end zone to midfield) is going to be credited somewhere — or, more specifically, to the group of the players on the field who most helped advance the disc there.

How Can We Use Expected Contribution Scores?

EC is a great starting place when trying to determine a team’s most valuable players. It captures contributions made throwing, receiving, and on defense. Rather than assigning them a subjective value (What is a score worth? Is a huck worth more?), EC uses what we know about scoring probabilities and field position to objectively quantify your on-disc involvement.

It is also a very flexible tool. First, by default, we scale the offensive metrics by the number of O possessions played and we scale the defensive number by the number of D possessions played.² Second, the figure can be calculated based on your team’s individual scoring chart (telling us how much value you added to your team, based on your team’s strengths and weaknesses) or based on a generic scoring chart, like the one shown above. EC derived from the generic chart tells us how much value you would add to a hypothetical elite club team, as compared to other elite club players; the resulting number allows us to fairly compare an elite club player from Ring of Fire with an elite club player from Ironside.³

The total EC number has a plain language interpretation as well. In an otherwise average situation and point, how much more likely is your team to score with you on the field compared to the average player?⁴ Simon Montague’s EC score of 5.8 suggests that elite teams like NexGen would be about 6% more likely to score when playing with him rather than replacing him with the average player in the elite men’s scene. From the little data we’ve seen, the top scores in any division max out around 12%; an EC score above 8.5% should be considered truly elite (top 5%). In other words, based on the data we have, Montague is knocking on that truly elite door.

What are the Limitations?

No single statistic can ever value a player completely, and while we’re excited about our EC metric, we try not to read it in a vacuum. First, subjective impressions are still very important. By definition, our statistics can only measure on-disc involvement; clearing space for another cutter is ignored. Breaking the mark, successfully swinging the disc, or changing the angle of attack are also not reflected in the overall number, so players that are good at this are underrated by EC.⁵ Finally, as sharp Ultiworld readers have noted, our defensive metrics fail to incorporate the increased difficulty of matching up against the other team’s star instead of the other team’s average player; getting a layout block on each counts the same for EC purposes, even though it probably shouldn’t.

We also like looking at other objective indicators to supplement our subjective player evaluations. Completion percentage is a great peg for evaluating efficiency; goals and assists per point or possession played is important to understand a team’s main scoring threats. And to some extent, we still think our Ultiworld D rating may be a better proxy for defensive evaluation.

The All-Stars Amongst All-Stars: First and Second Team NexGen 2013

Normally, we would put seven players on first and second teams. But that would include the entire NexGen roster! So we’re doing mini teams for first, second, and third teams, with explanations written by the statistical editors and Husayn Carnegie (who watched and coded all of the NexGen games to give us the data).

1st Team

Simon Montague:

Universally loved by all of our offensive metrics, Montague was easily NexGen’s best player in 2013. He led the team in involvement yards, rocked a ridiculously high usage rate, and sported the third-highest completion percentage on the team. While he didn’t generate a ton of blocks defensively, he was the team’s best defender in terms of denying touches and denying yardage and came in above average in the Ultiworld D Score. With an Expected Contribution score of 5.8%, Montague’s NexGen performance suggests that he could be an important player for even the most elite club teams in the country.

Jimmy Mickle:

After Montague, there is a group of 3 players who could lay claim to the second most valuable spot. But all three of us would still take Mickle on the first team. Obviously an amazing thrower, he punished teams for going flick and was able to put up big throws and break hucks to anywhere on the field. He can put a flick huck that either drops in perfectly or trails towards the break space at the end of its flight. Objectively, the offensive yardage metrics rate him as the second most impactful player — he also had a great assist rate. Not without some flaws, Mickle ranked “only” 4^th in EC on NexGen, below average in Ultiworld D, and was below the NexGen average in completion percentage at 91%.

Dylan Freechild:

Subjectively, he’s clearly as talented as Montague but a bit less consistent. Our Ultiworld D rating, which places a fair bit of value on blocks, loves him and rates him as the second strongest defender on the team. But, subjectively, we wonder if this also speaks to a poachy tendency. EC is less kind, ranking him 6^th on the team, but that might be because his break throws, an essential positive in his game, are completely ignored. Most impressive is that he rocks the second highest usage rate on the team (over 20%) with a 93% completion percentage. He wasn’t a unanimous pick here, but does any other player on the squad inspire so much fear?

2nd Team

Aaron Honn:

We had to snub one important player from the first team, and it’s Honn, who had first team credentials according to the advanced stats and garnered one first team vote. With an EC score of .02 – second highest on the team – our metric calculates a high level of reception contribution from Honn that he supplemented with average throwing and defense. The yardage metrics note that Kerns, Honn, and Driscoll were the most important deep cutters, but Honn didn’t necessarily stand out from that group. Final scary stat: He’s the youngest player on the team.

Elijah Kerns:

Appearances aside, Kerns is a surprisingly good defender that was often on the field when NexGen crucially needed a break. He rated well according to Ultiworld D rating. He also had the third highest EC score on the team, in large part due to a positive throwing number. A super reliable thrower with good range and breaks, the video impression is that he struck a good balance between throwing upfield and maintaining possession.

Aaron Neal:

At this point, we begin to discuss players who have notable weaknesses along with impressive strengths. Perhaps their defensive stopper by the end, Neal was the non-American player on the squad who got tougher matchups and a greater role as the tour wore on. From an offensive involvement perspective, he ranked second-worst, rarely touching the disc. His completion percentage was average. But – along with Janin, Honn, and Freechild – Neal is the rare defender that can deny his man yardage and touches, but still generate blocks at the same time. He gets a bit of a subjective boost for ending the tour on a strong note.

3rd Team

Camden Allison-Hall:

Despite rating out as one of the better players on the tour in EC – above Freechild and Neal – the video suggests that Allison-Hall benefits from his specialized role on the team. He’s a great deep threat: he’s fast, good in the air, and skillful with his cut timing. But his offensive numbers are clearly aided by this limited (albeit important) role. He has a high completion percentage, but a low usage rate, and was rarely put in difficult positions with the disc. His defensive numbers are the most divergent of all players: Defensive EC hates Allison-Hall, but Ultiworld D sees him as above average.⁶ Overall, the 2013 tour was a huge statement of quality by Allison-Hall in light of a mediocre 2012 performance; he was the first player onto our 3^rd team and you could argue that he should be on the 2^nd.

Tim Morrissy:

A player that might rate higher in traditional analysis, our system sees Morrissy as a player with limitations as well as strengths on both sides of the disc. On offense, he played the role of a goal scorer (he had the best goal scoring rate on the team) with very safe throws (highest completion percentage on the team). But he was more limited in touches and involvement yards. On defense, he scored great in the Ultiworld D rating and limited touches and yards, but generated only an average number of blocks. EC sees him as one of the most average players on the squad. One of the tougher players to rate.

Will Driscoll:

An important offensive threat and, truthfully, one of the top seven players on the tour, Driscoll rated number 6 on EC. But his early season defensive struggles, combined with a short sample size of games, meant he gave up too many yards and scores to earn a spot on the top two teams. If the tour were replayed, he could probably fight for first team. It would be a bit unfair to leave him off entirely because of defense, given that he often took the most difficult assignment.

Ranking The Rest Of NexGen

Jacob Janin: He was the bubble player that you could make an argument for on the third team. In last season’s NexGen tour, we noted him as the surprise breakout star, in part due to a ridiculous 99% completion percentage (that came down to 89% in the 2013 season). The truth on Janin is probably somewhere in between; his ability to get the offense going with a strong first cut is hugely beneficial and probably underrated by our numbers.

Chris Kocher: EC sees him as around average (for the elite club level) in all aspects. The yardage, usage, and scoring metrics suggest he just wasn’t as involved in the NexGen offense this season. Fought injuries towards the end of the season.

Mitch Cihon: Seemed noticeably less explosive than the elite players that made the top teams. EC thinks, based on his 2013 NexGen performance alone, that he would be between the middle and bottom of an elite club roster. Completed less than 50% of his hucks, but showed more solid defensive numbers.

Trent Dillon: Like Morrissy, a player with an interesting stat line: 2^nd highest completion percentage, a decent usage rate, and some assists. Along with Neal, Dillon was the other truly hustle defender, often times drawing tough matchups, which we wrote a bit about midtour. But while he was able to deny touches, he was less successful at denying yards or generating blocks – from the video, it seemed as though he was often an inch away. It would be interesting to see his defensive stats in a longer timeframe.

Jay Clark: Big body but a below-average defender at this level by both defensive EC and Ultiworld D rating. His positive contribution was in downfield receptions; he had a decent number of reception yards, but less than the other deep threats.

Elliot Erickson: A technically skilled and experienced player, Erickson (at his current age) didn’t show that he had the size or athleticism to compete on this level. We often note the even playing time style of NexGen, but, over the course of the tour, Montague played 1.5 times as many points as Erickson. More notable is that Montague threw almost 15 times as many passes as Erickson.

**Special thanks to everyone involved on the statistical project, including Ultiapps, Ian Guerin, Husayn Carnegie (who coded the NexGen 2013 games), Kahyee Fong, and our US Open volunteers.  If you would like the analysis you see on Ultiworld for your team, start tracking your teams games using the Ultiapps platform and email us at [email protected] **

1. We derive our one-dimensional scoring charts, which only look at vertical (upfield and backfield) yardage, rather than using the prettier two-dimensional charts (which show both upfield and lateral/swing yardage) that we’ve shown before. The reason behind the decision is that most points are played with force-flick; the backhand side of the two-dimensional chart almost always has a higher probability of scoring than the flick side. Using the two dimensional chart would thus credit players for moving the disc to the backhand side even if the force is backhand – the UltiApps program doesn’t track force. ↩

2. We treat all offensive and defensive possessions the same; we don’t care whether the player played O Line or D Line. Though, obviously, O Line players will always end up having a higher number of O Possessions than D Possessions. ↩

3. Of course, whether you would have been able to make those same throws, catches, and D’s on the elite club level that you made at a lower level is a much dicier proposition. ↩

4. Whether that is the average player on your team or the average player in your “division” (elite mens, elite womens, etc.) depends on whether the metric is calculated based on your own team’s scoring chart or on a division-wide scoring chart. When using your own team’s scoring chart, the resulting player rankings are going to be inevitably tied up to the overall quality of the team: Studs on college teams will have much higher EC scores than Revolver’s best players, as the gap amongst players is much higher in the first than the second. ↩

5. An earlier footnote emphasized that we use our one-dimensional (rather than two-dimensional) scoring chart. The basic intuition to keep in mind is that completing a swing pass across the field for neither positive nor negative vertical yards will result in no change to your EC score, because you haven’t moved along the one-dimensional line that only tracks upfield yardage. ↩

6. Give us time; we’ll look at potentially play styles that can lead to this odd result ↩