2015 Jeopardy Tournament of Champions: Semifinal Update

Couple of random thoughts before revealing my predictions for the ToC Semifinals:

– My system did pretty darn well this year, getting 4 of the 5 winners of the semifinals correct. Granted, it wasn’t much of a radical prediction to say that Matt Jackson and Alex Jacob would win their games. However, I’d argue that Kerry Greene, despite nominally being the top seed in her game, was not an obvious favorite, nor would Catherine Hardee be easy to pick out as a favorite to win from the third lectern. The system’s one miss was favoring Greg Seroka and Kristin Sausville over the eventual winner from Tuesday’s game, Brennan Bushee, though to be fair Bushee won the game from last place by being the only player to get Final Jeopardy correct.
– The Wild Card cutoff point was higher than average this year at $14,000. The average over the history of the tournament (after doubling the scores of the pre-double dollars era) stood at $10,464. Anecdotally, I’d think that may be the effect of almost all players going into their games with the goal of not necessarily winning, but playing to hit a self-determined goal score that would earn them a wild card. With much more data out there about things like historical wild card totals, I wonder if this is going to lead to a situation where the wild card cutoff will always be higher than historically expected. Then again, last year’s cutoff was $9,100, and most of the data was available then too, so it’s just as likely there’s no great reason for this year’s cutoff being so much higher.
– I would love to know how the Jeopardy team selects the semifinal matchups. I’ve tried to come up with some set of seeding rules, but nothing I can find explains the matchups perfectly. The only rules that I know for sure are that players will not face their opponents from their quarterfinal match, and two people with the same first name will not play each other. This is different from the quarterfinals, where the games are fairly obviously seeded so that in each match one of the top 5 players (ranked by games and money won) plays one of the second five and one of the bottom five.
– I need to thank Andy Saunders of The Jeopardy Fan for his guesses as to what the semifinal matchups would be, which turned out to be correct and give me a little more time to run the numbers. Jeopardy didn’t officially release the matchups until Monday morning (as far as I saw through the official channels), which is slightly annoying for those of us in the game-show-data-analysis business.

toc-sf1
Our prediction of Jackson vs. Jacob vs. [Seroka/Sausville/Hardee] isn’t going to be happening, since neither Greg Seroka nor Kristin Sausville made the second week, and Catherine Hardee is playing Matt Jackson in Wednesday’s game. Instead, the favorite for that third slot becomes Dan Feitel, winner of the Semifinal Matchup sweepstakes. He had the biggest movement in our prediction engine thanks to staying out of the path of the two juggernauts, increasing his chances of winning the tournament from 4% to 14%.

toc-sf2

Alex dominated his quarterfinal game, becoming the only player to have a lock game last week. We see no reason to expect a different result from his semifinal matchup against Brennan Bushee and Vaughn Winchell.

toc-sf3

If anybody is going to keep us from a final of M Jackson v. A Jacob v. AN Other, Catherine Hardee has the best chance of doing it. She could actually outbuzz Jackson, possibly the first time he’s ever had to face somebody who could do that.  If she can keep her number of wrong answers down and take a few Daily Doubles, she could still certainly crash the finals.  However, the smart money still has to be on Jackson winning this matchup.

toc-sf-odds

Thanks to a slightly easier Semifinal matchup, Alex Jacob takes the tag of favorite by a clear margin over Jackson. Both men are close to 1-in-4 odds of taking the title. As prevously stated, Dan Feitel moves from his quarterfinal position of “best of the rest” into a solid third place thanks to avoiding the two favorites. Good luck to all participants, and here’s hoping that the games to come are just as fun, interesting, and exciting as last week’s games.

Predicting the 2015 Jeopardy Tournament of Champions

Trivia Christmas is coming a bit early this year. Instead of the usual 18 month wait between events, we’re getting another installment of the Jeopardy Tournament of Champions a mere twelve months since the last incarnation. It’s always fun to try to predict how these things will go, so we’ll see if we can better our efforts from last year, where we had eventual winner Ben Ingram rated as the fifth most likely winner in a fairly wide-open field. This year, however, the field has broken down into three distinct groups.  The field is led by two co-favorites, each of whom have a one-in-five chance of victory.  There are three dark horse candidates for the title, with chances of victory ranging from 11% to 14%. Finally, we have the remaining ten contestants, none of whom have more than a 1-in-25 chance of taking home the title.

Before we reveal our predictions, let’s do a quick recap of our methodology. We analyse players’ ability using two metrics. The first is Buzzer Percentage, which attempts to measure how often a contestant will buzz in. Since we can’t measure this directly, we use a Monte Carlo simulation to estimate it based on how often a player has successfully rung in during their previous games. The second metric is Precision, which is the percentage of times that they are correct on clues where they buzzed in (ignoring Final Jeopardy & Daily Doubles). For reference’s sake, the average Jeopardy contestant has a buzzer percentage around 60% and a precision around 87%.  This system is based on the system that the designers of Watson used to simulate human opponents for Watson to practice against.

Once we have those numbers, we run a bunch of simulated games using each player’s Buzzer Percentage and Precision, and determine how likely they are to win their quarterfinal games. Using the results of these simulations, we track how likely a contestant is to win their quarterfinal or qualify for a wild card, and use that to weight a player’s potential matchups in the semifinals and finals.  Here’s what our simulation gives us for each of this week’s quarterfinal games:

Monday's Odds

Monday’s predictions.

If you want a dark horse candidate for your those office Jeopardy pools I know you’re all in, Catherine Hardee is a decent candidate. Despite winning one game fewer than Dan Feitel and Vaughn Winchell, her strong Buzzer Percentage and respectable precision gives her a slight edge in Monday’s matchup.

Tuesday's Game

Tuesday’s predictions.

Two of the three dark horse candidates face off on Tuesday’s show. Greg Seroka and Kristin Sausville both have a decent chance of taking down the whole tournament. Brennan Bushee is a bit unfortunate in his draw; despite his strong precision he could get squeezed out by two of the tournament’s strongest buzzers.

Wednesday's Odds

Wednseday’s Predictions.

96.8% precision. Ninety-six point eight percent precision. That is a crazy, insane, whacked-out number, and helps make Matt Jackson, unsurprisingly, one of the favorites for the tournament. Jackson draws a fairly easy group for his first game. John Schultz’s metrics are among the weakest in the field, and are actually comparable to an average contestant. Jennifer Giles could take an upset win, and in doing so become only the second Teacher’s Tournament winner to make the second week.  The first, of course, being Colby Burnett, who won the whole thing in 2013.

Thursday's Odds

Thursday’s predictions.

There’s no nice way to put this: the lineup for Thursday’s match is by far the weakest. All three contestants are very conservative buzzers; expect a higher than average number of clues going untried. Kerry Greene is at least precise when she buzzes in, which gives her quite an edge over the error-prone Andrew Haringer and Elliot Yates.  Any way you slice it, this game will give an automatic semifinal berth to a player who may have had a hard time qualifying in other games.

Friday's Odds

Friday’s predictions.

Alex Jacob has the busiest buzzer finger in the entire tournament. When paired with his excellent precision, it makes him the co-favorite to win the entire tournament. Scott Lord and Michael Bilow will have a hard time matching up.

Complete Picture
Alex Jacob and Matt Jackson are so close that it’s fair to call them co-favorites to win the tournament.  Assuming that they manage to avoid each other in the semifinals (which is what caused most of their eliminations in our simulations), they should get to lock horns in a great two-day final.  Chances are they’ll meet one of Greg Seroka, Kristin Sausville, or Catherine Hardee there.  At that point, the only winners I’d be willing to predict for certain would be the audience, who would get to watch an exhibition in excellent Jeopardy gameplay.

Jeopardy ToC Update: Semifinal Game 2

The Stats

You gotta hand it to Arthur Chu.  Despite being handed the toughest draw in the field, he’s making this look easy.

toc_sf2_table

Mark Japinga’s negging and Rider being crowded out on the buzzer could have made this a runaway, but Rider doubled up on a good Daily Double, and Japinga made a late run to remain relevant.

Daily Doubles

I’ll admit – as soon as Chu wagered $1,000 on his first Daily Double, my bad-bet-o-meter went crazy.  He found it early in the Jeopardy round, having called for the $600 and $800 clues in random categories looking for it as is his M.O.  He was trailing Japinga slightly at the time, $2,600 to $2,800, with Rider yet to open her mouth. I loaded up my still-in-development Daily Double evaluator, and fed it the parameters.  Here’s what it spat out at me:

dd_sf2

The five lines represent how comfortable you are with the category.  They have nothing to do with the expected difficulty of the clue, which has already been accounted for.  The table may be hard to read, but it suggests 3 wagers, depending on your level of confidence:

– $300-400 if you are not confident. Enough to take over the lead, but still enough to stay in touching distance if you miss it.

– $2,000 if you’re confidence is average. Beyond a certain point when you’re leading, every dollar you have is gives you a smaller chance of winning than the previous dollar. At this point, the diminishing returns of increasing your score catch up with the chances that you’ll actually get the clue right.

–  $2,400 if you’re supremely confident.  The system suggests the diminishing returns beyond that point are not worth an all-in wager, but I certainly wouldn’t begrudge one if that was your choice.

Chu’s selection of $1,000 is an odd one. It’s not conservative enough if you don’t like the category (which I imagine was Chu’s reasoning), as it leaves you at least two clues behind to catch up if you’re wrong. It’s also not aggressive enough to take advantage of the opportunity that a Daily Doubles represents.

Chu hit the first Daily Double in Double Jeopardy, and again wagered $1,000.  The situation was much different this time, with Chu on a commanding $13,200 over Japinga’s $3,600 and Rider’s $1,800.  Without looking too deep at the details this time, I think the two choices in wagers would be the minimum of $5 if you want to protect your lead, and a more aggressive wager of $6,000, looking to close out the game here but still keep twice Japinga’s score if you’re wrong.  I don’t begrudge Chu his wager that much, since I doubt there was any great difference between wagers of $5 and $1,000. And despite the disclaimer that this could be the stupidest thing she ever did, Rider’s true Daily Double later with her score of $4,600 a long way back of Chu’s $16,600 was the only real move she could have made, especially in a category that she seemed to like.  She answered correctly, and Japinga made a late run to set up a Final Jeopardy where all three players could still take the victory.

Final Jeopardy

The scores were Chu with $17,800, Rider with $12,800, and Japinga with $8,000.  I’ll leave the exact analysis of the situation to Keith at The Final Wager, who does a better job at it than I could ever hope to. But I do want to talk about something that I feel very strongly about which comes into play here: Stratton’s Dilemma.

Coined by Andy Saunders, Stratton’s Dilemma is the term for the situation Rider finds herself in, where she has to choose between two possible wagers. Let’s analyze her situation. We can assume that Chu will wager $7,801 or thereabouts, the amount needed to guarantee his victory as long as he gets Final Jeopardy correct. We must assume that Chu responds incorrectly, otherwise we have no chance of winning.  If Chu is incorrect, he will be left with $9,999. Thus, it is in our best interest to wager no more than $2,800, staying ahead of Chu no matter what our outcome is. However, the presence of Japinga complicates matters.  If Japinga wagers everything and doubles up, he’ll have $16,000.  To ensure that we remain ahead of Japinga in that situation, we have to bet at least $3,201.  Sometimes we can find a wager to satisfy both scenarios, but this is not one of those times. We have to choose one or the other, and know that some of the time we will choose incorrectly and lose when we could have won.  It’s an infuriating position. Which bet should we choose?

Well, let’s break down the scenarios.  Since each player can either be correct or incorrect in Final Jeopardy, there are 8 possibilities to consider.  Assuming that Chu wagers $7,801, and Japinga wagers his entire $8,000 (our worst case scenario), what happens when we bet $0?stratton1Compare that to what happens when we bet everything:

stratton2

We’ll always win in either case if we’re the only person to respond correctly. When we bet small, we will win if everybody answers incorrectly.  If we bet big, we will win if the leader misses and both we and the player in third respond correctly.  Which is more likely to happen?  If only we had a large repository of previously played Jeopardy games to look at…

I took a look at 2,184 games played over the last 10 years where all three players made it to Final Jeopardy.  Specifically, I counted the times when each of the above scenarios happened.  I separated the players by their ranking going into Final Jeopardy, so I know how often the leader missed Final Jeopardy while the other two got it right, or only the second-ranked player responded correctly, for example. And what were the results?stratton3

Unsurprisingly, the scenarios when all three players responded correctly and all three players responded incorrectly are more likely than any other case.  Players possess a shared knowledge base, and a question that one person knows is likely to also be known by the others, and vice versa.  The situation we’re hopping happens when we bet big, where 2nd and 3rd place answers correctly but the leader does not, is the least likely to happen. If we quantify our chances of winning using the above table, we have a 30.9% chance of victory with a small bet, while we only have a 19.1% chance of success if we wager big.  I don’t know about you, but if one choice of a dilemma increases my chances by over 60%, I wouldn’t call it much of a dilemma.

The Odds

Well, we’re going to get the matchup we wanted, Collins vs. Chu. And frankly, this tournament is now Chu’s to lose. Regardless of who wins the next match between Ben Ingram, Joshua Brakhage, and Sandie Baker, I can’t help but see Chu as a strong favorite, especially over a two-day affair where the normally high level of variance inherent in a Jeopardy match is lowered.

toc_sf2_odds

 

Jeopardy ToC Update: Semifinals Game 1

The Stats

The “curse” of second place continues.  This makes it the sixth straight time that the player with the second ranked chance by my system won the game.

toc_sf1_table

Terry O’Shea may have been extra choosy in her clue selection, but her 100% precision and her tendency to score rebounds off of the others still gave her a strong showing.  Jared Hall and Julia Collins played very similar games, but Hall found all three Daily Doubles.  Had he converted his all-in wager early in Double Jeopardy, the result may have been a lot different.  Instead, Julia played up to the expectations placed upon her, and became our first Finalist.

Strategy

Hall, as lampshaded by Trebek at one point, was the only player playing tonight who picked clues from the middle of the board at first, hunting for the Daily Double.  He was rewarded handsomely for his efforts, as all three Daily Doubles landed in his lap.  The first and third were rather straightforward efforts, with Hall having less than $1,000 in the Jeopardy round and $2,000 in Double Jeopardy, which almost always necessitates the maximum wager allowable wager of $1,000 or $2,000, depending on the round.  His second Daily Double, however, I think deserves some extra attention.

Hall had $4,200 early in the Double Jeopardy round, poised between Collins’ $6,800 and O’Shea’s $3,000.  The Daily Double was hiding in the $1,600 clue in the category Great American Novels.  The average player would probably pick a middling value without too much thought, probably about half their score or less.  Hall did what I imagine most good Jeopardy players would do, and bet it all.  There’s still enough money left on the board ($27,600 out of the $36,000 that started the round, as well as the other Daily Double) that even if you go bust, there’s still enough time to make a comeback.  And if you respond correctly and double up, you’ll have a lead and be well poised for the rest of the game.

I’ve mentioned that I’m working on a system that evaluates Daily Doubles and determines the expected win % of every possible wager, trying to find the “best” bet, if such a thing exists.  It’s still very rough around the edges, but I thought I’d give it this situation to mull over.  Here’s what it said:

dd_sf1

The five lines represent how comfortable you are with the category (not the clue value – that’s already been baked in to the system.), with red being the least comfortable to green being the most comfortable.  In the end, your comfort level with the category didn’t matter, as bets could be clustered into three categories:

– $0 – $1,200: Pretty bad.  You’re still going to be in second place no matter what.

– $1,200 – $2,600: Even worse.  Best case scenario is that you’re still trailing the leader, but now if you’re wrong you’re dumped down into 3rd place.

– $2,600+: Best. The benefit of being in first place if you are correct greatly outweighs the chance that you’ll be in third.  What’s interesting about this section is that your chances decrease as your wager increases beyond what it would take to get into first, indicating that perhaps what you gain by increasing your score above $6,801 is not worth what you lose if you by going further and further behind.  Like I said before, I don’t consider this system to be ready for prime-time, but it gives us something to think about: that a knee-jerk all-in bet in this scenario may be good, but not optimal.

Going into Final Jeopardy, it was still anybody’s game.  Collins led with $12,000, O’Shea had $8,200, and Hall was right behind them with $7,600.  This situation may look a little boring at first, but digging a little deeper into the math reveals a truly fascinating situation – and one that’s very scary for Collins, despite being in the lead.  Keith Williams of premier Jeopardy wagering blog The Final Wager has an excellent write-up and video detailing why, one that I highly recommend watching.

The producers of Jeopardy stop tape to give the players as much time as they want to calculate their wagers; Collins took 15 minutes to finally settle on hers.  In a cruel twist of fate, all of the strategizing proved moot, as only Collins was able to get the correct response in Final Jeopardy, and is our first confirmed finalist.  Well done to her.  She rode her luck a little, earning a wild card with a historically below-average score, and not getting any of the Daily Doubles in this match, but I don’t think anybody could begrudge her her place in the finals.

The Odds

Collins moves up to the top by virtue of having secured her finals place.  The remaining players’ chances only moved a fraction of a percentage point – Collins’s stats were darn near close to the average expected stats of the player coming out of the first semifinal.

toc_sf1_odds

We’ll see if tomorrow will get us the final match that everybody wants to see: Arthur Chu vs. Julia Collins, or if Mark Japinga or Rebecca Rider will upset the storyline.

Jeopardy ToC Update: Semifinals Preview

Sorry that I never got an update out on Friday’s match.  Jeopardy was preempted here, so I didn’t get to watch the match until late Saturday.  Quite a good match, mind, as all three players played well and wagered well, and all three were rewarded with advancement.

toc_5_table

The only player who played close to expectations was Mark Japinga.  Jared Hall played conservatively, but never made a misstep (outside of 1 Daily Double), and took the win after Japinga wagered for a Wild Card spot in Final Jeopardy.  Sandie Baker didn’t have the best game either, but still had enough money to bet big in Final and take the last Wild Card spot.

So, we have our nine players.  Stepping back to evaluate the performance of my system so far, I’d have to give the system’s predictive power a rating of B- so far.  It whiffed on Andrew Moore being the favorite, but on the other hand of the five players least likely to advance, four of them indeed failed to advance. On an individual game basis, the middle ranked player won all five games, although to the system’s credit in two of those games the favorite was leading heading into Final.toc_original_odds_after_week_1

Looking forward, the picture is much more clear.  We know what the three semifinal matchups are going to be (thanks again to The Final Wager for releasing them on Friday), and the picture of what our three finalists are going to be is getting clearer.  As a result, there have been some massive shakeups in our odds.  Let’s look at each matchup in detail.

Monday’s Game

toc_sf_1

(Our buzzing and precision scores have been recalculated to include each player’s quarterfinal performance)

After looking at the draw, I’m pretty sure that the semifinal matches are seeded based on each player’s quarterfinal score. The top three winners go into one group, the other two winners and the best wild card go into the second group, and the remaining wild cards go into the final group.  Each game is made up of one player in each group.  Terry O’Shea’s Thursday score was the third best score among the winners, and I bet Jared Hall and Julia Collins are thankful for that fact.  In another universe, they could be playing against one of the other top seeds: Arthur Chu or Ben Ingram.  Instead, Hall and Collins will likely battle it out among themselves for the finalist spot.  Hall’s stronger buzzing percentage gives him the edge over Collins’ better precision.

toc_sf_2

If Arthur Chu ends up winning the tournament, nobody could ever accuse him of getting an easy draw.  After taking down Andrew Moore in his quarterfinal match, he now faces off against Mark Japinga, our system’s current favorite. Rebecca Rider rounds out this trio – she’ll have to be at the top of her game to get past Chu and Japinga.

toc_sf_3

Wow.  Just … wow.  After factoring in each player’s performance in the quarters, we have a situation where each player’s buzzing percentage is practically the same.  Ben Ingram has the edge in the system thanks to his superior precision, but really this one could be anybody’s game.  It would not surprise me to see Joshua Brakhage or Sandie Baker advance.  Expect this game to be close going into Final Jeopardy.

The Odds

With the semifinal matches now set, our odds now look quite different:

toc_srf_odds

 

Mark Japinga now takes up the mantle of favorite.  If Arthur Chu beats him on Tuesday like he did to our previous favorite, I would be very surprised if he didn’t go on to win the tournament.  Ben Ingram moves up to #2 despite Wednesday’s game being little more than a crapshoot.  Monday’s winner will probably end up being the underdog going into the final games; the system only sees one of those three winning one time in four.

No matter who advances, expect some excellent knowledge and strategic game playing on display.  Check back all this week for updates.

Jeopardy ToC Update: Quarterfinal Match 4

When I suggested in my preview that you could “expect [a] game where a number of clues pass by unanswered”, I didn’t quite mean it to this extent.

The Stats

toc_4_table

Whether it was the material, the players, or the lunch break that would have preceded this taping, the game just never got off the ground. All three players underperformed on their buzzing, and only Terry O’Shea hit her previous level of precision. Thirteen out of the 57 non-Daily Double clues passed by our competitors without a single buzz, not to mention several more that only elicited incorrect responses.  It was a bad day at the office for all parties involved, and unfortunately it cost two of the players their shot at the title.

The Strategy

Normally I’d break down the strategic choices made by the players in this space, but there’s not much to say.  All three Daily Doubles were found by the player in distant last place, all three times the player correctly wagered big, all three times the wager was lost. Drew Horwood finished the match in the red, and didn’t play Final Jeopardy, while O’Shea had a slim lead of $8,800 to Sarah McNitt’s $8,600.  Both bet big, since you’d think their scores wouldn’t be worth a wild card, although this tournament is shaping up to have a low cutoff (again, they wouldn’t know that at the time).  O’Shea bet to lock-out McNitt, and was correct, earning the automatic advancement.  McNitt bet as big as she could, but still kept enough back to win on a Triple (Double?) Stumper, but missed Final Jeopardy.  Her final total of $500 is already known to be not enough to advance.

The Odds

On a more positive note, with Horwood and McNitt dropping out, we can now welcome Rebecca Rider and Julia Collins into the semifinal fold.  Rani Peffer’s $7,599 is looking pretty good at this point. and Jim Coury’s $5,600 still has a shot of holding up.  We’ll see what happens after the dust settles on tomorrow’s last quarterfinal.

toc_4_odds

Jeopardy ToC Update: Quarterfinal Match 2

Jeopardy tournaments are a harsh mistress.  One bad game, and you’re out on your backside.

Tonight’s Stats

 

toc_2_chart

Arthur Chu’s performance was nothing less than I expected from him coming into the game.  It was Andrew Moore’s underachievement (and to a lesser extent Rani Peffer as well) that turned the game into a blowout for Chu.

Daily Double

My gut instinct on Daily Doubles in the Jeopardy! round is to always bet aggressively.  When Moore found the Daily Double after he and Chu went on a wild goose chase for it, most of the high-dollar clues were off the board and the two of them were tied at $2,600, with Peffer trailing at $600.  Moore opted to bet only $1,000, and after analyzing the situation, I’m inclined to agree with him.  Given that the DD was hidden in the $1,000 space (indicating a harder clue), and that most of the money had been stripped from the board, a modicum of restraint was to be called for.

However, the situation was different when Moore found the first DD in the Double Jeopardy round early on.  He had a slight lead at the moment, $5,600 to Chu’s $5,200 and Peffer’s $2,200.  It’s a close decision, but this would have been the time to make a move.  A wager of all or almost all of his money would have given him the opportunity to take a stranglehold on the game with only one DD left on the board.  He instead chose to wager $2,000, and perhaps this was the right move, since he missed the clue.  After that stumble, Chu went on a hell of a run, and by the time he found the 2nd Daily Double late in the round, he had an unassailable lead.

Final Jeopardy

Not much to say here.  Chu had a lock on the game, and could sit this one out.  Moore and Peffer knew their current scores wouldn’t hack it, so each bet everything but $1.  Moore missed, and has gone from our favorite to all but out in the blink of an eye.  Peffer nearly doubled up, and takes the #2 spot on the wild card list for the nonce with $7,599.

Updated Odds

With our favorite knocked out of contention, there’s no surprises who took over at the top of the list.

toc_2_odds

 

A bit of change elsewhere on the list.  With Moore no longer a potential hurdle, many people saw their chances of getting to the finals and winning increase.  Two wild card scores coming in under expectations increases the chances of the people yet to play of taking a wild card.  Rebecca Rider’s score in the clubhouse of $11,600 is looking better; she’s even money to advance now.

Arthur Chu has dropped the gauntlet.  Will Julia Collins pick it up tomorrow and keep the chances of everybody’s dream matchup alive? Check back tomorrow for the analysis.

 

Jeopardy ToC Update: Quarterfinal Match 1

Congrats to Ben Ingram, becoming the first person to punch his ticket to the second week of the tournament.

Tonight’s Stats

toc_qf1

When I looked at the difference in contestant metrics between regular games and tournaments, buzzing percentage decreased by an average of 12% while precision remained relatively stable.  I’m happy to report that, whether by luck or design, both Ingram’s and John Pearson’s predicted metrics came darn close to matching the actual result.  Rebecca Rider had a tough match, and her poor precision is probably related to the fact that she only attempted 12 responses.

Daily Doubles

Strategically, the players went searching for Daily Doubles early, and attempted to use them to their best advantage.  Both Ingram’s early Jeopardy round DD and Pearson’s early Double Jeopardy DD came in situations where all-in wagers seemed logical; it was unlucky that both missed their responses.  Ingram’s second DD of the game came midway through the round with him in a commanding position, leading $11,600 to Rider’s $3,800 and Pearson’s $1,200.  I’m doing some preliminary work on theories behind Daily Double wagering, and even in the best of cases I think the maximum one should bet in this situation is around $2,500.  His bet of $100 was born out of smart quarterfinal strategy, looking to secure a high score rather than trying to lock the game down.

It’s possible that his failure to bet big prevented him from securing the lock game, as Pearson went on a late run to stay in the running for the automatic berth. The scores finished $16,100 to Ingram, $8,400 to Pearson, and $6,600 to Rider.

Final Jeopardy

toc_qf_fj

The way I saw it, Ingram had two choices for wagers: the traditional lock-out wager of $701, or the conservative $0, maximizing his chances of earning a wild card while still possibly winning the match.  He chose to wager $701, which I think is the correct choice. If he’s wrong and John doubles up, he’s only cost himself 4% on his chance of winning a wild card by losing $701.  On the other hand, his chances of winning the match outright go up by almost 20% with the wager of $701, since he now wins whenever he answers correctly and Pearson misses, whereas before he was at the mercy of Pearson’s response only.

Rider opted to wager only $5,000 of her $6,600, a wager that I cannot support. She’s playing solely for a wild card spot at this point – there should be no permutation that allows her to win on a Triple Stumper.  In that case, since $6,600 is below the expected wild card cutoff line, she really should be looking to double up to maximize her chances.  If she doubled up, I expect her score to earn a wild card 48% of the time.  As it stands, she answered correctly, and her final total of $11,600 leaves her with a 39% chance of advancing.  Had she missed, a score of $1,600 would have advanced only 2% of the time, so it’s likely not going to be worthwhile to leave something in reserve.

Pearson was in a similar situation as Rider, and chose to bet everything.  If he stood pat, he would only advance 12% of the time.  If he doubled up, in addition to the chances that he would win outright, his score of $16,800 would see him get a wild card about two-thirds of the time.  Unfortunately, he couldn’t come up with the right response, and barring a major miracle his tournament is over.

Updated Odds

The biggest winners tonight might actually have been the other 12 contestants.

toc_qf1_table

 

With Pearson out of the running and Rider putting up a solid, yet beatable score, everybody’s chances of earning a wild card go up a few percentage points.  Despite winning, the shifting wild card situation sees Ingram’s chances of winning the whole event go down thanks to the potential matchups in the semifinal becoming worse. Sandie Baker’s chances have actually tripled, due to the increased chances of getting a wild card and her own potential road to the final improving.

See you back here tomorrow, where two of the top three play against each other. And Rani Peffer probably beats Arthur Chu and Andrew Moore to throw all my predictions into the garbage.

Predicting the 2014 Tournament of Champions

B1ufETNCYAE4BYS

People who have made nearly $17.3 million from Jeopardy in the past 18 months, not including Colonial Penn Life kickbacks.

This Monday kicks off the 24th Jeopardy! Tournament of Champions, a two week sesquiennial bacchanalia of trivia where one lucky person gets to inscribe their name on the annals of Jeopardy! history.  It’s the closest thing to an end-of-season playoff that we have in the American game shows, and in the spirit of March Madness I thought I’d put on my Nate Silver hat and take a shot at predicting who the winners would be.  Let’s just ignore that in this case I’d be “predicting” the results of an event that was actually filmed a couple of months ago.

The Rules

classictoc

There is no truth to the rumor that the first Tournament of Champions was held inside a seaside photo booth.

The tournament format, which was supposedly created by Alex Trebek himself, sees fifteen previous champions come back to play over two weeks.  The fifteen participants are the winners of qualifying tournaments (currently the College Championship and the Teachers Tournament, previously winners of the Teen Tournament and the Seniors Tournament also got invites), and then enough champions who played since the last Tournament to fill in the remaining slots, ordered by number of games won, then amount of money won.  The cut-off this year was Mark Japinga, who won 4 games and $112,600. (One 6-game champion was not invited back due to legal issues.)  The first week sees the fifteen contestants placed semi-randomly (I’m pretty sure the contestants are seeded so that each game has one of the top five players, one of the middle five players, and one of the bottom five players) into five quarterfinal games.  The five winners of these games are guaranteed passage to the next round, along with the four highest-scoring non-winners as wild cards.  These nine players are again randomly drawn into three semifinal games, with the stipulation that you cannot play against someone you already played against in the first round.  The three semifinal winners then face off in a two-day final, where the cumulative amount won on both days is used to determine the tournament winner.

It’s an interesting format because proper play requires a shift in strategy.  Only the semifinal game is a traditional winner-take-all game of Jeopardy.  In the quarterfinal, you don’t necessarily need to win the game, just have enough money at the end of the game to be one of the top four non-winners. This leads to some conservative wagering on Daily Doubles and in Final Jeopardy.  The catch is that you are sequestered before you play your quarterfinal, so you do not necessarily know how much money would qualify you for the semifinal. There have been years where a score of $20,000 would have sent you home, and one year where multiple people finished with $0 but still made the semifinals.  (I’ve done some work with this subject, which I’ll save for a later post.)  The two-day final should be treated like a marathon compared to the usual format’s sprint, and strategic wagering in the first game could leave you with a strong platform to win, or leave you so far behind that it’ll take a miracle in the second game to come back.

Watson’s System

watson

“OK, you were just shown up on national TV by the monolith from 2001 … Smile!”

I was reading the Journal of Artificial Intelligence Research one night (as you do), and came upon a very interesting article written by part of the team that created Watson for IBM.  In it, they discussed some of the various challenges they faced in building an AI capable of the strategy of playing Jeopardy!. To test out strategies, they built a game simulator where Watson would take on two “average” Jeopardy! contestants. They measured a contestant’s ability using two numbers, the percentage of clues that they attempt to buzz in on (called buzzing percentage), and the percentage of clues that they respond correctly to having buzzed in (called precision).  Using data obtained from the Jeopardy! Archive, they determined that the average contestant attempts to ring in on 61% of clues, and answers 87% correctly.  I wondered if I could use this same methodology to evaluate individual contestants’ skill as well.

Precision was easy enough to determine, just by observation.  Buzzing percentage, however, is only an estimation.  We have no way of knowing for sure how many clues a contestant buzzes in on. Using the Watson team’s methodology, we can make an estimate based upon how many clues a contestant successfully buzzes in on compared to their opponents and the number of “triple stumpers”, clues that nobody attempts.

Now, I had an issue with their buzzing model, and it might be one that you’ve noticed as well.  Anybody who’s been on Jeopardy! or played a similar game knows that knowing the correct response is only half the battle. There’s also the dreaded lockout system that contestants have to conquer. Ever since the second season of the revived series in 1985, contestants have been forced to wait until the clue is read in its entirety by Alex before buzzing in.  If they buzz in before the buzzer is active (represented by a set of lights around the playing board, not shown to the viewers), they are locked-out of buzzing for two tenths of a second.  If they try to buzz in again during those two-tenths of a second, the lock-out resets.  If you have ever seen a contestant frantically trying to ring in but nothing is happening, even if nobody else has rung in, it’s because they’ve fallen prey to the lock-out.

People have spoken of buzzer skill as a major part of a Jeopardy player’s ability.  Some players manage to get in a rhythm with Alex’s cadence and seem to control the buzzer better than their opponents.  The Watson team chooses to ignores that ability.  It instead argues that a person who successfully buzzes in more just attempts to buzz in on more clues.  It assumes that everybody has the same amount of “buzzer skill”; if more than one person attempts to buzz in, the successful contestant is simply a matter of chance.

To test this out, I built a program that took these two contestant metrics and simulated a game, using Eric Feder’s work on Jeopardy Win Expectancy as a starting point.  When I ran simulations against the results of the last couple of tournaments, I was pleasantly surprised to see the simulation reasonably matching the results.  Despite my misgivings, I feel comfortable moving forward with predictions on this year’s tournament.

Quarterfinal Matchups

Let’s take a look at the first five games. (Thanks to The Final Wager for revealing the matchups early, and thanks to Buzzerblog for the use of the contestant images.)

toc1

Monday’s contestants, and their estimated win percentages.

The first match will be a contrast in styles.  Ben Ingram and Rebecca Rider both made their bones by being selective about their clue selection – they sport the two highest precision scores of the fifteen participants.  John Pearson, on the other hand, buzzed in nearly 10% more often than either of his opponents, but wound up giving the wrong answer almost 10% more often as well.  The system give a slight edge to Rider in this matchup, but Pearson should be selecting clues more often, giving him a higher chance of finding a well-timed Daily Double.

toc2

Tuesday’s contestants, and their estimated win percentages.

When I first calculated these statistics, I had to double-check Andrew Moore’s scores by hand to ensure their accuracy.  I couldn’t believe that a fairly anonymous 6-time champ in a season full of memorable winners could have a buzzing percentage almost 5% higher than any of the other competitors, but the numbers checked out.  While his precision is below average among tournament participants, it’s still good enough to give him the edge in this matchup, as well as in the overall tournament.  However, he could find trouble playing against Arthur Chu, who the system is also keen on.  Unfortunately, the tough matchup combined with her weak buzzing percentage does not bode well for Rani Peffer’s tournament chances.

toc3

Ednesdayway’s ontestantscay, anday eirthay estimateday inway ercentagespay.

20-game winner Julia Collins is given a slight edge against Joshua Brakhage and Jim Coury, although that’s more down to the quality of the competition than her own skill. Despite coming into the tournament as the favorite or co-favorite with Arthur Chu, the system sees Collins as merely an average ToC-quality player. If she were to play in either Tuesday’s or Friday’s game, she might have had a rough time making it to the second week.  Incidentally, all three players are fairly conservative on the buzzer – expect a good number of Triple Stumpers.

toc4

percentages win estimated their and, contestants Thursday’s.

This matchup threatens to play out much like Wednesday’s, with three conservative buzzers, none of which have shown extremely strong game play, pitted against each other.  Expect another game where a number of clues pass by unanswered.  Sarah McNitt has the edge over Drew Horwood and Terry O’Shea in both buzzing and precision, and as such is given the favorite tag.

toc5

Friday’s … you know the drill by now.

The week looks to end on a bang, with two strong competitors battling it out.  Mark Japinga may have gotten a bit lucky to even be invited back to take part in the tournament, but now that he’s here he’s a serious threat to win the whole thing.  Jared Hall is another strong player looking to go far.  Sandie Baker is probably drew the worst game to play out of all the contestants – if she were playing on Wednesday or Thursday she might have been a favorite to win.  Instead, she could get squeezed out early.

The Odds

Overall, I see this as a fairly wide open field.  Thirteen people have a better than even money chance of making it to the semifinals, and the favorite’s chances of winning is only about 1 in 5.  This could very well be the most exciting, unpredictable tournament in recent history.

toc-odds

Note: There are two ways of making the semifinals, by winning your quarterfinal or qualifying as a wild card. We model the chances of each event happening (the percentages in gray), and sum the two together to determine the total chance that a contestant makes the semifinals.

Check back with us after every game next week for analysis and updates.