Tournament of Champions

Trivia Christmas is coming a bit early this year. Instead of the usual 18 month wait between events, we’re getting another installment of the Jeopardy Tournament of Champions a mere twelve months since the last incarnation. It’s always fun to try to predict how these things will go, so we’ll see if we can better our efforts from last year, where we had eventual winner Ben Ingram rated as the fifth most likely winner in a fairly wide-open field. This year, however, the field has broken down into three distinct groups.  The field is led by two co-favorites, each of whom have a one-in-five chance of victory.  There are three dark horse candidates for the title, with chances of victory ranging from 11% to 14%. Finally, we have the remaining ten contestants, none of whom have more than a 1-in-25 chance of taking home the title.

Before we reveal our predictions, let’s do a quick recap of our methodology. We analyse players’ ability using two metrics. The first is Buzzer Percentage, which attempts to measure how often a contestant will buzz in. Since we can’t measure this directly, we use a Monte Carlo simulation to estimate it based on how often a player has successfully rung in during their previous games. The second metric is Precision, which is the percentage of times that they are correct on clues where they buzzed in (ignoring Final Jeopardy & Daily Doubles). For reference’s sake, the average Jeopardy contestant has a buzzer percentage around 60% and a precision around 87%.  This system is based on the system that the designers of Watson used to simulate human opponents for Watson to practice against.

Once we have those numbers, we run a bunch of simulated games using each player’s Buzzer Percentage and Precision, and determine how likely they are to win their quarterfinal games. Using the results of these simulations, we track how likely a contestant is to win their quarterfinal or qualify for a wild card, and use that to weight a player’s potential matchups in the semifinals and finals.  Here’s what our simulation gives us for each of this week’s quarterfinal games:

If you want a dark horse candidate for your those office Jeopardy pools I know you’re all in, Catherine Hardee is a decent candidate. Despite winning one game fewer than Dan Feitel and Vaughn Winchell, her strong Buzzer Percentage and respectable precision gives her a slight edge in Monday’s matchup.

Two of the three dark horse candidates face off on Tuesday’s show. Greg Seroka and Kristin Sausville both have a decent chance of taking down the whole tournament. Brennan Bushee is a bit unfortunate in his draw; despite his strong precision he could get squeezed out by two of the tournament’s strongest buzzers.

96.8% precision. Ninety-six point eight percent precision. That is a crazy, insane, whacked-out number, and helps make Matt Jackson, unsurprisingly, one of the favorites for the tournament. Jackson draws a fairly easy group for his first game. John Schultz’s metrics are among the weakest in the field, and are actually comparable to an average contestant. Jennifer Giles could take an upset win, and in doing so become only the second Teacher’s Tournament winner to make the second week.  The first, of course, being Colby Burnett, who won the whole thing in 2013.

There’s no nice way to put this: the lineup for Thursday’s match is by far the weakest. All three contestants are very conservative buzzers; expect a higher than average number of clues going untried. Kerry Greene is at least precise when she buzzes in, which gives her quite an edge over the error-prone Andrew Haringer and Elliot Yates.  Any way you slice it, this game will give an automatic semifinal berth to a player who may have had a hard time qualifying in other games.

Alex Jacob has the busiest buzzer finger in the entire tournament. When paired with his excellent precision, it makes him the co-favorite to win the entire tournament. Scott Lord and Michael Bilow will have a hard time matching up.

Alex Jacob and Matt Jackson are so close that it’s fair to call them co-favorites to win the tournament.  Assuming that they manage to avoid each other in the semifinals (which is what caused most of their eliminations in our simulations), they should get to lock horns in a great two-day final.  Chances are they’ll meet one of Greg Seroka, Kristin Sausville, or Catherine Hardee there.  At that point, the only winners I’d be willing to predict for certain would be the audience, who would get to watch an exhibition in excellent Jeopardy gameplay.

The Stats

You gotta hand it to Arthur Chu.  Despite being handed the toughest draw in the field, he’s making this look easy.

Mark Japinga’s negging and Rider being crowded out on the buzzer could have made this a runaway, but Rider doubled up on a good Daily Double, and Japinga made a late run to remain relevant.

Daily Doubles

I’ll admit – as soon as Chu wagered $1,000 on his first Daily Double, my bad-bet-o-meter went crazy.  He found it early in the Jeopardy round, having called for the $600 and $800 clues in random categories looking for it as is his M.O.  He was trailing Japinga slightly at the time, $2,600 to $2,800, with Rider yet to open her mouth. I loaded up my still-in-development Daily Double evaluator, and fed it the parameters.  Here’s what it spat out at me:

The five lines represent how comfortable you are with the category.  They have nothing to do with the expected difficulty of the clue, which has already been accounted for.  The table may be hard to read, but it suggests 3 wagers, depending on your level of confidence:

– $300-400 if you are not confident. Enough to take over the lead, but still enough to stay in touching distance if you miss it.

– $2,000 if you’re confidence is average. Beyond a certain point when you’re leading, every dollar you have is gives you a smaller chance of winning than the previous dollar. At this point, the diminishing returns of increasing your score catch up with the chances that you’ll actually get the clue right.

–  $2,400 if you’re supremely confident.  The system suggests the diminishing returns beyond that point are not worth an all-in wager, but I certainly wouldn’t begrudge one if that was your choice.

Chu’s selection of $1,000 is an odd one. It’s not conservative enough if you don’t like the category (which I imagine was Chu’s reasoning), as it leaves you at least two clues behind to catch up if you’re wrong. It’s also not aggressive enough to take advantage of the opportunity that a Daily Doubles represents.

Chu hit the first Daily Double in Double Jeopardy, and again wagered $1,000.  The situation was much different this time, with Chu on a commanding $13,200 over Japinga’s $3,600 and Rider’s $1,800.  Without looking too deep at the details this time, I think the two choices in wagers would be the minimum of $5 if you want to protect your lead, and a more aggressive wager of $6,000, looking to close out the game here but still keep twice Japinga’s score if you’re wrong.  I don’t begrudge Chu his wager that much, since I doubt there was any great difference between wagers of $5 and $1,000. And despite the disclaimer that this could be the stupidest thing she ever did, Rider’s true Daily Double later with her score of $4,600 a long way back of Chu’s $16,600 was the only real move she could have made, especially in a category that she seemed to like.  She answered correctly, and Japinga made a late run to set up a Final Jeopardy where all three players could still take the victory.

Final Jeopardy

The scores were Chu with $17,800, Rider with $12,800, and Japinga with $8,000.  I’ll leave the exact analysis of the situation to Keith at The Final Wager, who does a better job at it than I could ever hope to. But I do want to talk about something that I feel very strongly about which comes into play here: Stratton’s Dilemma.

Coined by Andy Saunders, Stratton’s Dilemma is the term for the situation Rider finds herself in, where she has to choose between two possible wagers. Let’s analyze her situation. We can assume that Chu will wager $7,801 or thereabouts, the amount needed to guarantee his victory as long as he gets Final Jeopardy correct. We must assume that Chu responds incorrectly, otherwise we have no chance of winning.  If Chu is incorrect, he will be left with $9,999. Thus, it is in our best interest to wager no more than $2,800, staying ahead of Chu no matter what our outcome is. However, the presence of Japinga complicates matters.  If Japinga wagers everything and doubles up, he’ll have $16,000.  To ensure that we remain ahead of Japinga in that situation, we have to bet at least $3,201.  Sometimes we can find a wager to satisfy both scenarios, but this is not one of those times. We have to choose one or the other, and know that some of the time we will choose incorrectly and lose when we could have won.  It’s an infuriating position. Which bet should we choose?

Well, let’s break down the scenarios.  Since each player can either be correct or incorrect in Final Jeopardy, there are 8 possibilities to consider.  Assuming that Chu wagers $7,801, and Japinga wagers his entire $8,000 (our worst case scenario), what happens when we bet $0?Compare that to what happens when we bet everything:

We’ll always win in either case if we’re the only person to respond correctly. When we bet small, we will win if everybody answers incorrectly.  If we bet big, we will win if the leader misses and both we and the player in third respond correctly.  Which is more likely to happen?  If only we had a large repository of previously played Jeopardy games to look at…

I took a look at 2,184 games played over the last 10 years where all three players made it to Final Jeopardy.  Specifically, I counted the times when each of the above scenarios happened.  I separated the players by their ranking going into Final Jeopardy, so I know how often the leader missed Final Jeopardy while the other two got it right, or only the second-ranked player responded correctly, for example. And what were the results?

Unsurprisingly, the scenarios when all three players responded correctly and all three players responded incorrectly are more likely than any other case.  Players possess a shared knowledge base, and a question that one person knows is likely to also be known by the others, and vice versa.  The situation we’re hopping happens when we bet big, where 2nd and 3rd place answers correctly but the leader does not, is the least likely to happen. If we quantify our chances of winning using the above table, we have a 30.9% chance of victory with a small bet, while we only have a 19.1% chance of success if we wager big.  I don’t know about you, but if one choice of a dilemma increases my chances by over 60%, I wouldn’t call it much of a dilemma.

The Odds

Well, we’re going to get the matchup we wanted, Collins vs. Chu. And frankly, this tournament is now Chu’s to lose. Regardless of who wins the next match between Ben Ingram, Joshua Brakhage, and Sandie Baker, I can’t help but see Chu as a strong favorite, especially over a two-day affair where the normally high level of variance inherent in a Jeopardy match is lowered.

 

Sorry that I never got an update out on Friday’s match.  Jeopardy was preempted here, so I didn’t get to watch the match until late Saturday.  Quite a good match, mind, as all three players played well and wagered well, and all three were rewarded with advancement.

The only player who played close to expectations was Mark Japinga.  Jared Hall played conservatively, but never made a misstep (outside of 1 Daily Double), and took the win after Japinga wagered for a Wild Card spot in Final Jeopardy.  Sandie Baker didn’t have the best game either, but still had enough money to bet big in Final and take the last Wild Card spot.

So, we have our nine players.  Stepping back to evaluate the performance of my system so far, I’d have to give the system’s predictive power a rating of B- so far.  It whiffed on Andrew Moore being the favorite, but on the other hand of the five players least likely to advance, four of them indeed failed to advance. On an individual game basis, the middle ranked player won all five games, although to the system’s credit in two of those games the favorite was leading heading into Final.

Looking forward, the picture is much more clear.  We know what the three semifinal matchups are going to be (thanks again to The Final Wager for releasing them on Friday), and the picture of what our three finalists are going to be is getting clearer.  As a result, there have been some massive shakeups in our odds.  Let’s look at each matchup in detail.

Monday’s Game

(Our buzzing and precision scores have been recalculated to include each player’s quarterfinal performance)

After looking at the draw, I’m pretty sure that the semifinal matches are seeded based on each player’s quarterfinal score. The top three winners go into one group, the other two winners and the best wild card go into the second group, and the remaining wild cards go into the final group.  Each game is made up of one player in each group.  Terry O’Shea’s Thursday score was the third best score among the winners, and I bet Jared Hall and Julia Collins are thankful for that fact.  In another universe, they could be playing against one of the other top seeds: Arthur Chu or Ben Ingram.  Instead, Hall and Collins will likely battle it out among themselves for the finalist spot.  Hall’s stronger buzzing percentage gives him the edge over Collins’ better precision.

If Arthur Chu ends up winning the tournament, nobody could ever accuse him of getting an easy draw.  After taking down Andrew Moore in his quarterfinal match, he now faces off against Mark Japinga, our system’s current favorite. Rebecca Rider rounds out this trio – she’ll have to be at the top of her game to get past Chu and Japinga.

Wow.  Just … wow.  After factoring in each player’s performance in the quarters, we have a situation where each player’s buzzing percentage is practically the same.  Ben Ingram has the edge in the system thanks to his superior precision, but really this one could be anybody’s game.  It would not surprise me to see Joshua Brakhage or Sandie Baker advance.  Expect this game to be close going into Final Jeopardy.

The Odds

With the semifinal matches now set, our odds now look quite different:

 

Mark Japinga now takes up the mantle of favorite.  If Arthur Chu beats him on Tuesday like he did to our previous favorite, I would be very surprised if he didn’t go on to win the tournament.  Ben Ingram moves up to #2 despite Wednesday’s game being little more than a crapshoot.  Monday’s winner will probably end up being the underdog going into the final games; the system only sees one of those three winning one time in four.

No matter who advances, expect some excellent knowledge and strategic game playing on display.  Check back all this week for updates.

When I suggested in my preview that you could “expect [a] game where a number of clues pass by unanswered”, I didn’t quite mean it to this extent.

The Stats

Whether it was the material, the players, or the lunch break that would have preceded this taping, the game just never got off the ground. All three players underperformed on their buzzing, and only Terry O’Shea hit her previous level of precision. Thirteen out of the 57 non-Daily Double clues passed by our competitors without a single buzz, not to mention several more that only elicited incorrect responses.  It was a bad day at the office for all parties involved, and unfortunately it cost two of the players their shot at the title.

The Strategy

Normally I’d break down the strategic choices made by the players in this space, but there’s not much to say.  All three Daily Doubles were found by the player in distant last place, all three times the player correctly wagered big, all three times the wager was lost. Drew Horwood finished the match in the red, and didn’t play Final Jeopardy, while O’Shea had a slim lead of $8,800 to Sarah McNitt’s $8,600.  Both bet big, since you’d think their scores wouldn’t be worth a wild card, although this tournament is shaping up to have a low cutoff (again, they wouldn’t know that at the time).  O’Shea bet to lock-out McNitt, and was correct, earning the automatic advancement.  McNitt bet as big as she could, but still kept enough back to win on a Triple (Double?) Stumper, but missed Final Jeopardy.  Her final total of $500 is already known to be not enough to advance.

The Odds

On a more positive note, with Horwood and McNitt dropping out, we can now welcome Rebecca Rider and Julia Collins into the semifinal fold.  Rani Peffer’s $7,599 is looking pretty good at this point. and Jim Coury’s $5,600 still has a shot of holding up.  We’ll see what happens after the dust settles on tomorrow’s last quarterfinal.

Jeopardy tournaments are a harsh mistress.  One bad game, and you’re out on your backside.

Tonight’s Stats

 

Arthur Chu’s performance was nothing less than I expected from him coming into the game.  It was Andrew Moore’s underachievement (and to a lesser extent Rani Peffer as well) that turned the game into a blowout for Chu.

Daily Double

My gut instinct on Daily Doubles in the Jeopardy! round is to always bet aggressively.  When Moore found the Daily Double after he and Chu went on a wild goose chase for it, most of the high-dollar clues were off the board and the two of them were tied at $2,600, with Peffer trailing at $600.  Moore opted to bet only $1,000, and after analyzing the situation, I’m inclined to agree with him.  Given that the DD was hidden in the $1,000 space (indicating a harder clue), and that most of the money had been stripped from the board, a modicum of restraint was to be called for.

However, the situation was different when Moore found the first DD in the Double Jeopardy round early on.  He had a slight lead at the moment, $5,600 to Chu’s $5,200 and Peffer’s $2,200.  It’s a close decision, but this would have been the time to make a move.  A wager of all or almost all of his money would have given him the opportunity to take a stranglehold on the game with only one DD left on the board.  He instead chose to wager $2,000, and perhaps this was the right move, since he missed the clue.  After that stumble, Chu went on a hell of a run, and by the time he found the 2nd Daily Double late in the round, he had an unassailable lead.

Final Jeopardy

Not much to say here.  Chu had a lock on the game, and could sit this one out.  Moore and Peffer knew their current scores wouldn’t hack it, so each bet everything but $1.  Moore missed, and has gone from our favorite to all but out in the blink of an eye.  Peffer nearly doubled up, and takes the #2 spot on the wild card list for the nonce with $7,599.

Updated Odds

With our favorite knocked out of contention, there’s no surprises who took over at the top of the list.

 

A bit of change elsewhere on the list.  With Moore no longer a potential hurdle, many people saw their chances of getting to the finals and winning increase.  Two wild card scores coming in under expectations increases the chances of the people yet to play of taking a wild card.  Rebecca Rider’s score in the clubhouse of $11,600 is looking better; she’s even money to advance now.

Arthur Chu has dropped the gauntlet.  Will Julia Collins pick it up tomorrow and keep the chances of everybody’s dream matchup alive? Check back tomorrow for the analysis.

 

This Monday kicks off the 24th Jeopardy! Tournament of Champions, a two week sesquiennial bacchanalia of trivia where one lucky person gets to inscribe their name on the annals of Jeopardy! history.  It’s the closest thing to an end-of-season playoff that we have in the American game shows, and in the spirit of March Madness I thought I’d put on my Nate Silver hat and take a shot at predicting who the winners would be.  Let’s just ignore that in this case I’d be “predicting” the results of an event that was actually filmed a couple of months ago.

The Rules

The tournament format, which was supposedly created by Alex Trebek himself, sees fifteen previous champions come back to play over two weeks.  The fifteen participants are the winners of qualifying tournaments (currently the College Championship and the Teachers Tournament, previously winners of the Teen Tournament and the Seniors Tournament also got invites), and then enough champions who played since the last Tournament to fill in the remaining slots, ordered by number of games won, then amount of money won.  The cut-off this year was Mark Japinga, who won 4 games and $112,600. (One 6-game champion was not invited back due to legal issues.)  The first week sees the fifteen contestants placed semi-randomly (I’m pretty sure the contestants are seeded so that each game has one of the top five players, one of the middle five players, and one of the bottom five players) into five quarterfinal games.  The five winners of these games are guaranteed passage to the next round, along with the four highest-scoring non-winners as wild cards.  These nine players are again randomly drawn into three semifinal games, with the stipulation that you cannot play against someone you already played against in the first round.  The three semifinal winners then face off in a two-day final, where the cumulative amount won on both days is used to determine the tournament winner.

It’s an interesting format because proper play requires a shift in strategy.  Only the semifinal game is a traditional winner-take-all game of Jeopardy.  In the quarterfinal, you don’t necessarily need to win the game, just have enough money at the end of the game to be one of the top four non-winners. This leads to some conservative wagering on Daily Doubles and in Final Jeopardy.  The catch is that you are sequestered before you play your quarterfinal, so you do not necessarily know how much money would qualify you for the semifinal. There have been years where a score of $20,000 would have sent you home, and one year where multiple people finished with $0 but still made the semifinals.  (I’ve done some work with this subject, which I’ll save for a later post.)  The two-day final should be treated like a marathon compared to the usual format’s sprint, and strategic wagering in the first game could leave you with a strong platform to win, or leave you so far behind that it’ll take a miracle in the second game to come back.

Watson’s System

I was reading the Journal of Artificial Intelligence Research one night (as you do), and came upon a very interesting article written by part of the team that created Watson for IBM.  In it, they discussed some of the various challenges they faced in building an AI capable of the strategy of playing Jeopardy!. To test out strategies, they built a game simulator where Watson would take on two “average” Jeopardy! contestants. They measured a contestant’s ability using two numbers, the percentage of clues that they attempt to buzz in on (called buzzing percentage), and the percentage of clues that they respond correctly to having buzzed in (called precision).  Using data obtained from the Jeopardy! Archive, they determined that the average contestant attempts to ring in on 61% of clues, and answers 87% correctly.  I wondered if I could use this same methodology to evaluate individual contestants’ skill as well.

Precision was easy enough to determine, just by observation.  Buzzing percentage, however, is only an estimation.  We have no way of knowing for sure how many clues a contestant buzzes in on. Using the Watson team’s methodology, we can make an estimate based upon how many clues a contestant successfully buzzes in on compared to their opponents and the number of “triple stumpers”, clues that nobody attempts.

Now, I had an issue with their buzzing model, and it might be one that you’ve noticed as well.  Anybody who’s been on Jeopardy! or played a similar game knows that knowing the correct response is only half the battle. There’s also the dreaded lockout system that contestants have to conquer. Ever since the second season of the revived series in 1985, contestants have been forced to wait until the clue is read in its entirety by Alex before buzzing in.  If they buzz in before the buzzer is active (represented by a set of lights around the playing board, not shown to the viewers), they are locked-out of buzzing for two tenths of a second.  If they try to buzz in again during those two-tenths of a second, the lock-out resets.  If you have ever seen a contestant frantically trying to ring in but nothing is happening, even if nobody else has rung in, it’s because they’ve fallen prey to the lock-out.

People have spoken of buzzer skill as a major part of a Jeopardy player’s ability.  Some players manage to get in a rhythm with Alex’s cadence and seem to control the buzzer better than their opponents.  The Watson team chooses to ignores that ability.  It instead argues that a person who successfully buzzes in more just attempts to buzz in on more clues.  It assumes that everybody has the same amount of “buzzer skill”; if more than one person attempts to buzz in, the successful contestant is simply a matter of chance.

To test this out, I built a program that took these two contestant metrics and simulated a game, using Eric Feder’s work on Jeopardy Win Expectancy as a starting point.  When I ran simulations against the results of the last couple of tournaments, I was pleasantly surprised to see the simulation reasonably matching the results.  Despite my misgivings, I feel comfortable moving forward with predictions on this year’s tournament.

Quarterfinal Matchups

Let’s take a look at the first five games. (Thanks to The Final Wager for revealing the matchups early, and thanks to Buzzerblog for the use of the contestant images.)

The first match will be a contrast in styles.  Ben Ingram and Rebecca Rider both made their bones by being selective about their clue selection – they sport the two highest precision scores of the fifteen participants.  John Pearson, on the other hand, buzzed in nearly 10% more often than either of his opponents, but wound up giving the wrong answer almost 10% more often as well.  The system give a slight edge to Rider in this matchup, but Pearson should be selecting clues more often, giving him a higher chance of finding a well-timed Daily Double.

When I first calculated these statistics, I had to double-check Andrew Moore’s scores by hand to ensure their accuracy.  I couldn’t believe that a fairly anonymous 6-time champ in a season full of memorable winners could have a buzzing percentage almost 5% higher than any of the other competitors, but the numbers checked out.  While his precision is below average among tournament participants, it’s still good enough to give him the edge in this matchup, as well as in the overall tournament.  However, he could find trouble playing against Arthur Chu, who the system is also keen on.  Unfortunately, the tough matchup combined with her weak buzzing percentage does not bode well for Rani Peffer’s tournament chances.

20-game winner Julia Collins is given a slight edge against Joshua Brakhage and Jim Coury, although that’s more down to the quality of the competition than her own skill. Despite coming into the tournament as the favorite or co-favorite with Arthur Chu, the system sees Collins as merely an average ToC-quality player. If she were to play in either Tuesday’s or Friday’s game, she might have had a rough time making it to the second week.  Incidentally, all three players are fairly conservative on the buzzer – expect a good number of Triple Stumpers.

This matchup threatens to play out much like Wednesday’s, with three conservative buzzers, none of which have shown extremely strong game play, pitted against each other.  Expect another game where a number of clues pass by unanswered.  Sarah McNitt has the edge over Drew Horwood and Terry O’Shea in both buzzing and precision, and as such is given the favorite tag.

The week looks to end on a bang, with two strong competitors battling it out.  Mark Japinga may have gotten a bit lucky to even be invited back to take part in the tournament, but now that he’s here he’s a serious threat to win the whole thing.  Jared Hall is another strong player looking to go far.  Sandie Baker is probably drew the worst game to play out of all the contestants – if she were playing on Wednesday or Thursday she might have been a favorite to win.  Instead, she could get squeezed out early.

The Odds

Overall, I see this as a fairly wide open field.  Thirteen people have a better than even money chance of making it to the semifinals, and the favorite’s chances of winning is only about 1 in 5.  This could very well be the most exciting, unpredictable tournament in recent history.

Check back with us after every game next week for analysis and updates.