Subject Matter Turing Test: Bughouse (4 player chess)

The Game

The game is Bughouse, a four player chess derivative. Two players comprise a team and each play a chess board against a single opponent from the other team, the standard chess rules for each board remain the same except a player can drop any piece their partner captures onto their own board in lieu of a standard chess move. A player cannot drop a piece in a place that would result in a check; however they can drop a piece to block a check.

Partners can communicate the need for a specific piece (for instance “I need a knight”) but cannot otherwise give advice on their partner’s game, and this communication is shared with the opponents. A player can also publically respond to a request with a yes or no, but cannot provide any explanation of that decision. This is intended to prevent collusion. All players can observe both boards at all times.

The game is timed with two standard chess clocks, one for each board, and is in lightning 0:30/0:10 where each side has 30 seconds plus 10 seconds for each move they make. The goal of each partnership is to achieve a checkmate on one of their boards before time runs out, the winner of the game is the team who achieves a checkmate or the partnership who has time remaining on both of their clocks when the same is not true of the opponent partnership.

The AI

The AI would need to do several things well to be competitive in bug house:

1.    Play standard chess soundly

2.    Take into account how taking (or losing) a piece effects it’s partner’s board

3.    Have a model for how it’s (human) partner plays and apply those predications

4.    Have a model for how it’s opponents play and apply those predications

While modern chess AI’s can regularly beat GM’s today, to this author’s knowledge there is no such equivalent in most derivative games such as bug house. The primary reason for this is that chess AI’s rely on massive move history databases and simply look and winning percentages, and in fact have very little understanding of an actual position.

This AI would have to be designed to think positionally, to understand the strengths and weaknesses of a move, and while this would likely by definition be inferior to the approach taken by today’s chess AI’s, it is critical for a game such as this.

It is the author’s opinion that while this AI would be a significant undertaking it is within reach given current technology, and could be developed quite rapidly with the right people on the project.

The Test

The Turing Test of this AI revolves around an online chess tournament where human players will be randomly partnered with a human or AI player and then play with them for a day. This will give the partnership time to grow so that the players, be it human or AI, can learn how to work with each other.

The tournament is structured as a three day event with each participant being paired with a partner for the entire day and then randomly assigned to another partner each subsequent day. 50% of the time the partner will be paired with a human and 50% of the time an AI. One third of the player pool will be pure human partnerships, one third mixed human-AI partnerships and one third pure AI partnerships.

The player information is completely anonymized, neither the partners nor the opponents know who or what the players other than themselves is, including the AI’s. At the beginning of each day the player will play 10 standard chess games, timed for lightning, against their partner to get a feel for how their partner plays.

During a tournament day, the partnership will play best of 3 matches against a large number of different partnerships, likely more than a hundred. After each round each player will have to fill out a survey indicating if they think their opponents are human or AI, and at the end of each day the players will have to fill out the same survey about their partner.

In addition to the players other people would be welcome to act as observers; as such they would randomly be assigned to a match and asked to fill out a survey indicating the quality of the match and their belief of whether each player is human or AI.

The Turing Test would be a success if there is there is substantial confusion when players are deciding between assigning the labels “human” and “machine” to the player.

Analysis

In order to be competitive in this game the AI must by definition be able to recognize and adapt to their partner’s, and less importantly their opponents, style of play. This type of behavior will make it impossible to determine the humanity of a player with any degree of accuracy since this will result in sub-optimal play from an analysis stand point while still remaining highly effective, characteristics previously not observed in chess AI’s.

Thus a competitive bug house AI player would pass the Turing Test described above, fooling humans into thinking that they are interacting with another human when in fact it is a machine. The fact that the test allows for its subjects to know that it is possibly interacting with a machine and can still produce a false negative only strengthens the veracity of this exercise.