Can LLMs like ChatGPT Understand Chess?

Has anyone tried Chess champ by Google? It's powered by Gemini 3. I'm surprised this didn't talk about it, I once played it and it beat the shit out of me! I couldn't even make out of the opening alive, here is the game: https://gemini.google.com/share/d5924204ccef

There is a chance though that it was some classic opening trap, I'm saying this because I know my opening knowledge isn't really good , by the way I'm 1900-2000 in blitz so I think this is impressive.

do you know https://adjva4.dpdns.org/@/RuyLopez1000 i was asking Chat GPT to slove a puzzle of the day GPT said i cant

@yassine00 I asked Gemini 3 to analyze this position:

And it first suggested ...d5 (an illegal move), and then on deep think it suggested ...Ng4 a blunder.

1. ... Ng4! (The Precision Strike)
   This is the engine's favorite and the most "Deep Think" move.

Why? You are immediately challenging the e3 pawn and creating a "Fork" threat.

The Tactic: If White ignores it, you can potentially cause chaos on the light squares. If White plays h3 to kick you, they’ve just weakened their own king’s shelter.

So there is room for improvement.

@yassine00 said ^

Has anyone tried Chess champ by Google? It's powered by Gemini 3. I'm surprised this didn't talk about it, I once played it and it beat the shit out of me! I couldn't even make out of the opening alive, here is the game: https://gemini.google.com/share/d5924204ccef

There is a chance though that it was some classic opening trap, I'm saying this because I know my opening knowledge isn't really good , by the way I'm 1900-2000 in blitz so I think this is impressive.

Fun Game. Doesn't seem to be an opening trap since the game position in not in the database from move 10.

LLMs do better in the opening, since the positions are closer to their training data. So if you can get into a unfamiliar position then the LLM will get confused.

@Akavall said ^

@yassine00 I asked Gemini 3 to analyze this position:

https://adjva4.dpdns.org/vJFcsXIa/black#33

And it first suggested ...d5 (an illegal move), and then on deep think it suggested ...Ng4 a blunder.

1. ... Ng4! (The Precision Strike)
   This is the engine's favorite and the most "Deep Think" move.

Why? You are immediately challenging the e3 pawn and creating a "Fork" threat.

The Tactic: If White ignores it, you can potentially cause chaos on the light squares. If White plays h3 to kick you, they’ve just weakened their own king’s shelter.

So there is room for improvement.

Yeah normal raw Gemini sucks, it's Chess champ that is interesting, I think they're giving it some guidance like allowing it to see the list of legal moves and blocking it from playing illegal moves, which is fair since honestly.
Aside from this, I tried testing it the position you gave but couldn't, it says it can only play a game from the standard starting position, I think the chess tool it has access to is fairly limited and doesn't support loading a new position.

I think this Chess Champ is getting some assistance from a chess engine. In the game @yassine00 played against it, it was able to severely punish some opening inaccuracies. It is much easier for a LLM to rationalize a good engine move, compared to finding the move itself. So I think Chess Champ is cheating :).

The original article was more about using chess to evaluate the "intelligence" of several LLMs. Not about how to make a LLM play chess as good as possible. Even though the latter is much more interesting for us.

I asked Copilot to analyze some positions. It can decipher FEN notations accurately, and maybe list all legal moves. But even if it gets these thing right, it still cannot really make sense of positions. It is like watching a very smart human with zero chess experience analyze a position.

I asked Copilot:

Can you evaluate this chess position:
4k3/8/6r1/4P3/3K4/8/7R/8 w - - 1 2
Who is winning?

It gave this answer:

Position: 4k3/8/6r1/4P3/3K4/8/7R/8 w - - 1 2
Side to move: White
Short answer
With best play, the position is a draw.
Black has the classic Philidor (sixth‐rank) defense set up: the king is in front of the pawn (...Ke8), and the rook sits on the sixth rank (...Rg6), preventing White’s king from reaching the sixth rank and stopping the pawn from advancing safely.

Why it’s a draw (ideas and plan)
Defender’s setup (Philidor defense):

Keep the rook on the sixth rank (here, ...Rg6) while the pawn is on e5.

This prevents White’s king from stepping to e6/f6 and also prevents e6 because ...Rxe6 would be possible as long as the white king is not on d5/c5 to recapture the rook.

If White ever manages to push e6, the defender immediately switches to side checks (...Rg5+ etc.), because after e6 the e5 square is vacated and side checks along the 5th rank become available.
The black king stays in front of the pawn (...Ke7/...Ke8), avoiding rook trades and shuffling as needed.

Attacker’s goals:

Bring the king to e5 or d5 and the rook to an active file/rank (ideally behind the pawn) to try to build Lucena.
But in this exact setup, Black’s rook on the sixth rank plus side checks after e6 make progress impossible with best defense.

Illustrative line (showing the drawing method)
One natural try for White is to improve the king and probe with checks:

Kd5 Ke7
Rh7+ Ke8
e6 (finally pushing) Rg5+!

Now that the pawn has vacated e5, the rook can give side checks from the 5th rank:

Kd6 (or Ke4, etc.) Rg6+
Re7+ Kf8 (or similar)

Black keeps checking from the side/back and holds. White cannot shelter from the checks without losing the pawn or allowing the king to be driven back—this is standard Philidor technique.

Important: After 3. e6, 3...Rxe6?? would lose to 4. Kxe6, leaving a winning K+R vs K. That’s why the correct move is ...Rg5+!, switching to side checks immediately.

A common trap (White’s try; Black’s must-know reply)
White might try to force progress with checks on the 8th/7th ranks:

Rh8+ Kf7
Rh7+

Bad for Black: 2...Rg7?? runs into 3. Rxg7+ Kxg7, and the resulting king‐and‐pawn endgame (White: Kd4 & e5 vs Black: Kg7) is winning for White with correct play.
Correct for Black: 2...Ke6! (or ...Ke8 in some move-orders), avoiding rook trades and keeping the king in front of the pawn, while the rook stays poised to resume the sixth‐rank/side‐check defense.

Verdict

Outcome with best play: Draw.
Reason: Classic Philidor (sixth‐rank) setup—Black keeps the rook on the sixth until the pawn advances, then switches to side checks, and avoids rook trades that lead to a winning K+P vs K for White.

If you’d like, I can give you:

A move-by-move defensive blueprint for Black, or
The best winning attempts for White and where they fall short.

So (after a few minutes of thinking) it was able to figure out this position is a Philidor position, and therefore a draw. But then it blundered with its response after white pushed the pawn to 6th rank. It suggested Rg5+, while the move is Rg8 (checking from behind).

Quite interesting.

@Willem_D said ^

I think this Chess Champ is getting some assistance from a chess engine. In the game @yassine00 played against it, it was able to severely punish some opening inaccuracies. It is much easier for a LLM to rationalize a good engine move, compared to finding the move itself. So I think Chess Champ is cheating :).

The original article was more about using chess to evaluate the "intelligence" of several LLMs. Not about how to make a LLM play chess as good as possible. Even though the latter is much more interesting for us.

I asked Copilot to analyze some positions. It can decipher FEN notations accurately, and maybe list all legal moves. But even if it gets these thing right, it still cannot really make sense of positions. It is like watching a very smart human with zero chess experience analyze a position.

I also thought maybe it was cheating but I was like nah Google wouldn't do us that dirty, right?! then just to be sure I went now and talked to Chess champ and tried to get it to confess if it uses an engine lol, and lo and behold it did: https://gemini.google.com/share/5cf584ac13f1
it uses what it calls a chess engine to find moves, so it's literally just an interface to play with stockfish :))), I'm in shambles.

To be fair, it may have lied about that and maybe it doesn't understand that it is the model that generates the move but that doesn't make any sense

I also tried to play against the Gemini Chess champ: https://gemini.google.com/share/1e0b8e8d61e3

But to make things fair, I also cheated (I used my local Leela Chess Zero). I played my normal opening, but let LC0 take over on move 7 or so.

Even though my engine destroyed this Gemini Chess champ, I suspect it is still better than any human. But I have no way of knowing since I am not very good (around 1300 Rapid on chess.com), and not able to judge this.

What is interesting is the end of the game. I am up two rooks, but Gemini refuses to believe this. So even with engine assistance, it is still spewing nonsense.

I did some research on this topic and wrote a UCI proxy for OpenAI-compatible APIs. It can be used with a GUI like Cutechess. https://github.com/b1ek/uci-llm