The vast majority of human chess players need to look at the board to know where the pieces are. Only a few people can know where all the pieces are if you just give them a list of moves. Have you tried evaluations where you give the LLM a representation of the board state at every move, as most human players would have, and which all chess engines track?
My prompt is: "Let's play chess. Show me the board each time." Which gives something like:
a b c d e f g h
8 | r n b q k b n r | 8
7 | p p p p . . p p | 7
6 | . . . . . . . . | 6
5 | . . . . . . . . | 5
4 | . . . . P p . . | 4
3 | . . . . . . . . | 3
2 | P P P P . . P P | 2
1 | R N B Q K B N R | 1
a b c d e f g h
I suppose I could use an external representation and paste that in, but I could also have it write a python script to use stockfish.
Comments
The vast majority of human chess players need to look at the board to know where the pieces are. Only a few people can know where all the pieces are if you just give them a list of moves. Have you tried evaluations where you give the LLM a representation of the board state at every move, as most human players would have, and which all chess engines track?
My prompt is: "Let's play chess. Show me the board each time." Which gives something like:
I suppose I could use an external representation and paste that in, but I could also have it write a python script to use stockfish.If it's forgetting the board after showing you that each turn then I suppose that is a legitimate criticism.