LLM Chess: jev-latest

# LLM Chess: Benchmarking Reasoning and Instruction-Following in LLMs

[](https://maxim-saplin.github.io/llm_chess/)
[](https://arxiv.org/abs/2512.01992)

LLM Chess is a benchmark that evaluates Large Language Models (LLMs) on their reasoning and instruction-following abilities in an agentic setting. LLMs engage in multi-turn dialogs to play chess against opponents like a Random Player or the Komodo Dragon chess engine. This setup tests both strategic reasoning (chess skill) and protocol adherence (sustained interaction without errors).

Key insights from the benchmark:
- Early models (2024) struggled with basic instruction following, often hallucinating illegal moves or failing dialogs.
- Advanced reasoning models (e.g., o1, o3, o4-mini) in 2025 saturated random-based evaluations, prompting the addition of Dragon as a stronger opponent for Elo anchoring.
- Metrics separate chess skill (Win/Loss, Elo) from durability (Game Duration), revealing trade-offs in model capabilities.

See the [live leaderboard](https://maxim-saplin.github.io/llm_chess/) for rankings and the [NeurIPS FoRLM 2025 paper](docs/LLM%20CHESS%2C%20Benchmarking%20Reasoning%20and%20Instruction-Following%20in%20LLMs%20through%20Chess%20-%20NeurIPS%20FoRLM%202025.pdf) for full details.

<img width="2118" height="1582" alt="image" src="https://github.com/user-attachments/assets/4375a8a8-e226-4ed1-820f-86006d0404e2" />

## Installation and Setup

1. **Clone the repository**:
   ```
   git clone https://github.com/maxim-saplin/llm_chess.git
   cd llm_chess
   ```

2. **Create a virtual environment** (recommended):
   ```
   # Using uv (recommended)
   uv sync
   ```
   For the data-analysis notebooks, use `uv sync --group dev --extra analysis`.

3. **Install dependencies**:
   ```
   # Already handled by `uv