# Elo Rating Leaderboard from Pairwise Comparisons

**Experiment 6.8**: Building Model Leaderboard from Pairwise Comparison Data

This project implements an Elo rating system from scratch to analyze model performance using Chatbot Arena's public voting data. The implementation demonstrates how the Bradley-Terry model extracts relative model capabilities from millions of pairwise comparison votes.

## Overview

The Elo rating system is a method for calculating the relative skill levels of players (or in this case, AI models) in zero-sum games. Originally developed for chess, it has been adapted to rank AI language models based on head-to-head comparisons from user votes.

### Key Features

- **High-performance implementation**: NumPy + Numba JIT + parallel processing for optimal speed
- **Real voting data analysis**: Uses actual Chatbot Arena voting data with millions of pairwise comparisons
- **Win rate prediction**: Calculates expected win probabilities between any two models
- **Historical tracking**: Builds time-series snapshots showing ranking evolution
- **Interactive visualizations**: Multiple visualization types including animated bar chart races
- **Scalable**: Efficiently handles 2GB datasets with hundreds of thousands of matches

## Mathematical Foundation

The Elo system is based on the Bradley-Terry model, which models the probability that model A beats model B as:

```
P(A beats B) = 1 / (1 + 10^((R_B - R_A) / 400))
```

After each match, ratings are updated using:

```
R_A_new = R_A + K * (S_A - E_A)
```

Where:
- `R_A` is the current rating of model A
- `K` is the learning rate (K-factor)
- `S_A` is the actual score (1 for win, 0 for loss, 0.5 for tie)
- `E_A` is the expected score (predicted win probability)

## Requirements

- **Disk Space**: At least 3GB free (2GB for data file, 1GB for processing)
- **RAM**: 4GB+ recommended for full dataset analysis
- **Internet**: Stable connection for ~2GB download
- **Python**: 3.8+

## Installation

```bash
cd projects/week6/elo-leaderboard
pip install -r requirements.txt
```

## Quick Start

The project implements **two ranking methods** following official Chatbot Arena:

### 1. Bradley-Terry Model (Default - Recommended)

```bash
python main.py
# or explicitly:
python main.py bradley-terry
```

**Use this for**: Official leaderboard, stable rankings, production use

**Key features**:
- ✅ Official Chatbot Arena method
- ✅ Uses sklearn LogisticRegression for Maximum Likelihood Estimation
- ✅ Order-independent (processes all matches simultaneously)
- ✅ Includes 95% confidence intervals via bootstrap (100 samples)
- ✅ More stable and reliable rankings

**Processing time**: ~2-3 minutes (including bootstrap)

### 2. Online Elo (K=4)

```bash
python main.py online-elo
```

**Use this for**: Understanding Elo mechanics, educational purposes, faster computation

**Key features**:
- ✅ K-factor = 4 (official value used by Chatbot Arena)
- ✅ Simple sequential rating updates
- ✅ Order-dependent (processes matches chronologically)
- ✅ Faster computation (~30 seconds)
- ⚠️ Less stable, can vary based on match order

### Method Comparison

| Feature | Bradley-Terry | Online Elo |
|---------|--------------|------------|
| **Stability** | High (MLE fit) | Medium (sequential) |
| **Order dependence** | None | High |
| **Confidence intervals** | Yes (bootstrap) | No |
| **Speed** | Slower (~3 min) | Faster (~30 sec) |
| **Official method** | ✅ Yes | For comparison only |
| **Recommended** | ✅ Production | Educational |

### What Both Methods Do

1. Download Chatbot Arena voting data (~2GB, 5-15 minutes depending on connection)
2. Apply official filters:
   - Anonymous votes only (blind evaluation)
   - Deduplication (removes top 0.1% redundant prompts)
3. Compute model ratings using selected method
4. Calculate predicted win rates between all model pairs
5. Generate visualizations:
   - `leaderboard.png` - Top 20 models ranked by rating
   - `rating_distribution.png` - Rating histogram and statistics
   - `win_rate_matrix.png` - Predicted win rates (top 30 models)

**Note**: The initial data download is ~2GB and may take several minutes. A progress bar shows download status.

### Quick Demo (Synthetic Data)

To quickly understand Elo mechanics without downloading 2GB:

```bash
python quickstart.py
```

This runs a small demo with synthetic matchups between GPT-4, Claude, Llama, and Gemini.

### Benchmark

To compare both methods:

```bash
python benchmark.py
```

This shows performance and accuracy differences between online Elo and Bradley-Terry approaches.

## Project Structure

```
elo-leaderboard/
├── main.py                     # Main analysis script
├── optimized_elo.py            # NumPy + Numba Elo rating system
├── parallel_processing.py      # Multi-core parallel processing utilities
├── data_loader.py              # Data download and preprocessing
├── leaderboard.py              # Leaderboard calculation and analysis
├── visualization.py            # Static and interactive visualizations
├── animation.py                # Animated bar chart race generator
├── benchmark.py                # Performance benchmark tool
├── quickstart.py               # Quick demo with synthetic data
├── elo_rating.py               # Reference implementation (for comparison)
├── test_elo.py                 # Unit tests
├── requirements.txt            # Python dependencies
└── README.md                   # This file
```

## Usage Examples

### Building Elo Leaderboard

```python
from optimized_elo import build_leaderboard_optimized

# Build Elo leaderboard from DataFrame
elo = build_leaderboard_optimized(
    df,  # DataFrame with columns: model_a, model_b, winner
    initial_rating=1000.0,
    k_factor=32.0,
    show_progress=True
)

# Get leaderboard
leaderboard = elo.get_leaderboard()
for rank, (model, rating, matches, wins) in enumerate(leaderboard[:10], 1):
    win_rate = wins / matches * 100 if matches > 0 else 0
    print(f"{rank}. {model}: {rating:.1f} ({matches} matches, {win_rate:.1f}% win rate)")
```

### Loading and Filtering Data

```python
from data_loader import load_arena_data, filter_data

# Load data
df = load_arena_data("arena_data.json")

# Filter for blind votes only (reduces bias)
df_filtered = filter_data(
    df,
    anony_only=True,      # Only anonymous votes
    language="English",    # Specific language
    min_turn=1            # Minimum conversation turn
)
```

### Building Historical Leaderboards

```python
from data_loader import get_time_slices
from leaderboard import build_historical_leaderboards, get_rating_history

# Create weekly time slices
time_slices = get_time_slices(df, interval='W')

# Build leaderboard for each time point
historical_leaderboards = build_historical_leaderboards(
    df, time_slices, initial_rating=1000.0, k_factor=32.0
)

# Get rating history DataFrame
history_df = get_rating_history(historical_leaderboards)
```

### Creating Visualizations

```python
from visualization import (
    plot_leaderboard,
    plot_win_rate_matrix,
    plot_rating_history,
    create_interactive_leaderboard
)

# Static leaderboard chart
plot_leaderboard(leaderboard, top_n=20, save_path="leaderboard.png")

# Win rate heatmap
win_rate_df = calculate_win_rate_matrix_from_data(df)
plot_win_rate_matrix(win_rate_df, top_n=15, save_path="matrix.png")

# Rating evolution
plot_rating_history(history_df, models=["gpt-4", "claude-v1"], 
                   save_path="history.png")

# Interactive chart
fig = create_interactive_leaderboard(history_df, top_n=15)
fig.write_html("interactive.html")
```

### Creating Animated Bar Chart Race

```python
from animation import create_simple_animation

# Generate animated HTML
animation_file = create_simple_animation(
    history_df,
    output_path="animation.html",
    top_n=15
)

# Open animation.html in browser to view
```

## Output Files

After running `main.py`, the following files are generated:

### Static Images (PNG)
- `leaderboard.png` - Current top 20 models ranked by Elo rating
- `rating_distribution.png` - Histogram and box plot of rating distribution
- `win_rate_matrix.png` - Heatmap showing pairwise win rates
- `rating_history.png` - Line chart showing rating evolution over time

### Interactive Visualizations (HTML)
- `interactive_rating_evolution.html` - Interactive chart with zoom/pan
- `interactive_rank_evolution.html` - Interactive rank tracking
- `leaderboard_animation.html` - **Animated bar chart race** showing ranking evolution

## Key Parameters

### Elo System Parameters (Online Elo Method)

- **initial_rating** (default: 1000.0): Starting rating for all models
- **k_factor** (default: 4.0): Learning rate controlling update magnitude
  - Official Chatbot Arena uses K=4 for stability
  - Higher K-factor (e.g., 32): More volatile, faster adaptation to new data
  - Lower K-factor (e.g., 4): More stable, less influenced by recent matches

### Bradley-Terry Parameters

- **SCALE** (400): Elo scale parameter - determines rating point interpretation
- **BASE** (10): Base for logistic function - standard for Elo calculations  
- **INIT_RATING** (1000): Initial rating for all models
- **bootstrap_rounds** (100): Number of bootstrap samples for confidence intervals

### Time Slice Intervals

For historical analysis, you can adjust the time granularity:
- `'D'` - Daily snapshots
- `'W'` - Weekly snapshots (recommended)
- `'M'` - Monthly snapshots

### Visualization Parameters

- **top_n**: Number of top models to display (10-20 recommended)
- **Animation speed**: Adjustable in the HTML interface (1x to 10x)

## Data Format

The Chatbot Arena data includes the following fields:

- `model_a`: Identifier for first model
- `model_b`: Identifier for second model
- `winner`: Match outcome ('model_a', 'model_b', or 'tie')
- `tstamp`: Unix timestamp of the vote
- `judge`: User who made the vote
- `turn`: Conversation turn number
- `anony`: Whether vote was anonymous/blind
- `language`: Language of the conversation

## Validation

The implementation validates the Elo predictions against empirical win rates:

```python
from leaderboard import compare_win_rates

# Compare predicted vs actual win rates
comparison = compare_win_rates(elo_system, empirical_win_rates)
mean_error = comparison['error'].mean()
print(f"Mean Absolute Error: {mean_error:.4f}")
```

A low MAE (< 0.05) indicates the Elo model fits the data well.

## Analysis Insights

The project helps identify:

1. **Current Rankings**: Which models are currently strongest
2. **Rating Trends**: How model performance evolves over time
3. **Breakthrough Moments**: When new models enter or shake up rankings
4. **Competitive Dynamics**: Which models are closely matched
5. **Long-term Trajectories**: Models in ascent vs. decline
6. **Rating Stability**: Volatility in model performance

## Performance Architecture

The implementation is designed for high performance on large datasets (2GB+).

### Core Optimizations

#### 1. **NumPy + Numba JIT Compilation**

Uses NumPy arrays and Numba's just-in-time compilation:
- **NumPy arrays** for O(1) integer indexing (vs O(n) dictionary lookups)
- **Numba JIT** compiles hot loops to machine code (50-100x speedup)
- **Pre-allocated arrays** eliminate dynamic memory allocation overhead
- **Integer indices** instead of string model names for cache-friendly access

#### 2. **Multi-Core Parallel Processing**

Parallelizes independent operations across all CPU cores:
- **Historical analysis**: Each time slice processed independently
- **Win rate matrices**: Model pairs computed in parallel chunks
- **Data filtering**: DataFrame operations distributed across cores

```python
from parallel_processing import build_historical_leaderboards_parallel

# Automatically uses all available CPU cores
historical_lb = build_historical_leaderboards_parallel(
    df, time_slices, n_jobs=-1
)
```

#### 3. **Memory Optimization**

Reduces memory footprint through intelligent data types:
- Downcasts numeric types (int64 → int32, float64 → float32)
- Converts repetitive strings to categorical types
- Achieves 30-50% memory reduction

```python
from parallel_processing import optimize_dataframe

df = optimize_dataframe(df)  # Automatic memory optimization
```

### Performance Characteristics

On typical hardware (4-8 core CPU) with the full 2GB dataset:

| Component | Technique | Impact |
|---|---|---|
| Elo Computation | NumPy + Numba JIT | 50-100x faster |
| Historical Analysis | Multi-core parallel | 4-8x faster |
| Win Rate Matrix | Parallel processing | 4-8x faster |
| Memory Usage | Type optimization | 30-50% reduction |
| **Overall** | **Combined** | **~10-15x speedup** |

**Processing time**: 1-2 minutes for full dataset (hundreds of thousands of matches)

## Advanced Usage

### Custom Analysis

The modular design allows for flexible customization:

### Focused Analysis

```python
from optimized_elo import build_leaderboard_optimized
from data_loader import load_arena_data, filter_data

df = load_arena_data("arena_data.json")

# Analyze only recent data
df_recent = filter_data(df, min_date="2024-01-01")
elo_recent = build_leaderboard_optimized(df_recent)

# Analyze specific model family
gpt_models = [m for m in df['model_a'].unique() if 'gpt' in m.lower()]
df_gpt = df[df['model_a'].isin(gpt_models) & df['model_b'].isin(gpt_models)]
elo_gpt = build_leaderboard_optimized(df_gpt)
```

### Export Results

```python
import pandas as pd

# Export leaderboard to CSV
lb_df = pd.DataFrame(leaderboard, columns=['model', 'rating', 'matches', 'wins'])
lb_df.to_csv('leaderboard.csv', index=False)

# Export rating history
history_df.to_csv('rating_history.csv', index=False)
```

## Troubleshooting

### Data Download Issues

If automatic download fails:
1. Manually download from: https://storage.googleapis.com/arena_external_data/public/clean_battle_20240814_public.json
2. Save as `arena_data.json` in the project directory
3. Run `python main.py` again

### Memory Issues

The dataset is large (~2GB, hundreds of thousands of battles). If you encounter memory issues on systems with limited RAM:

```python
from data_loader import load_arena_data, filter_data, get_time_slices
from optimized_elo import build_leaderboard_optimized

# Load and immediately filter to reduce memory usage
df = load_arena_data("arena_data.json")

# Filter to recent data only
df_filtered = filter_data(df, min_date="2024-01-01", anony_only=True)

# Use monthly instead of weekly intervals for historical analysis
time_slices = get_time_slices(df_filtered, interval='M')  # vs 'W' for weekly

# Analyze with smaller top_n for visualizations
elo = build_leaderboard_optimized(df_filtered)
```

The built-in memory optimization reduces footprint by 30-50%, but very large analyses may still require 4GB+ RAM.

### Visualization Issues

- Ensure matplotlib, seaborn, and plotly are installed: `pip install -r requirements.txt`
- For HTML animations, use a modern web browser (Chrome, Firefox, Safari, Edge)
- If plots don't display in Jupyter, use `%matplotlib inline` or save to file

## References

- **Chatbot Arena**: https://chat.lmsys.org/
- **Elo Rating System**: https://en.wikipedia.org/wiki/Elo_rating_system
- **Bradley-Terry Model**: https://en.wikipedia.org/wiki/Bradley–Terry_model
- **LMSYS Paper**: "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena"

## Learning Objectives

This experiment demonstrates:

1. **Statistical Modeling**: How pairwise comparisons reveal relative abilities
2. **Online Learning**: Incremental rating updates as new data arrives
3. **Probabilistic Prediction**: Converting rating differences to win probabilities
4. **Data Visualization**: Effective techniques for showing temporal dynamics
5. **Model Evaluation**: Alternative to traditional benchmark approaches

## Technical Details

### Why Elo Computation is Hard to Parallelize

Elo rating computation is **inherently sequential** because each match's rating update depends on the current ratings, which were modified by all previous matches. This is why we can't simply split matches into chunks and process them independently.

However, we can still achieve significant speedups through:

1. **Algorithmic optimization**: NumPy arrays + Numba JIT
2. **Parallelizing independent operations**: Historical analysis, win rate matrices
3. **Memory efficiency**: Better cache utilization
4. **Data structure optimization**: Integer indexing, pre-allocation

### Numba JIT Compilation

The core Elo update loop is compiled to machine code using Numba:

```python
@jit(nopython=True)
def process_elo_updates_vectorized(ratings, model_a_indices, model_b_indices, 
                                   outcomes, k_factor, match_counts, win_counts):
    for i in range(len(model_a_indices)):
        # This loop runs at C speed, not Python speed
        # Typical speedup: 50-100x over pure Python
        ...
```

### Memory Layout

Using NumPy arrays with proper data types:
- `ratings`: float64 array (8 bytes per model)
- `match_counts`: int32 array (4 bytes per model)
- `model_indices`: int32 array (4 bytes per match)

For 500 models and 500K matches: ~10 MB vs ~500 MB for dictionaries.

## Extensions

Potential enhancements:

- Implement Glicko or Glicko-2 rating systems (account for rating uncertainty)
- Add confidence intervals for rating estimates
- Analyze rating by language or task type
- Compare with other ranking methods (e.g., TrueSkill, PageRank)
- Implement time-decay for older matches
- Add statistical significance testing
- Build prediction model for future rankings
- GPU acceleration using CuPy for even larger datasets
- Distributed processing using Dask for multi-machine scaling

## License

This project is part of the AI Agent practical training course materials.

## Contact

For questions or issues, please refer to the course materials or discussion forums.

