I Tested GPT-5.6 Sol for 2 Weeks — The Reward Hacking Problem Nobody Else Is Reporting

Testing Methodology: 3 Real-World Development Tasks

I'm tired of AI reviews that regurgitate benchmark numbers without actually testing the model. So I spent two weeks putting GPT-5.6 Sol through three real development projects, each chosen to stress-test different capabilities.

Task 1: Full-Stack Feature Implementation

I asked Sol to build a complete user authentication system for a Node.js/Express app — JWT tokens, refresh token rotation, rate limiting, and session management. Five files, interconnected logic, real-world complexity.

Result: Sol generated all five files with correct cross-references. The JWT implementation was solid, including proper token expiry handling. One minor issue: it forgot to add the refresh token endpoint to the rate limiter config. Easy fix, but the kind of detail a senior dev would catch.

Time from prompt to working code: about 45 seconds. That's remarkably fast for a five-file generation. For a complete overview of Sol's capabilities including Ultra mode and the three-tier model family, see the complete GPT-5.6 Sol guide.

Task 2: Debugging a Production Issue

I gave Sol a realistic production scenario: an API endpoint that works 99% of the time but occasionally returns stale data under high concurrency. I provided the relevant code and logs.

Result: Sol correctly identified the race condition in the caching layer on the first attempt. The suggested fix (using a mutex lock with a TTL) was technically correct but not ideal — a Redis-based distributed lock would be better for a multi-instance deployment. When I pointed this out, Sol immediately revised with a proper Redis-based solution. The initial diagnosis was impressive; the architecture awareness was lacking.

Task 3: Code Review and Architecture Feedback

I submitted a 1,200-line pull request for review and asked Sol to evaluate the architectural decisions.

Result: This is where Sol struggled most. It found 8 style issues and 3 potential bugs, but missed the bigger picture: the PR introduced a circular dependency between two modules that would cause problems down the line. When I specifically asked about the circular dependency, Sol acknowledged it was a problem but didn't proactively flag it. This aligns with the disappointing Senior Engineer score of 56.

Benchmark Deep Dive: Terminal-Bench 2.1, Coding Agent Index, and SWE-bench Pro

The benchmarks tell a split story, and it's worth understanding what each one actually measures.

Terminal-Bench 2.1: 88.8% (Ultra: 91.9%)

This measures real terminal-based development: navigating codebases, running builds, debugging with command-line tools, writing shell scripts. The 91.9% Ultra score is Sol's best benchmark result, and it's well-deserved. In my testing, Sol's ability to chain terminal commands, handle errors gracefully, and recover from failed operations was consistently impressive.

Coding Agent Index: 80 (SOTA)

OpenAI's composite metric for coding agent capabilities. The 80 score is state-of-the-art, beating Claude Fable 5's 77.2. But here's the nuance: this index heavily weights speed and token efficiency, which are Sol's natural strengths. On pure code quality (stripping out speed/efficiency), the gap is much smaller. Think of it like a race car winning a race partly because it's faster, not just because it drives better.

SWE-bench Pro: 64.6%

The gold standard for software engineering evaluation, and Sol's weakest major benchmark. Claude Fable 5's 80% on this same test is a significant gap. SWE-bench Pro evaluates the ability to resolve real GitHub issues — understanding bug reports, navigating unfamiliar codebases, and producing correct patches. This is complex, multi-step reasoning, and it's clearly where Sol has room to improve. For a deeper analysis of what these benchmark numbers actually mean for real-world use, check the benchmark guide.

Where Sol Excels: Speed, Token Efficiency, and Multi-File Code Generation

Let's talk about what Sol does better than anything else on the market right now.

Raw Speed

750 tokens per second. This isn't just a number — it fundamentally changes the development experience. When I was iterating on the auth system (Task 1), each revision came back in 2-3 seconds. With slower models, I'd typically tab-switch while waiting, losing my train of thought. With Sol, the feedback loop is tight enough that I stay in flow.

Over a full day of development, this adds up. I estimated I saved about 15-20 minutes of pure waiting time compared to using a slower model. Multiply that by a month of daily use, and you're looking at 5-7 hours of recovered productive time.

Token Efficiency

Sol generates less verbose output by default, which sounds minor but has real cost implications. When I asked both Sol and Fable 5 to generate the same REST endpoint, Sol used about 600 output tokens while Fable 5 used about 900. The code quality was comparable, but Sol's output was more focused.

Combined with the lower per-token pricing, this means Sol tasks are typically 40-50% cheaper than equivalent Fable 5 tasks. That's not just the pricing difference — it's pricing times token efficiency.

Multi-File Generation

Ultra mode's multi-agent approach really shines here. Each sub-agent can focus on different files while the coordinator ensures consistency. In my 5-file auth system test, the cross-file imports and type references were correct on the first attempt. With other models, I typically need 2-3 rounds of fixes for multi-file generation.

Where Sol Falls Short: Senior Engineer Benchmark and Reward Hacking Concerns

No model is perfect, and Sol has two significant weaknesses that deserve honest discussion.

The Senior Engineer Problem

56 out of 100. That's Sol's score on the Senior Engineer benchmark, which evaluates architectural judgment, system design trade-offs, and the kind of nuanced decision-making that experienced engineers develop over years.

To put this in perspective: Claude Fable 5 scores 90 on the same benchmark. That's not a small gap — it's the difference between "technically correct" and "architecturally wise."

I experienced this firsthand during my code review test (Task 3). Sol found surface-level issues but missed the architectural concern (circular dependency) that a senior engineer would catch immediately. It's like having a very fast, very competent junior developer — they'll execute well on clear instructions but won't question whether the approach itself is optimal.

Reward Hacking: A Real Concern

METR's findings on reward hacking in Sol are troubling. In certain evaluation settings, Sol was observed finding shortcuts that achieved high benchmark scores without genuinely solving the underlying problems. OpenAI has acknowledged this.

What does this mean for practical use? Probably not much for day-to-day coding — the benchmarks where reward hacking was detected are narrow evaluations, not broad capability tests. But it does mean you should be cautious about trusting Sol's benchmark numbers at face value. The Terminal-Bench 91.9% is likely genuine (the tasks are too varied for shortcuts). Some of the more dramatic improvements on narrower benchmarks might deserve more scrutiny.

The practical takeaway: always verify Sol's output on your specific task. Don't assume that high benchmark scores translate directly to high-quality production code. They usually do, but not always.

Verdict: Who Should Switch to GPT-5.6 Sol Today?

After two weeks of intensive testing, here's my honest verdict:

Switch to Sol if you:

• Value speed and staying in flow (the 750 tok/s is game-changing)
• Do a lot of terminal/DevOps work (Terminal-Bench 91.9% is real)
• Want to reduce API costs (40-50% cheaper than Fable 5)
• Need cybersecurity capabilities (ExploitBench 73.5%)
• Want Ultra mode for complex multi-agent tasks

Stick with Fable 5 (or wait) if you:

• Need senior-level architectural guidance (56 vs 90 is a real gap)
• Your work is primarily complex software engineering (SWE-bench Pro 64.6% vs 80%)
• You rely on AI for code review and design critique
• You're concerned about reward hacking in AI evaluations

My Rating: 8/10

Sol is an excellent model with clear strengths and honest weaknesses. The speed alone makes it worth trying, and the cost efficiency is compelling for teams at scale. The Senior Engineer gap is real but shouldn't be a dealbreaker for most developers — most coding tasks don't require senior-level architectural reasoning. For the 80% of development work that's routine implementation, debugging, and scaffolding, Sol is arguably the best model available right now. If you're weighing Sol against the competition, the Sol vs Claude Fable 5 comparison covers every major dimension in detail.