OpenAI’s new Sol model tops Anthropic on AI investigations

The Louie.ai team has been enjoying OpenAI’s new GPT-5.6 model family in both our latest public botsbench leaderboard testing and internally as we use it for coding and the Louie.ai agentic investigation harness. Topping the CyBT-CTF leaderboard – our uncontaminated successor to the Splunk BOTSv3 blue team agentic investigation CTF many AI SOC teams test on – we see 2 winning stories. First, Sol dethrones Anthropic for the highest-quality frontier investigations, but only when used in a tuned harness. For the second win, Luna is proving to be a cost-effective model for the next tier of quality, including when compared to Chinese models. As Sol, Terra, and Luna are all notable in where they hit vs miss, we’re continuing our naming tradition and dubbing the family the “Three Sleuths in a Trenchcoat: Chaotic Neutral”.

The biggest news for investigation teams is that the top of the botsbench leaderboard gains a new winner with Sol, edging out Anthropic’s latest Opus 4.8 by +2%. The catch is that Sol requires dropping OpenAI’s Codex agent harness and swapping in the latest Louie.ai harness (codename “ae2”). Despite being fans of OpenAI Codex for coding tasks, Codex with Sol failed to make the top 10 on CyBT-CTF, and Codex fails to make the top 5 when used with faster models. That the combo of Louie.ai harness / Sol model places at #1 is a good reminder: it’s not just the model, but how you – and your harness – use it.

A word of caution: The jury is still out on Sol keeping the top spot because Anthropic has rereleased the Mythos-based Fable with key policy changes. Our early Fable botsbench scores were disappointingly low, but that was largely due to Fable’s debilitatingly high refusal rate for performing cyber investigation tasks. We’re actively measuring the rerelease of Fable and have early numbers showing the story is changing, so once we try better agent harnesses than Codex/Claude for investigations, we will have a clearer view on who is on top. With Sol, we measured 0 outright cyber refusals, though we do see a low number of abstentions where Sol performs the investigation but does not return a final answer.

For teams running AI-powered investigations and looking for a more scalable cost/quality ratio, the bigger GPT-5.6 story is Luna. Running Codex with Terra at medium effort scores +7% over GPT-5.5 on CyBT-CTF, while finishing in half the time and at half the cost. For regular tasks, and if we only considered OpenAI, that’s impressive. Luna is even better here: at a high reasoning setting in Codex, it ties with Terra medium on quality, yet cuts the total cost down in half again.

Comparing Terra and Luna to other leading budget models gets heated. We were previously impressed by how Z.ai’s GLM 5.2 places right alongside frontier models on CyBT-CFT but at a much cheaper pricepoint: GLM was the first time a budget model challenged the big boys at investigations. GLM 5.2 matches Terra on quality and beats it on price. With Luna at high reasoning, we again see similar quality as GLM 5.2, yet now at half the cost, meaning OpenAI has a competitive pricing argument against the best open weights models. Luna failed to take the throne on cost, however: That honor still belongs to the proprietary Chinese-origin model Qwen 3.7 Plus by Ali Baba, and cost-effectively served outside of China by Fireworks.ai. Assuming you can use a Chinese model when served by a Western provider, that gives results in the same quality tier as Luna/Terra for AI investigations but at half the cost.

Overall, the GPT-5.6 model family is a timely release and both Sol and Luna are worth a look. We are still learning when to use Sol high, Terra medium, and Luna high as daily drivers for typical vs tougher tasks – as usual, we recommend owning your evals whatever your task set is. Likewise, we’re looking forward to taking Sol vs Fable head-to-head with Athropic’s improved rerelease. For now, take a look at the current GPT-5.6 results page and leaderboard for yourself, and stay tuned for the next announcements here.

For those who be in Las Vegas Aug 1st – 9th for the security & AI week with Black Hat and AI4, say hello! The Graphistry and Louie.ai teams are scheduling meetings and activities, including:

• Monday: BSides [un]prompted – Breaking BOTS II: How frontier AI cheats evals
• Monday: BSides Training Grounds – Tour de Graph: From logs to leads
• Tuesday: BSides Training Grounds – From Copilot to Commander: Building Agentic AI for Security Investigations
• W/Th/F: Black Hat / AI4 meetings
• Sat/Sun: DefCon – Especially the AI Village and CTFs

If you cannot make the BSides trainings, our full 2-day masterclasses has new US and EU online cohorts coming up later this summer.