Fables and Mythos-conceptions: The defender’s perspective, with receipts

Before Anthropic’s new Fable 5 model got shut down by the US Government, we took it through its paces, so we wanted to share our benchmark-driven defender’s perspective, and as heavy agentic coders, a practitioner report.

The tl;dr: For coding, Fable is remarkably autonomous and clever. Unfortunately for defenders, Fable is also remarkable in simultaneously being both bland for investigations and chaotic-evil in packaging.

First, the good news. Fable is remarkably easy to throw at nasty coding problems because it is a step up in self-planning and being clever.

For context, the core of Graphistry is some of the trickier code out there – we helped create the first versions of what became Apache Arrow, which most databases use nowadays to run faster – which means upgrading our older CPU+GPU libraries can be the stuff of nightmares even for our seniormost engineers.  Fable is well-named: A 2-day Claude Code job cranked through a wide-sweeping modernization project we’d been start-and-stopping on for years. Compared to Opus 4.8 and Codex 5.5, which need frequent manual steering as plans unfold, Fable largely steered itself. Along the way, Fable autonomously detected and patched the exact kind of bugs we were dreading such as in redoing custom binary HTTP protocol plugins for arrow and fastify.

Fable’s autonomy and cleverness did come at a cost: Even with generous Max 20X subscriptions, we quickly burned our quota. Our 2-day job was really two 1-day jobs with some waiting in between while weekly quotas reset.

Chart 1: Fable unlocked (Mythos) extrapolated to be a nothingburger when no refusals. 

Now the bad news: Cybersecurity

Fable 5 is a configuration of Mythos 5 that bakes in AI safety policies, which sounds good in theory, but it suffers from frequent unpredictable task refusals. If Fable was amazingly intelligent on investigation tasks, we’d be more excited to stomach that cost, but it wasn’t.

For fair, open, and representative investigation tasks, we ran botsbench.com, our public tracker of Splunk Boss of the SOC (BOTS) cybersecurity investigation CTF.  The 59 BOTS cybersecurity tasks take a professional human team a day or two, and require using a database to reason through incidents real recorded logs from computers, networks, cloud services, and identity systems. We also ran Fable on Cyber Blue Team CTF (CyBT-CTF), our new addition to botsbench. We’ll describe CyBT-CTF more in the coming weeks – it riffs on BOTS by being a hold-out layer for anti-cheating purposes.

The first bad news is, when Fable 5 tackles a BOTS or CyBT-CTF task, it is not statistically distinguishable from Opus for investigations even in the best conditions.

Claude Code scores 40-50% on CyBT-CTF. We know more is achievable: Our Louie.ai agent harness typically gives a 10-15 point boost out-of-the-box over Claude Code, Codex, and OpenCode, and we announced at RSAC week that we can even get a 96% on BOTS (!) when we add a small amount of special prompting. The hype around Mythos gave us high hopes for seeing the 10-15% kind of boost we see from Louie harness engineering, but it mostly just alternated between giving Opus-level results and failures. Even when we compared Fable to Opus on just the cases where Fable didn’t do a downgrade or refusal – so the Mythos case – we saw no statistically significant quality benefit for investigations over the 110+ combined tasks.

The second bad news we struggled with in Fable is it is frequent and chaotically evil in how it does refusals. Before Fable, we occasionally had frontier model API providers refuse benign defender tasks like searching firehoses like logs, news, and social media, but they were occasional enough to tolerate. With Fable, 36% of CyBT-CTF tasks hit refusals, and a whopping 86% of BOTSv3 tasks were refused. These were not exploit-development tasks we would expect to be refused. They were routine defender questions like “find what service triggered login alert X.”

Worse, the failures were a bewildering mix of empty responses, model downgrades, and some other errors. Imagine your anti-fraud, customer support, and cyber teams being trusted to keep everyone safe in a boring predictable way using APIs that are a coin toss away from failing closed, and trying to keep you guessing on the fact that they refused to do their work. Fable is essentially post-training safeguards on top of the Mythos model, and for cybersecurity purposes, Anthropic is hurting defenders more than attackers on this release.

Ultimately, the defender perspective often ends up an economic view of attack vs defense. Yes, Fable can sometimes help attackers, but OSS models can already be used to find most of the same security holes, generate exploits, and orchestrate attacks. The important question becomes whether the marginal attacker value exceeds the defender damage from higher cost, worse reliability, and opaque refusals. Our experience says the main value to defenders is in easier novice coding, and for both attackers and defenders, the value for the vast majority of security tasks is demonstrably questionable compared to much more affordable models, whether from open source or previous-generation frontier models.

We like to say “evals or GTFO” and “own your evals”, and for those wanting to upgrade their models, this experience shows why. For more info, you can drill into the Fable page on botsbench (https://botsbench.com/cybt-ctf.html).

PS: A plea to the frontier model API providers – Do not make running regular cyber, anti-fraud, customer support, and other investigative tasks that most companies do every day feel like playing Russian roulette on one side, and a Kafka-esque chatbot support nightmare on the other. We almost always love these new models for coding, but for defense automation, we have enough chaos from the attacker side to now have to treat our API infra providers as an adversary as well!