Botsbench AI security openweight model roundup TL;DR

Somewhere between the Black Hat pool parties and our standing-room-only research presentations & AI trainings at BSides, we ran our over-worked botsbench agents on the new Kimi K3 and DeepSeek V4 Flash releases that everyone has been talking about. Continuing our naming tradition, we’re dubbing K3 the disappointing chaotic evil “Coin-Eating Dragon” due to its high cost relative to others in its weight class. DeepSeek, however, wins the crown yet again: it’s the price leader for open weight models on the second-tier side of the open weight spectrum, so we’re calling it the chaotic good “Pauper Prince”.

DeepSeek V4 Flash makes for an interesting sweet spot. With regular harnesses, generally only frontier provider models (Sonnet 5, Opus 5, …) and the open weight GLM 5.2 make it to the 50-60% score range in our blue team investigation eval CyBT-CTF. With V4 Flash at 37%,  what it loses in quality, it makes up in price: 1/10th the effective cost of GLM 5.2 once all tasks are performed . If you must do open weight, and do not need the quality of GLM 5.2,  it’s attractive. However, when proprietary models are allowed, GPT-5.6 Luna’s recent price cuts make OpenAI cost the same as V4 Flash yet only OpenAI gives the top-tier results. If you can’t use OpenAI’s latest, the proprietary Qwen 3.7 Plus also runs much better, albeit at a slightly higher cost than V4.

Kimi K3,  however, is a disappointment so far. It costs 2X more than GLM 5.2 yet produces much worse results. For other evals, the situation may be different, but for our investigations so far, the balance is discouraging.

Full model cards and results at https://botsbench.com/open-source.html 

We’ll be reporting more open weight experiments ahead, such as on the harness comparison, and as our Louie harness generally gives a +15% boost to models, how the leaderboard gets disrupted when it enters the picture.

Meanwhile, if you’ll be at SANS Cloud Summit in SF, say hello, and if you couldn’t make our sold-out AI trainings at BSides LV, we’ll be running longer 2-day masterclasses on Ai investigations online through the end of summer / early fall.