Two things landed on Hacking News this week that I keep thinking about. Moonshot dropped the weights for Kimi K3.1 Pro, and a day or so later Baidu shipped ERNIE 5.5 Flash. Both open. Both free to download. And the Kimi benchmark table had it going toe to toe with Opus 5.5 and GPT-6 Astra on coding tasks. Not “promising for an open model.” Close enough that you have to squint at the numbers.
I spent a good chunk of my career in the enterprise AEM world where “the big vendor” was always the safe answer. Nobody got fired for picking the expensive, closed, supported option. So I’ve watched the open-weight catch-up with a bit of suspicion. Every few months someone announces the Linux moment for LLMs and then you try it and the thing can’t hold a function signature in its head for more than three turns. This week felt different, and I want to talk about why, and also where I think the “open models won” crowd is getting ahead of themselves.
What actually shipped
Kimi K3.1 Pro is a mixture-of-experts model. 2.8 trillion total parameters, but only about 104 billion active per token, which is the whole point of MoE. You pay the memory cost of a giant model but the compute cost of a much smaller one. It has a one-million-token context window and it’s natively multimodal. The weights are on Hugging Face under a license you can actually read.
The benchmark table is the part that made the rounds. On SWE Marathon it posted 51.7 against Opus 5.5’s 46.3. On Terminal-Bench 4.0 it was 65.7 to Opus’s 66.4, basically a coin flip. It lost badly on a few others, Program Bench being the ugly one, where it managed 31.5 against Opus at 65. So it’s not a clean sweep. It’s spiky. Great at some agentic coding, mediocre at others.
ERNIE 5.5 Flash is a different animal, a smaller fast model aimed at cheap high-volume inference rather than topping leaderboards. I care about it less for its scores and more for what it signals: Baidu is shipping open weights on a cadence now, not as a one-off PR move.
Why spiky benchmarks matter more than the averages
Here’s the thing about a model that’s excellent at SWE Marathon and bad at Program Bench. For most of us that’s fine, because we don’t do “benchmarks,” we do our own repo. The question was never “is this model good in general.” It’s “is this model good at the five things I actually ask it to do all day.” And the only way to find that out is to run it against your own work, which almost nobody does because it’s tedious.
I pulled K3.1 down and pointed it at a mid-sized TypeScript refactor I’d already done by hand, so I had a known-good answer to compare against. Not a formal benchmark, just me reading diffs. It got the structure right and missed one edge case that a closed frontier model caught last month. Honestly, for a model I can run myself, that’s a good trade. I’m not going to pretend I measured a percentage. I read maybe a dozen diffs and formed an impression. The impression was “yeah, I’d use this.”
The million-token context is the sleeper feature. I’ve gotten used to chunking large codebases and babysitting what goes into the window. With a context that size you can throw a whole service in and stop playing librarian. Whether the model actually attends to all of it or just pretends to is a separate question, and I’m genuinely not sure yet. Long-context benchmarks are famously easy to game. Take that one with salt.
The part the hype crowd keeps skipping
Downloadable weights are not the same as usable weights. A 2.8T MoE model, even with 104B active, is not running on your laptop. You need serious hardware or you rent it, and once you’re renting GPU time to self-host, the math against just paying for an API gets complicated fast. I ran the back-of-envelope numbers for a small team and for anything short of heavy sustained volume, the closed API was still cheaper when you count the engineering time to keep an inference stack alive. I didn’t run a rigorous TCO model, so don’t quote me, but the gap wasn’t close.
So “open” here mostly buys you three things. One, you’re not locked to a vendor who can change pricing or deprecate your model out from under you, which anyone who’s lived through a platform migration takes seriously. Two, you can fine-tune on your own data without shipping it to someone else, which for regulated industries is the whole ballgame. Three, you get a hard floor on availability. Nobody can rate-limit a model sitting on your own disk.
None of those are “it’s as good as GPT-6 and free.” They’re real, boring, enterprise reasons. The kind I spent years caring about.
What I’d actually do with this
If I were setting up a stack today, I wouldn’t pick a side. I’d route. Easy high-volume stuff, classification, summarization, first-pass drafts, goes to a cheap open model you host or rent cheaply. The hard agentic work where a mistake costs you an hour of debugging stays on whatever closed frontier model is winning that month. You treat models like a CDN, not a religion.
The interesting shift is that this routing strategy is now viable. A year ago the open option was so far behind that routing meant accepting bad output on half your traffic. With K3.1 trading blows on real coding tasks, the floor came up enough that the cheap tier is no longer the embarrassing tier. That’s the actual news, and it’s less exciting to tweet than “open models beat OpenAI,” which is why you didn’t see it in the headlines.
Where I’d be careful
I wouldn’t bet a production pipeline on a model released this week, no matter how good the table looks. Give it a month. Watch for the quiet threads where people hit the rough edges the launch post didn’t mention. Every model has them. The ones that score 92 on a benchmark and then fall apart on your particular flavor of legacy code are the expensive surprises.
And I’d keep an eye on the license, not just whether it exists but what it actually permits for commercial use at scale. “Open weights” covers a lot of ground, from genuinely permissive to “free until you make money, then call us.”
If you’ve got the hardware or the GPU budget, pull K3.1 and run it against your own worst code. Not the demo prompts. Your codebase, the one with the weird build step and the module nobody understands. That’s the only benchmark that tells you anything. I’d rather read ten diffs from my own repo than a hundred leaderboard rows.