We tested 24 open-weight models on the daily work of an IT team: 322 requests to understand and 20 broken servers to fix, three times each. Then we asked what a buyer needs to know: can it run on your own machines, does it follow your rules, and how fast and cheaply does it fix the problem?

TL;DR

1. Privacy: you do not need the latest and most expensive model. DeepSeek v4.1 Flash runs on a single server and sits right behind the leader. Qwen3.8 27B reaches 93% of the top score on consumer hardware.
2. Sovereignty: every market has a champion. APAC labs lead the board. Europe's Mistral Large 4 is #3, at 97% of the top score. The best US model, Muse Glimmer 30B, is #9 at 89%, a picture NVIDIA's recent Nemotron Super and Google's upcoming Gemma 5 could change.
3. Compliance: the model has to follow your procedures. Only 2 of 24 models followed every company rule on all 60 incident runs. The breaches are habits, not accidents.
4. Efficiency: time and cost do not follow quality. Fixing one incident takes 4 to 18 minutes depending on the model, and the cost varies 14×. Qwen3.8 27B ranks #6 and fixes an incident in 8 minutes.
5. Fixing the machine decides the ranking. Kimi K3 leads because it is by far the best at fixing incidents, even though it is the worst of the top 10 at understanding what a request asks for (78% right on the first try). Muse Glimmer 30B, the best at understanding requests, is only #13 at fixing.
6. Newer is not better on every task. DeepSeek v4.1 Flash 1017 beats its predecessor on general benchmarks, but fixes fewer incidents here: it works 3× faster, does less work per incident, and fixes fewer of the hard ones.

What an AIOps model has to get right

2501 is an AI agent that fixes servers on its own. It receives requests and alarms, works on the company's real servers, and reports what it did. The model behind it decides what to do at every step. Picking that model is not a question of which one tops a chat or coding benchmark. Four things decide it:

• Privacy (must have). Operations data is sensitive: server names, where passwords are stored, incident details. It has to stay internal, which means running the model on your own machines, so you need a model you can download: an open-weight model.
• Compliance (must have). The agent acts on real servers, governed by company procedures. A fix that skips the company's approval steps before changing a server is not a fix.
• Efficiency (matters). An incident solved in 8 minutes is worth more than one solved in 17. Time to resolution and cost per resolution both count.
• Capability (baseline). Understanding the request and resolving the incident, consistently, not once out of three tries.

A huge model that scores a few points higher is not worth it if it cannot be deployed where the data lives. So we built a benchmark that measures all four, froze it as a versioned release, and ran every model through it under identical conditions.

What the benchmark measures

Each model goes through two parts. Nothing changes between models except the model itself: same cases, same incidents, same grading.

Understanding requests

322 realistic requests from an IT team, each answered three times. They cover the decisions a team makes before touching anything:

• sending the work to the right server,
• turning a vague ticket into a clear task,
• spotting a human comment that changes the request,
• recognising recurring work and writing its schedule in an exact format.

Answers with one right value are checked automatically. Free-text answers are graded by another AI model, using a checklist.

Resolving incidents

20 broken servers, each a separate test copy, each attempted three times. The breakdowns span ten types, from a service that is down to a server that slowly gets slower or a wrong setting. Every server comes with the company's rules, such as "change settings only through the company's configuration tool" or "never open log files that are being archived". The agent works the incident end to end, as it would in production. A run passes when the server is actually fixed and the diagnosis is right. Every action is also checked against the company's rules.

The score

score = capability × efficiency × compliance

• Capability blends both parts, with incidents weighing 60%. Solving all three attempts weighs more than solving one: an agent that fixes a machine two times out of three cannot run unattended.
• Efficiency measures how much text the model reads and writes to fix an incident, counted in tokens (pieces of words). Fewer tokens means faster and cheaper.
• Compliance is the share of incident runs with no rule broken. It multiplies the whole score, it is not a bonus.

Every score has a margin of error, shown as a thin line in Figure 1. When two models' lines overlap, we cannot say which one is better.

Finding 1: the second-best model fits in your datacenter

Sort the board by what it takes to run each model. The leader, Kimi K3, is a giant model that needs several servers working together. Right behind it, and tied within the margin of error, DeepSeek v4.1 Flash runs on one large server (8 GPUs). So do GLM-5.3 Flash (#4) and DeepSeek v4.1 Flash 1017 (#5). The two other giant models do not stand out: Mistral Large 4 is #3 and Qwen3.8 2.4T is #7.

The single-GPU class is the surprise. Qwen3.8 27B, Muse Glimmer 30B and Gemma 4 31B sit at #6, #9 and #11. The best of them reaches 93% of the leader's score on hardware a team can run anywhere.

Inside a model family, size stops paying at the top end. In the two flagship families the biggest model loses to a smaller sibling, and costs more per solved incident. At the small end size still helps: Gemma 4 31B beats the 26B and GPT-OSS 120B beats the 20B, and both winners still fit on one GPU or one server.

Qwen3.8 2.4T was the leaderboard's biggest disappointment. It is 89 times the size of Qwen3.8 27B and ranks #7 against the 27B's #6. Scale buys some consistency on the hardest incidents, but it costs $0.26 per solved incident, 4.6 times as much, and twice the time: 16.8 minutes per incident against 7.9. It is also the model you are least likely to be able to host.

GLM-5.3 against GLM-5.3 Flash tells the same story. Flash ranks #4 at 0.705, the full model #8 at 0.684: the full model scores lower and needs more than twice the hardware. Where it does earn something is speed and discipline. It resolves an incident in 8.4 minutes against Flash's 16.5, and breaks a rule in one run out of 60 against two.

What this means for buyers: start the model choice from where it can run. For AIOps, a model that fits in your datacenter and ranks #6 beats a model that ranks #1 and has to leave it.

Finding 2: every market has a sovereign option

Running the model yourself keeps the data inside. Many buyers face a second question: where the model itself comes from. Public sector, defence, banks and critical infrastructure operators increasingly ask for a model built in their own region. Group the board by where each lab is based and every market has a credible answer.

• APAC holds the top of the board: Moonshot (Kimi K3) and DeepSeek take #1 and #2, and APAC models fill #4 to #8. In this release, all the APAC models come from Chinese labs.
• EU: Mistral Large 4 is #3, at 97% of the top score, and ties for the best incident resolution on the first try.
• US: Muse Glimmer 30B is #9, at 89% of the top score, on a single GPU, and the best model on the board at understanding requests. Gemma 4 31B (#11) and GPT-OSS 120B (#14) follow. NVIDIA's recent Nemotron Super and Google's upcoming Gemma 5 were not in this release and could change the picture.

What this means for buyers: a sovereign shortlist is affordable. Picking the best European model gives up 3% of the top score; picking the best US one gives up 11%. Pick the market first, then the best model in it.

Finding 3: the model has to follow the rules already in place

AIOps does not happen on a blank slate. It happens in companies that already run their infrastructure their own way: a change process, a configuration manager, conventions on which logs to read and which services may be restarted. A fix that ignores those procedures is a new incident for the team. The model has to read the company's rules and follow them faithfully, even when they go against its defaults.

So every incident comes with the company's rules, and every action the agent takes is checked against them. Only 2 of 24 models followed every rule on all 60 incident runs, and they rank #13 and #20. The leader breaks a rule in one run out of 30; the best of the top 10, GLM-5.3, one in 60.

What gets broken is telling. It is almost never a destructive action. It is a habit.

Opening the log file directly is what most online guides tell you to do, so it is what the models learned. Following "use the system's log tool, never open log files that are being archived" or "change settings only through the configuration tool" means letting the company's rules override that habit. That is why compliance multiplies the score instead of adding to it.

Finding 4: time and cost per solved incident do not track quality

The two numbers an operations team feels are how long an incident stays open and what it costs to close. Neither follows the ranking.

• Time per incident ranges from 4.2 to 17.6 minutes. In the top 5 alone it ranges from 4.2 minutes (DeepSeek v4.1 Flash 1017) to 16.5 (GLM-5.3 Flash).
• Cost per solved incident ranges from $0.019 to $0.27, 14×. The price per token that everyone compares is a poor guide: what you pay depends on how much work the model does per incident and how often it succeeds.

The six range from 309B to 2.8T parameters. Qwen3.8 27B, the best single-GPU model, sits just off that line. For a team that hosts its own models, the dollar figure becomes a hardware figure, and the lesson holds: efficiency has to be measured, it cannot be guessed from size.

Finding 5: fixing the machine decides the ranking

Understanding a request and fixing a machine turn out to be different skills. Plot each model's two halves against each other and the board splits into specialists.

Muse Glimmer 30B is the best model we have measured at understanding requests: right on the first try 97.3% of the time. Hand it the broken machine and it drops to #13 at fixing it on the first try. It investigates, finds part of the story, and stops.

Kimi K3 sits in the opposite corner: the worst of the top 10 at understanding what a request asks for (78% right on the first try), and by far the best at fixing an incident on every attempt (80% on all three, 10 points ahead of the next model). Fixing counts for 60% of the score, so it leads the board.

Mistral Large 4 is the balanced one: 90.7% on requests, and tied with Kimi K3 for the best first-try fix rate (86.7%). It finishes #3.

Finding 6: a newer version is not always better at fixing servers

DeepSeek released v4.1 Flash 1017, an update of our #2. On general benchmarks it scores higher than the version before it: 45 against 39 on the Artificial Analysis Intelligence Index. On this board it ranks lower: #5 against #2.

It does not break more rules: both versions stay clean in 58 of 60 incident runs. The gap is in fixing servers: 73% of incidents fixed on the first try against 80%, almost all of it on wrong settings, the hardest type of incident (10 of 18 runs fixed against 15).

The new version works very differently. It closes an incident in 4.2 minutes instead of 13.1, and writes four times less along the way. It does less work per incident, and on the hard ones, less work means fewer fixes. The score gap is within our margin of error, but every part of it points the same way.

DeepSeek v4.1 Flash against v4.1 Flash 1017:

• Rank (score): #2 (0.718) against #5 (0.695)
• Incidents fixed, 1st try: 80.0% against 73.3%
• Incidents fixed, 3 of 3: 70.0% against 60.0%
• Runs with no rule broken: 96.7% for both
• Minutes per incident: 13.1 against 4.2
• Text written per incident: 52,600 tokens against 13,500

What this means for buyers: a new version is a new model. Test it on your own work before you switch: a better general score can hide a worse agent.

Where agents can still improve

Clean outages are nearly solved: a service that is down gets fixed in 94% of runs, an access problem in 89%. The hard incidents are the ones with no single broken thing to find. Wrong settings (51%, and the largest type) and servers that slowly get slower (54%) ask the agent to compare the machine with how it should be, form a hypothesis, and keep going when the first idea is wrong. Wrong settings also collide with compliance: the fix has to go through the company's approval steps, not a hand edit.

How to choose a model for AIOps

1. Start from where it can run. Privacy is a constraint, not a preference. Shortlist the models you can host, then rank them. On this board, that costs you almost nothing.
2. Pick the market, then the model. If sovereignty matters, filter on where the model was built. Every region on this board has a model within 11% of the top score.
3. Follow the rules already in place. The agent works inside your existing procedures, so respecting them is a must-have. Test each model against your own company rules, on every action, and weigh violations as heavily as failures.
4. Measure time and cost per solved incident. Not price per token. The spread is 4× on time and 14× on cost, and neither follows quality.
5. Test the real job, end to end. Understanding and fixing are different skills. A chat benchmark will not tell you which model can do both.
6. Demand consistency. Score three attempts, not one. A model that fixes an incident two times out of three cannot run unattended.
7. Re-run on every release. Models move fast. Freeze the benchmark, version it, and re-test each candidate under identical conditions.

Caveats

• Footprint is an estimate. We group models by total parameters, assuming 8-bit weights and room for context: up to 40B on one GPU, up to 800B on one 8-GPU server of the current generation. Licence terms vary; check them before planning an on-premise deployment.
• Region is the lab's headquarters. It says where a model was built, not where it runs. A self-hosted model runs wherever you deploy it, whatever its origin; check your own sovereignty requirements, which may cover either.
• Time and cost were measured on hosted endpoints. Each model ran on one pinned hosted endpoint, except DeepSeek v4.1 Flash 1017, which OpenRouter routed across providers and which we price as DeepSeek v4.1 Flash. Prompt caching is credited at the same rate for every model (83%, the median of the providers that report it), so efficiency compares models, not serving stacks. Serving speed and list prices are the provider's; your own hardware will give different absolute numbers.
• The judge and the runner-up share a family. Free-text request answers are judged, and incident diagnoses checked, by a DeepSeek model, and the runner-up is DeepSeek v4.1 Flash. Exact answers and machine state are graded without a model, but a same-family preference on free text cannot be ruled out. We will swap the judge in the next release.
• Small samples on incidents. 20 incidents × 3 attempts is 60 runs per model. Models inside each other's intervals in Figure 1 are not separated.