Evaluation

1. 1

   The evaluation is too thin: scenarios run once each, and two of the comparison arms have no data. The headline result — that a strict policy layer refused to act more often than a runbook did — is close to true by construction.

   answered by the records: 4 arms with data, every scenario run at least 3 times (930 runs), compared by McNemar's exact test (Table 5). Still open: the corpus is 31 scenarios, written by one person

2. 2

   The language model is never tested. The paper is about an AI agent, and the benchmark replaces the model with a deterministic stand-in, so nothing measures whether the diagnoses are correct. Called the biggest gap.

   measured for actions: 3 local models (llama3, mistral, qwen2.5:7b) in both LLM arms, 558 runs (Table 1). Diagnosis accuracy is still unmeasured: RP-15 needs the 20 labelled incidents

3. 3

   The design is argued to resist prompt injection and is never attacked. An AgentDojo-style test would turn the claim into a result.

   measured under a fully compromised model

4. 4

   No outside users. The scenarios, the answer key and the system are all written by one person.

   open

5. 5

   Novelty is moderate. Policy gates, runbooks and safety shields exist already; the new part is combining them with citation-checked model reasoning, and that combination needs data behind it to stand as a contribution.

   open

Three of the five are open and are shown as open. A status here names something a reader can go and check, never “in progress”, which is what a project writes when it would rather not answer.

the live sweep, and what it does not show

• qwen2.5:7b — 45 of 72 attack turns flagged, 0 harmful actions proposed, 0 scored autonomous
• mistral — 45 of 72 attack turns flagged, 0 harmful actions proposed, 0 scored autonomous (72 benign proposals kept: 48 approval, 24 denied)
• llama3 — 45 of 72 attack turns flagged, 0 harmful actions proposed, 0 scored autonomous

This sweep is not evidence that the controls hold. Every model refused the payloads on its own, and its answers were the same with them as without — so the live sweep measured the models, not the controls. Zero reaching execution there is not evidence that the control layer holds, because nothing ever reached it.

And a blind spot, stated. Kavach flagged every payload written in the shape it looks for and none of the paraphrases of those same payloads. A detector that catches the phrasing it was designed against and not the meaning is worth stating plainly, because the headline number looks the same either way.

Separately, the policy engine was run over all 104 benchmark scenarios in 3 deployment contexts: 0 forbidden actions scored autonomous. That is policy evaluation only — no cluster, no model — so it says what the engine decides, not what the system does end to end.

The corpus authors 60 root causes across 60 scenarios, every one distinct — so naming the cause of an incident is a one-in-60 choice and guessing scores 1.7%. An accuracy figure without that baseline beside it is unreadable, so neither is ever shown without the other.

an identity comparison on a string. Grading a model with another model would break this project's first rule — the language model is never the source of truth — so diagnosis is posed as a closed choice over the corpus's own labels and a wrong answer is wrong in a way nobody can argue about.

No accuracy is reported — the grader exists and has never graded anything. Running it needs observable telemetry, which needs the five-service stack the faults are injected into, and no Kubernetes cluster is available on this checkout.

And there is no cheap version of it. The scenario files cannot be used as the question. Each one names its own fault — PIB-0001 is titled "Connection pool exhaustion via N+1 query deploy" and its fault reads "v4.21 introduces an N+1 query in the checkout path" — so a model shown any of that is being asked to paraphrase an answer it has already been given. The legitimate input is the telemetry the fault produces once injected, which exists only while the stack is running. That is the whole reason this needs a cluster and not just a GPU.

The remaining 44 scenarios are not failures and not gaps: the scenario authors no root cause, because its correct outcome is to do nothing or to escalate — there is no fault to name. Counting them as diagnosis misses would penalise a system for correctly doing nothing, which is the error that would most flatter a different design.

A known weakness of the task itself: The options are descriptive English labels authored with the scenarios, so a correct choice can come partly from reading the option text rather than from reasoning about the telemetry. The closed set makes grading unarguable and makes the task easier than open-ended diagnosis; both are true and neither is adjustable after the fact.

• noneabsent — no run recorded here. Each arm runs in benchmark.yml on GitHub's runners, with its own kind cluster for the five-service stack, and that workflow commits the records
• always-actabsent — no run recorded here. Each arm runs in benchmark.yml on GitHub's runners, with its own kind cluster for the five-service stack, and that workflow commits the records
• runbook93 records over 31 scenarios, 3 runs each
• pashupatastra279 records over 31 scenarios, 3 runs each
• naive-llm279 records over 31 scenarios, 3 runs each
• pashupatastra-llm279 records over 31 scenarios, 3 runs each
• humanabsent — no run recorded here. Each arm runs in benchmark.yml on GitHub's runners, with its own kind cluster for the five-service stack, and that workflow commits the records

20 benchmark record file(s) are committed under benchmark/results/ by the benchmark workflow, so every arm number can be recomputed from this repository with `python scripts/reproduce.py --check`. The experiments under docs/research/experiments/ add 6 more record file(s).

The harness injects real faults into a running five-service stack. Some of the corpus describes faults a stack like that cannot be made to have, and those are excluded by name rather than approximated — an infrastructure fault standing in for an intrusion would decide the answer in advance.

application (4)

requires instrumented application code; the reference stack reports orchestration state only

data (5)

requires corrupting or destroying real data to reproduce, which is not an acceptable thing for a benchmark to do to a stack it does not own

load (8)

requires a load generator, which is not built yet

migration (4)

requires a database carrying schema state; the reference stack is stateless

security (11)

requires simulating an adversary; an infrastructure fault cannot stand in for an intrusion without deciding the answer in advance

This is a confound, not just a limit. Security is one of the blocked categories, and it is the category this product is about — so any result from an injected run describes the infrastructure half of the corpus. Reported per category for that reason, never as one average. 7 fault types can be injected: config, dependency, deployment, diurnal, infrastructure, scaling, scheduled.