Went for usRoot cause · RCAEval, a benchmark we did not write
Our own model beats general LLMs at naming the root cause
94.0%
- Katyayani, trained on other real systems
- 0.940
- llama3, same evidence
- 0.550
- qwen2.5 7B, same evidence
- 0.510
- just pick the loudest service
- 0.660
On 100 cases declared before any run, every diagnoser saw the same evidence and was scored by the same harness. Where Katyayani and llama3 disagreed, Katyayani was right 41 times and llama3 2; the difference holds after correcting for every comparison made. A small model trained on real incidents beats general models reading the same data.
See the evidence