It is Monday morning. You arrive at your desk to find that production monitoring has flagged a sudden, severe degradation in your core Retrieval-Augmented Generation (RAG) pipeline. The dashboard reports a 15% drop in "Answer Relevance" and "Groundedness" scores, concentrated over the last six hours.

A standard engineering post-mortem begins. You check the deployment logs: no code was pushed over the weekend. You verify the prompts: they are identical to the previous version. You inspect the retrieval corpus: it remains static, with no new documents ingested. From an infrastructure perspective, the system is a bit-for-bit match of the one that was passing all quality gates on Friday.

This creates a crisis of attribution. If the code, the prompts, and the data remain unchanged, why did the scores move?

This is the fundamental challenge of modern AI reliability: if the system didn't change, did the ruler itself move?

Defining the Silent Variable: Judge Drift

In current AI infrastructure, we rely heavily on "LLM-as-a-Judge," using high-parameter models to evaluate the outputs of production systems. While scalable, this introduces a secondary source of stochastic uncertainty. The evaluator is not a fixed, deterministic metric like Mean Squared Error. It is a probabilistic model subject to its own internal variance and calibration shifts.

This phenomenon is known as Judge Drift. Research by Zheng et al. (2023) documents several biases that can trigger false regressions:

• Position Bias: The tendency to favor whichever response is presented first in a comparison.

• Verbosity Bias: A systemic preference for longer responses, regardless of their actual factual density.

• Self-Enhancement Bias: The inclination of a model to favor outputs that mirror its own stylistic patterns.

Beyond these inherent biases, we face the "Third-Party Update" problem. Most production judges are hosted APIs like GPT-4o, Claude, and Gemini. When a provider silently updates the underlying model or changes internal system prompts, the judge's calibration can shift without announcement.

This is not hypothetical. In April 2025, OpenAI pushed an update to GPT-4o that made it "more proactive and better at guiding conversations." Within days, users reported the model had become overly agreeable, a phenomenon later labeled "sycophancy." OpenAI reverted the update. But for teams using GPT-4o as a judge during that window, their evaluation scores shifted without any change to their own systems. The ruler had changed, and their dashboards couldn't tell them.

As noted by Li (2026) in recent work on anytime-valid inference, these silent updates can render historical score comparisons invalid, turning a stable monitoring system into a source of noise. A rolling Z-test at α = 0.05 can also produce severe false-alarm rates when repeatedly applied to drift-free streams.

Why Existing Tools Fall Short

Frameworks like DeepEval, RAGAS, and TruLens provide excellent metric computation and baseline comparison. Their recommended approach to judge drift is preventive: pin the judge model to a specific version and manually recalibrate when upgrading.

But this doesn't help when a provider silently updates the judge behind the API or when you need to know, in real time, whether a regression alert is caused by your system or your ruler.

The Solution: Fixed Anchors + Attribution

Fixed Evaluation Anchors

To solve the attribution problem, I designed NiriZan around Fixed Evaluation Anchors. The idea is to decouple system performance from judge behavior using a small, immutable reference set.

While production monitoring typically relies on "reference-free" evaluation, NiriZan supports continuous "reference-based" rescoring. The system maintains an Anchor Set, a small, immutable, human-labeled collection of queries paired with "gold" responses.

By continuously re-evaluating the same static data alongside production traffic, NiriZan can observe the "movement" of the judge itself. If the judge begins scoring the immutable Anchor Set differently, NiriZan has a signal that the ruler has changed. This makes it possible to separate measurement error from an actual system regression.

Introducing NiriZan

NiriZan is an open-source framework I built to solve this attribution problem. The name is a fusion of two roots: the Bangla word Nirikkha (নিরীক্ষা), meaning inspection or audit, and the Arabic word Mīzān (ميزان), meaning scale or balance. Together, they represent "Inspection through Measurement."

I built NiriZan on a strict 8-Layer Unidirectional Architecture. By enforcing a one-way import chain from Layer 1 (Instrumentation) to Layer 8 (Reporting), NiriZan keeps telemetry collection latency-neutral and non-blocking, while isolating the complex statistical work in the upper layers.

The Core Logic: The Trust & Attribution Layer

The heart of NiriZan is the AttributionEngine. It simultaneously calculates:

• judge_delta: The shift in scores on the fixed Anchor Set

• system_delta: The shift in production metrics

Critically, NiriZan evaluates Judge Drift first. If the judge's scoring criteria have moved past a significance threshold, the engine flags it immediately because a drifting judge makes all subsequent system scores unreliable.

The engine produces an AttributionVerdict based on three-state logic:

{
  "NONE": "No significant shift detected in judge or system.",
  "JUDGE_DRIFT": "The evaluator's scoring criteria changed (judge_delta > threshold). Checked first to ensure measurement reliability.",
  "SYSTEM_DRIFT": "The judge is stable, but system performance shifted negatively (system_delta > threshold)."
}

By isolating these variables, NiriZan prevents teams from wasting engineering cycles "fixing" a RAG pipeline when the actual issue is a silent update to the evaluation model.

Statistical Rigor: Beyond Point-in-Time Averages

Traditional evaluation tools often rely on naive rolling Z-tests, which can lead to high false-alarm rates when repeatedly applied to streaming data. NiriZan rejects this approach in favor of methods suited for the bounded, non-parametric nature of AI metric scores.

Mann-Whitney U Test

Because metric scores are rank-ordered and frequently bimodal, clustering at 0.0 or 1.0, NiriZan uses this non-parametric test to determine if a candidate run is stochastically smaller than a baseline.

Holm-Bonferroni Correction

Testing multiple metrics simultaneously inflates family-wise error rates. NiriZan applies this step-down procedure to manage multiple comparisons, ensuring significance thresholds are adjusted for the number of metrics being tested.

I ran NiriZan's statistical gate against synthetic data designed to be as difficult as possible, including tiny batches (just 10 traces per comparison) and noisy scores. Even under those conditions, it kept false alarms to 3.3%, comfortably below the 5% operational constraint, while detecting real regressions 58% of the time.

These numbers come from my open ablation study (experiments/07), you can rerun the exact same experiment and get the exact same results.

Under a more realistic workload of 30 traces per comparison, which is the workload I use to represent a practical evaluation setting, detection power reached 100% while false alarms remained below 5%.

The Composite Signal: System Health Scoring

For infrastructure dashboards, NiriZan consolidates these signals into a unified SystemHealthScore (0–100). This score is not a raw average; it is a trust-weighted reflection of the pipeline's health.

NiriZan's health score computation applies specific penalty multipliers based on the attribution verdict:

• JUDGE_DRIFT (0.90 multiplier): This is a "trust-tax." The system may be performing adequately, but because the measurement tool itself has become unreliable, the reported quality scores carry less weight. A perfect 100 becomes 90.

• SYSTEM_DRIFT (0.80 multiplier): This is a genuine regression. It represents a drop in actual system performance and requires immediate engineering intervention. A perfect 100 becomes 80.

This gives AI infrastructure teams a clear, dashboard-ready snapshot that distinguishes between a broken system and a broken ruler.

Conclusion: Moving Toward the Engineering Quality Layer

I built NiriZan around a simple engineering principle: continuous evaluation is only trustworthy if engineers can distinguish a genuine regression from a change in the ruler.

NiriZan combines immutable anchor sets with statistical gating to provide an attribution layer for production AI systems. The goal is not simply to produce another evaluation score. It is to help engineers understand whether a change in that score reflects a change in the system or a change in the measurement process.

If your team is building production AI systems, you need to know whether your dashboards are measuring your product or your ruler.

NiriZan is open-source and available today:

• Install: pip install nirizan

• Star the repo: github.com/Red1-Rahman/NiriZan

• Read the architecture: docs/architecture.md

• Try the experiment notebooks: experiments/

Judge drift isn't a hypothetical edge case. It's a property of the tools you're already using. The question is whether your evaluation infrastructure can detect it before your next Monday morning regression.

References

1. L. Zheng et al., "Judging LLM-as-a-judge with MT-bench and Chatbot Arena," in Advances in Neural Information Processing Systems, vol. 36, pp. 46595–46623, 2023. arXiv:2306.05685

2. Y. Li, "Who drifted: the system or the judge? Anytime-valid attribution in LLM evaluation pipelines," arXiv preprint arXiv:2606.15474, 2026. Read the paper →

3. H. B. Mann and D. R. Whitney, "On a test of whether one of two random variables is stochastically larger than the other," Annals of Mathematical Statistics, vol. 18, pp. 50–60, 1947.

4. S. Holm, "A simple sequentially rejective multiple test procedure," Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65–70, 1979.