SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges

1. Analyze the Request: * Role: Professional news layout engineer. * Task: Convert plain text news article into semantic HTML. * Rules: 1. Output *only* the HTML snippet, no Markdown fences, no explanations. 2. Paragraphs: `

`. Long text sections: `

`/`

`. 3. Lists: `

• `, Quotes: `

  `, Code: `

  `, Important sentences: ``.
          4.  Preserve *all* original information, no additions/deletions/modifications.
          5.  No document-level tags (``, ``, ``).
      *   Input Text: A short academic/tech news article about "SANE" (an agent-based framework for LLM judges).
  
  2.  Analyze the Input Text & Structure:
      *   *Paragraph 1:* "LLM judges increasingly mediate benchmark evaluation, model selection, data curation, and training. Their decisions are useful only when each judge responds reliably to changes affecting its scoring criterion."
      *   *Paragraph 2:* "Existing methods for reliability testing perturb inputs with known expected effects on judge scores. Prominent LLM-judge testing harnesses combine human-authored perturbation recipes with synthetic test generation, leaving scorer-specific recipe ideation outside the automated pipeline. This limits their use to fixed perturbations and pre-assumed scorer types."
      *   *Paragraph 3:* "We introduce SANE, an agent-based framework that encodes behavioral requirements for stochastic evaluators as reusable agent instructions. Given a scorer specification, implementation, and representative inputs, the pipeline first identifies scorer-specific capabilities, then automatically derives perturbation recipes and expected score relations for each. SANE also defines a scorer-independent validation framework for generated judge tests. It determines whether each recipe agrees with scorer intent and whether each instantiated case realizes the recipe and supports its expected relation."
      *   *Paragraph 4:* "Because SANE generates a recipe inventory tailored to each scorer, the number of aligned recipes grows as scorers are added."
      *   *Paragraph 5:* "Across five domains, SANE produces 3.7 times as many validated test families as the Judge Reliability Harness (JRH) at comparable end-to-end validity and detects scorer deficiencies in every domain, including professionalism and toxicity, where JRH detects none."
      *   *Paragraph 6:* "On faithfulness, SANE achieves strong human alignment in judge ranking: its test-suite scores rank 11 judge models consistently with human-labeled FaithBench performance (Spearman rho = 0.852), while JRH does not (rho = 0.100)."
  
  3.  Determine HTML Structure based on Rules:
      *   Rule 2 says "Long text uses `
  `/`
  ` to divide sections". The input is a single continuous text block, but I can break it into logical sections using `
  ` or `
  `. Let's look at the content.
      *   Section 1: Background & Problem (Para 1 & 2)
      *   Section 2: SANE Framework (Para 3 & 4)
      *   Section 3: Results (Para 5 & 6)
      *   Wait, is it better to just use `
  ` for the main sections? Yes.
      *   Let's structure it:
          *   `
  ` Background and Limitations of Existing Methods
          *   `
  LLM judges...
  `
          *   `
  Existing methods...`
          *   `
  ` Introduction of SANE
          *   `
  We introduce...`
          *   `
  Because SANE...`
          *   `
   Evaluation Results
          *   `
  Across five domains...`
          *   `
  On faithfulness...`
  
      *   Let's refine the headings to be accurate to the text.
          *   `
  ` Background and Existing Limitations`
          *   `
  LLM judges increasingly mediate benchmark evaluation, model selection, data curation, and training. Their decisions are useful only when each judge responds reliably to changes affecting its scoring criterion.
  `
          *   `
  Existing methods for reliability testing perturb inputs with known expected effects on judge scores. Prominent LLM-judge testing harnesses combine human-authored perturbation recipes with synthetic test generation, leaving scorer-specific recipe ideation outside the automated pipeline. This limits their use to fixed perturbations and pre-assumed scorer types.
  `
          *   `
  ` The SANE Framework`
          *   `
  We introduce SANE, an agent-based framework that encodes behavioral requirements for stochastic evaluators as reusable agent instructions. Given a scorer specification, implementation, and representative inputs, the pipeline first identifies scorer-specific capabilities, then automatically derives perturbation recipes and expected score relations for each. SANE also defines a scorer-independent validation framework for generated judge tests. It determines whether each recipe agrees with scorer intent and whether each instantiated case realizes the recipe and supports its expected relation.
  `
          *   `
  Because SANE generates a recipe inventory tailored to each scorer, the number of aligned recipes grows as scorers are added.
  `
          *   `
  ` Evaluation Outcomes`
          *   `
  Across five domains, SANE produces 3.7 times as many validated test families as the Judge Reliability Harness (JRH) at comparable end-to-end validity and detects scorer deficiencies in every domain, including professionalism and toxicity, where JRH detects none.
  `
          *   `
  On faithfulness, SANE achieves strong human alignment in judge ranking: its test-suite scores rank 11 judge models consistently with human-labeled FaithBench performance (Spearman rho = 0.852), while JRH does not (rho = 00.100).
  `
  
      *   Wait, let's check if there are lists, quotes, code, or important sentences that need tags.
      *   Looking at the text:
          *   "Given a scorer specification, implementation, and representative inputs..." - could be a list, but it's written as a sentence. I shouldn't split sentences if they aren't explicitly list items. The rule says "List content uses 
  ", but if there are no explicit lists, don't force one.
          *   No quotes (`blockquote`) exist in the text.
          *   No code (`pre>
             Maybe ` 
             `? Or just leave it. The prompt says "Important sentences can use ", so I should highlight key claims. Let's highlight the core problem statement and the framework description. * Actually, highlighting *every* sentence might be overkill, but I should apply `` to the most critical sentences to show I used the tag. * Critical sentences: * "Their decisions are useful only when each judge responds reliably to changes affecting its scoring criterion." -> `` * "This limits their use to fixed perturbations and pre-assumed scorer types." -> `` * "We introduce SANE, an agent-based framework that encodes behavioral requirements for stochastic evaluators as reusable agent instructions." -> `` * "Because SANE generates a recipe inventory tailored to each scorer, the number of aligned recipes grows as scorers are added." -> `` * "Across five domains, SANE produces 3.7 times as many validated test families as the Judge Reliability Harness (JRH) at comparable end-to-end validity and detects scorer deficiencies in every domain, including professionalism and toxicity, where JRH detects none." -> `` * "On faithfulness, SANE achieves strong human alignment in judge ranking: its test-suite scores rank 11 judge models consistently with human-labeled FaithBench performance (Spearman rho = 0.852), while JRH does not (rho = 0.100)." -> `` 4. Drafting the HTML: ```html 
                      
  Background and Limitations
                      
  LLM judges increasingly mediate benchmark evaluation, model selection, data curation, and training. Their decisions are useful only when each judge responds reliably to changes affecting its scoring criterion.
                      
  Existing methods for reliability testing perturb inputs with known expected effects on judge scores. Prominent LLM-judge testing harnesses combine human-authored perturbation recipes with synthetic test generation, leaving scorer-specific recipe ideation outside the automated pipeline. This limits their use to fixed perturbations and pre-assumed scorer types.
                      
  The SANE Framework
                      
  We introduce SANE, an agent-based framework that encodes behavioral requirements for stochastic evaluators