Large language models increasingly generate structured outputs consumed by programs and downstream systems, where semantic correctness alone does not guarantee reliable execution.
SERIAL-7: Korean-English Benchmark with Executable Verification
We introduce SERIAL-7, a Korean-English benchmark built around an executable verification framework for machine-verifiable generation across seven representations.
Three-Stage Pipeline: Derive, Realize, Preserve
- Semantic derivation (Derive)
- Constrained artifact generation (Realize)
- Cross-representation preservation (Preserve)
Using 177 validated canonical artifacts, SERIAL-7 separates semantic derivation (Derive), constrained artifact generation (Realize), and cross-representation preservation (Preserve).
43 Atomic Representation Constraints and Executable Checks
Its 43 atomic representation constraints are paired with executable checks over format, structure, and task-specific semantics, enabling progressively stricter verification from representation compliance to full artifact validity.
Results Across Ten Models
- REALIZE-L3 HSR ranges from 42.0% to 93.9% across ten models.
- Full Compliance exposes additional artifact-level failures beyond correct final answers and valid representations.
These results show that reliable structured generation requires explicit, executable verification of how artifacts are realized and preserved across representations.