Relation extraction from aviation accident narratives is challenging because genuine safety relations are sparse among possible concept pairs.
This study evaluates a decomposed framework separating relation classification from concept identification.
A benchmark of 300 National Transportation Safety Board (NTSB) narratives containing 2,524 unique concept-text nodes was expanded into 21,038 ordered candidate pairs, with only 248 (1.18%) representing RESULTS_IN or CONTRIBUTES_TO relations.
A TF-IDF Linear Support Vector Machine (SVM) and weighted RoBERTa classifier were developed using an independent graph-annotated resource with hard-negative sampling and frozen before sealed evaluation.
Development Macro-F1 scores of 0.4738 (SVM) and 0.4809 (RoBERTa) declined to 0.2741 and 0.2420 under exhaustive evaluation, with Matthews correlation coefficients of 0.0128 and 0.0197.
False positives dominated: the SVM recovered 96 reference relations but generated 7,018 false positives, while RoBERTa recovered 125 but generated 9,824.
Post-test diagnostics revealed overlapping RoBERTa confidence distributions and false positives even within valid relation-category directions.
Complementary end-to-end SmolLM2 evaluation identified missed and unsupported relations alongside malformed outputs.
These findings demonstrate that known concept endpoints do not guarantee reliable relation extraction and that sampled-negative development performance may poorly reflect exhaustive evaluation.
Controlling false positives and managing candidate-space sparsity remain essential for aviation safety knowledge extraction.