首页 > AI前沿 > The FlightFinder Occurrence Corpus: harmonising 128 aviation-safety source codes into one deduplicated cross-national accident dataset

The FlightFinder Occurrence Corpus: harmonising 128 aviation-safety source codes into one deduplicated cross-national accident dataset

arXiv自然语言 2026-09-07 06:19 5 阅读 查看原文

National aviation-investigation databases are deep but territorial; global compilations such as the Aviation Safety Network and the Bureau of Aircraft Accidents Archives are hand-curated, apply their own inclusion thresholds, and publish no per-record lineage.

The FlightFinder Occurrence Corpus harmonizes 279,485 occurrences and 131,882 attached narratives from 128 source codes -- national accident-investigation authorities, occurrence-reporting systems and four global compilations.

Records are mapped into one schema; each normalized record reduces to exactly one key from an ordered ladder -- event signature, registration and date, rounded position and date, or source and native identifier -- so a missing field never creates a link, and the key is the occurrence identifier.

Records the ladder cannot reach are scored in a blocked pass into an explicit merge table and applied only within an automatically approved score band or after a person confirms them.

Field conflicts follow a documented source ranking; one provenance row per contributing source -- its native identifier and URL -- is retained, with attribution and licence terms per source code.

Linkage error is measured on five adjudicated samples totalling 200 pairs. A recorded uniform random draw of 60 from the 3,441 merges the automatic band applies as it ships found no error; the exact two-sided 95% Clopper-Pearson upper bound on its false-merge rate is 6.0%, or 205 pairs. The measured error covers the scored band only; the ladder's deterministic joins are not adjudicated.

The corpus has no exposure denominators and inherits the under-reporting of its sources.

It is offered as a base for cross