Cognitive workload reflects the mental effort required during task performance and is central to the design of adaptive human-machine systems.
The use of biosignals to measure cognitive workload has been extensively researched and documented; however, studies examining the effects of combining heterogeneous biosignal modalities for this purpose remain limited.
To provide insight into this area, we developed a unified, modality-agnostic, hierarchical Transformer-based architecture to process heterogeneous biosignal modalities within a single model.
We use this framework in a pilot study evaluating all $31$ possible combinations of five modalities: Electrocardiogram (ECG), Electrodermal Activity (EDA), Respiration (RESP), Peripheral Oxygen Saturation (SpO$_2$), and Electroencephalogram (EEG), under leave-one-subject-out validation across three cognitively distinct tasks: abstract reasoning (IQ), arithmetic problem solving (MATH), and a game task (GAME).
In this pilot setting, the results suggest that:
- EEG is the strongest single modality, ranking highest in IQ, GAME, and the pooled ALL setting, where samples from all three tasks are combined;
- adding more modalities does not consistently improve performance;
- the full five-modality combination achieves the highest Average score of $73.02%$ on IQ and $68.08%$ when the Average scores are averaged over the four evaluation settings: IQ, MATH, GAME, and ALL;
- the proposed method reduces model size by approximately $50%$ compared with late-fusion alternatives while maintaining a lower inference time.