Multimodal fusion for anticipating human decision performance

Article Type

Research Article

Publication Title

Scientific Reports

Abstract

Anticipating human decisions while performing complex tasks remains a formidable challenge. This study proposes a multimodal machine-learning approach that leverages image features and electroencephalography (EEG) data to predict human response correctness in a demanding visual searching task. Notably, we extract a novel set of image features pertaining to object relationships using the Segment Anything Model (SAM), which enhances prediction accuracy compared to traditional features. Additionally, our approach effectively utilizes a combination of EEG signals and image features to streamline the feature set required for the Random Forest Classifier (RFC) while maintaining high accuracy. The findings of this research hold substantial potential for developing advanced fault alert systems, particularly in critical decision-making environments such as the medical and defence sectors.

DOI

10.1038/s41598-024-63651-2

Publication Date

12-1-2024

Comments

Open Access; Gold Open Access; Green Open Access

Link to Full Text

Share

COinS