Authors:
Marwa Masmoudi
1
;
Salma Jarraya
1
;
2
and
Mohamed Hammami
1
;
3
Affiliations:
1
Mir@cl Laboratory, University of Sfax, Tunisia
;
2
CS Department, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia
;
3
Department of Computer Science, Faculty of Science, Sfax, Tunisia
Keyword(s):
Multimodal Approach, Meltdown Crisis Detection, Autism, Behavior Analysis.
Abstract:
This paper presents an innovative method for addressing the challenge of recognizing and responding to meltdown crises in autistic children. It focuses on integrating information from emotional and physical modalities, employing multimodal fusion with an emphasis on the early fusion technique. Existing literature outlines three fusion techniques – early, late, and hybrid fusion, each with unique advantages. Due to the distinct nature of datasets representing emotions and physical activities, late and hybrid fusion were considered impractical. Therefore, the paper adopts the early fusion method and introduces a Multi-modal CNN model architecture for efficient meltdown crisis recognition. The architecture comprises three Convolution layers, Max-pooling Layers, a Fully Connected (FC) layer, and Softmax activation for classification. The decision to opt for early fusion is driven by the inconsistent detection of children’s faces in all video frames, resulting in two different output size
s for emotion and physical activity systems. The presented pseudo-code outlines the architecture development steps. The proposed model’s efficiency is highlighted by its outstanding recognition rate and speed, making it the preferred choice for the time-sensitive Smart-AMD (Smart-Autistic Meltdown Detector) System. Beyond technical aspects, the model aims to enhance the well-being of autistic children by promptly recognizing and alerting caregivers to abnormal behaviors during a meltdown crisis. This paper introduces a comprehensive system that integrates advanced technology and a profound understanding of autism, offering timely and effective support to those in need.
(More)