Multimodal Approach Based on Autistic Child Behavior Analysis for Meltdown Crisis Detection

Marwa Masmoudi, Salma Jarraya, Salma Jarraya, Mohamed Hammami, Mohamed Hammami

2024

Abstract

This paper presents an innovative method for addressing the challenge of recognizing and responding to meltdown crises in autistic children. It focuses on integrating information from emotional and physical modalities, employing multimodal fusion with an emphasis on the early fusion technique. Existing literature outlines three fusion techniques – early, late, and hybrid fusion, each with unique advantages. Due to the distinct nature of datasets representing emotions and physical activities, late and hybrid fusion were considered impractical. Therefore, the paper adopts the early fusion method and introduces a Multi-modal CNN model architecture for efficient meltdown crisis recognition. The architecture comprises three Convolution layers, Max-pooling Layers, a Fully Connected (FC) layer, and Softmax activation for classification. The decision to opt for early fusion is driven by the inconsistent detection of children’s faces in all video frames, resulting in two different output sizes for emotion and physical activity systems. The presented pseudo-code outlines the architecture development steps. The proposed model’s efficiency is highlighted by its outstanding recognition rate and speed, making it the preferred choice for the time-sensitive Smart-AMD (Smart-Autistic Meltdown Detector) System. Beyond technical aspects, the model aims to enhance the well-being of autistic children by promptly recognizing and alerting caregivers to abnormal behaviors during a meltdown crisis. This paper introduces a comprehensive system that integrates advanced technology and a profound understanding of autism, offering timely and effective support to those in need.

Paper Citation

in Harvard Style

Masmoudi M., Jarraya S. and Hammami M. (2024). Multimodal Approach Based on Autistic Child Behavior Analysis for Meltdown Crisis Detection. In Proceedings of the 19th International Conference on Software Technologies - Volume 1: ICSOFT; ISBN 978-989-758-706-1, SciTePress, pages 339-350. DOI: 10.5220/0012724800003753

in Bibtex Style

@conference{icsoft24,
author={Marwa Masmoudi and Salma Jarraya and Mohamed Hammami},
title={Multimodal Approach Based on Autistic Child Behavior Analysis for Meltdown Crisis Detection},
booktitle={Proceedings of the 19th International Conference on Software Technologies - Volume 1: ICSOFT},
year={2024},
pages={339-350},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012724800003753},
isbn={978-989-758-706-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 19th International Conference on Software Technologies - Volume 1: ICSOFT
TI - Multimodal Approach Based on Autistic Child Behavior Analysis for Meltdown Crisis Detection
SN - 978-989-758-706-1
AU - Masmoudi M.
AU - Jarraya S.
AU - Hammami M.
PY - 2024
SP - 339
EP - 350
DO - 10.5220/0012724800003753
PB - SciTePress