Deep Learning-Based Models for Performing Multi-Instance Multi-Label Event Classification in Gameplay Footage

Etienne Julia, Marcelo Zanchetta do Nascimento, Matheus Faria, Rita Julia

2024

Abstract

In dynamic environments, like videos, one of the key pieces of information to improve the performance of autonomous agents are the events, since, in a broad manner, they represent the dynamic changes and interactions that happen in the environment. Video games stand out among the most suitable domains for investigating the effectiveness of machine learning techniques. Among the challenging activities explored in such research, it highlights that which endows the automatic game systems with the ability of identifying, in game footage, the events that other players, interacting with them, provoke in the game environment. Thus, the main contribution of this work is the implementation of deep learning models to perform MIML game event classification in gameplay footage, which are composed of: a data generator script to automatically produce multi-labeled frames from game footage (where the labels correspond to game events); a pre-processing method to make the frames generated by the script suitable to be used in the training datasets; a fine-tuned MobileNetV2 to perform feature extraction (trained from the pre-processed frames); an algorithm to produce MIML samples from the pre-processed frames (each sample corresponds to a set of frames named chunk); a deep neural network (NN) to perform classification of game events, which is trained from the chunks. In this investigation, Super Mario Bros is used as a case study.

Paper Citation

in Harvard Style

Julia E., Zanchetta do Nascimento M., Faria M. and Julia R. (2024). Deep Learning-Based Models for Performing Multi-Instance Multi-Label Event Classification in Gameplay Footage. In Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: VISAPP; ISBN 978-989-758-679-8, SciTePress, pages 110-121. DOI: 10.5220/0012358900003660

in Bibtex Style

@conference{visapp24,
author={Etienne Julia and Marcelo Zanchetta do Nascimento and Matheus Faria and Rita Julia},
title={Deep Learning-Based Models for Performing Multi-Instance Multi-Label Event Classification in Gameplay Footage},
booktitle={Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: VISAPP},
year={2024},
pages={110-121},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012358900003660},
isbn={978-989-758-679-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: VISAPP
TI - Deep Learning-Based Models for Performing Multi-Instance Multi-Label Event Classification in Gameplay Footage
SN - 978-989-758-679-8
AU - Julia E.
AU - Zanchetta do Nascimento M.
AU - Faria M.
AU - Julia R.
PY - 2024
SP - 110
EP - 121
DO - 10.5220/0012358900003660
PB - SciTePress