Converting Web Pages Mockups to HTML using Machine Learning

Tiago Bouças, António Esteves

2020

Abstract

Converting Web pages mockups to code is a task that developers typically perform. Due to the time required to accomplish this task, the time available to devote to application logic is reduced. So, the main goal of the present work was to develop deep learning models to automatically convert mockups of Web graphical interfaces into HTML, CSS and Bootstrap code. The trained model must be deployed as a Web application. Two deep learning models were built, resulting from two different approaches to integrate in the Web application. The first approach uses a hybrid architecture with a convolutional neuronal network (CNN) and two recurrent networks (RNNs), following the encoder-decoder architecture commonly adopted in image captioning. The second approach is focused on the spatial component of the problem being addressed, and includes the YOLO network and a layout algorithm. Testing with the same dataset, the prediction’s correction achieved with the first approach was 71.30%, while the second approach reached 88.28%. The first contribution of the present paper is the development of a rich dataset with Web pages GUI sketches and their captions. There was no dataset with sufficiently complex GUI sketches before we start this work. A second contribution was applying YOLO to detect and localize HTML elements, and the development of a layout algorithm that allows us to convert the YOLO result into code. It is a completely different approach from what is found in the related work. Finally, we achieved with YOLO-based architecture a prediction’s correction higher than reported in the literature.

Paper Citation

in Harvard Style

Bouças T. and Esteves A. (2020). Converting Web Pages Mockups to HTML using Machine Learning.In Proceedings of the 16th International Conference on Web Information Systems and Technologies - Volume 1: WEBIST, ISBN 978-989-758-478-7, pages 217-224. DOI: 10.5220/0010116302170224

in Bibtex Style

@conference{webist20,
author={Tiago Bouças and António Esteves},
title={Converting Web Pages Mockups to HTML using Machine Learning},
booktitle={Proceedings of the 16th International Conference on Web Information Systems and Technologies - Volume 1: WEBIST,},
year={2020},
pages={217-224},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010116302170224},
isbn={978-989-758-478-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Web Information Systems and Technologies - Volume 1: WEBIST,
TI - Converting Web Pages Mockups to HTML using Machine Learning
SN - 978-989-758-478-7
AU - Bouças T.
AU - Esteves A.
PY - 2020
SP - 217
EP - 224
DO - 10.5220/0010116302170224