Text-to-SQL Meets the Real-World

Eduardo Nascimento, Eduardo Nascimento, Grettel García, Lucas Feijó, Wendy Victorio, Wendy Victorio, Yenier Izquierdo, Aiko R. de Oliveira, Gustavo Coelho, Melissa Lemos, Melissa Lemos, Robinson Garcia, Luiz Leme, Marco Casanova, Marco Casanova

2024

Abstract

Text-to-SQL refers to the task defined as “ given a relational database D and a natural language sentence S that describes a question on D, generate an SQL query Q over D that expresses S”. Numerous tools have addressed this task with relative success over well-known benchmarks. Recently, several LLM-based text-to-SQL tools, that is, text-to-SQL tools that explore Large Language Models (LLMs), emerged that outperformed previous approaches. When adopted for industrial-size databases, with a large number of tables, columns, and foreign keys, the performance of LLM-based text-to-SQL tools is, however, significantly less than that reported for the benchmarks. This paper then investigates how a selected set of LLM-based text-to-SQL tools perform over two challenging databases, an openly available database, Mondial, and a proprietary industrial database. The paper also proposes a new LLM-based text-to-SQL tool that combines features from tools that performed well over the Spider and BIRD benchmarks. Then, the paper describes how the selected tools and the proposed tool, running under GPT-3.5 and GPT-4, perform over the Mondial and the industrial databases over a suite of 100 carefully defined natural language questions that are closely related to those observed in practice. It concludes with a discussion of the results obtained.

Paper Citation

in Harvard Style

Nascimento E., García G., Feijó L., Victorio W., Izquierdo Y., R. de Oliveira A., Coelho G., Lemos M., Garcia R., Leme L. and Casanova M. (2024). Text-to-SQL Meets the Real-World. In Proceedings of the 26th International Conference on Enterprise Information Systems - Volume 1: ICEIS; ISBN 978-989-758-692-7, SciTePress, pages 61-72. DOI: 10.5220/0012555200003690

in Bibtex Style

@conference{iceis24,
author={Eduardo Nascimento and Grettel García and Lucas Feijó and Wendy Victorio and Yenier Izquierdo and Aiko R. de Oliveira and Gustavo Coelho and Melissa Lemos and Robinson Garcia and Luiz Leme and Marco Casanova},
title={Text-to-SQL Meets the Real-World},
booktitle={Proceedings of the 26th International Conference on Enterprise Information Systems - Volume 1: ICEIS},
year={2024},
pages={61-72},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012555200003690},
isbn={978-989-758-692-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 26th International Conference on Enterprise Information Systems - Volume 1: ICEIS
TI - Text-to-SQL Meets the Real-World
SN - 978-989-758-692-7
AU - Nascimento E.
AU - García G.
AU - Feijó L.
AU - Victorio W.
AU - Izquierdo Y.
AU - R. de Oliveira A.
AU - Coelho G.
AU - Lemos M.
AU - Garcia R.
AU - Leme L.
AU - Casanova M.
PY - 2024
SP - 61
EP - 72
DO - 10.5220/0012555200003690
PB - SciTePress