Using Google Books Ngram in Detecting Linguistic Shifts over Time

Alaa El-Ebshihy, Nagwa El-Makky, Khaled Nagi

2018

Abstract

The availability of large historical corpora, such as Google Books Ngram, makes it possible to extract various meta information about the evolution of human languages. Together with advances in machine learning techniques, researchers recently use the huge corpora to track cultural and linguistic shifts in words and terms over time. In this paper, we develop a new approach to quantitatively recognize semantic changes of words during the period between 1800 and 1990. We use the state-of-the-art FastText approach to construct word embedding for Google Books Ngram corpus for the decades within the time period 1800-1990. We use a time series analysis to identify words that have a statistically significant change in the period between 1900 and 1990. We conduct a performance evaluation study to compare our approach against related work, we show that our system is more robust against morphological language variations.

Paper Citation

in Harvard Style

El-Ebshihy A., El-Makky N. and Nagi K. (2018). Using Google Books Ngram in Detecting Linguistic Shifts over Time. In Proceedings of the 10th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2018) - Volume 1: KDIR; ISBN 978-989-758-330-8, SciTePress, pages 332-339. DOI: 10.5220/0007188703320339

in Bibtex Style

@conference{kdir18,
author={Alaa El-Ebshihy and Nagwa El-Makky and Khaled Nagi},
title={Using Google Books Ngram in Detecting Linguistic Shifts over Time},
booktitle={Proceedings of the 10th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2018) - Volume 1: KDIR},
year={2018},
pages={332-339},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0007188703320339},
isbn={978-989-758-330-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 10th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2018) - Volume 1: KDIR
TI - Using Google Books Ngram in Detecting Linguistic Shifts over Time
SN - 978-989-758-330-8
AU - El-Ebshihy A.
AU - El-Makky N.
AU - Nagi K.
PY - 2018
SP - 332
EP - 339
DO - 10.5220/0007188703320339
PB - SciTePress