![](bg8.png)
by a significant margin. By comparing TLPRB and
RoBERTa Embedding, it can be seen that further pre-
training RoBERTa by MLM on article sequences is
significantly improving the performance, indicating
that TLPRB is learning contexutality in sequences of
titles and category names, which benefits prediction
of neighboring new nodes.
TLPFT is better than the baseline models, indi-
cating that semantic similarity captured by FastText,
which is capable of embedding out-of-vocabulary
words in article texts by k-gram decomposition, and
TLPFT is even surpassing RoBERTa Embedding.
However, TLPFT is not as effective as TLPRB, in-
dicating that embeddings of FastText do not capture
contexts in article sequences, while TLPRB is further
pretrained by a large number of article sequences, to
learn semantic similarity and contextual relationship
between neighboring articles.
Figure 6 shows micro-averaged Precision@k, Re-
call@k, and F1@k curves with k=5, 10, 50, 100
on the three datasets. Combining the results on the
aggregated performance by the AUC scores and the
ranking performance by the Top-K results, our overall
conclusion is that TLPRB performs best in both AUC
and Top-K evaluations, and TLPFT performs the next
in the AUC evaluation.
6 CONCLUSION AND FUTURE
WORK
In this paper, we proposed a new method for predict-
ing article links in Wikipedia, which utilizes tempo-
ral random walk to generate graph embeddings. Our
graph embedding model is based on temporal random
walk, biased by temporal feature and semantic rela-
tionships of article texts between node pairs. We eval-
uated on three temporal graph datasets extracted from
Wikipedia dump. Our experimental results show that
our proposed model TLPRB outperforms the base-
lines and simple RoBERTa-based model in this tem-
poral link prediction on versioned articles.
For future work, we consider that the current defi-
nition of common categories is based on identical cat-
egory names. Semantic similarity and hierarchical re-
lationships between categories can be explored.
REFERENCES
Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T.
(2017). Enriching word vectors with subword infor-
mation. Transactions of the Association for Computa-
tional Linguistics, 5:135–146.
Crichton, G., Guo, Y., Pyysalo, S., and Korhonen, A.
(2018). Neural networks for link prediction in realis-
tic biomedical graphs: a multi-dimensional evaluation
of graph embedding-based approaches. BMC bioin-
formatics, 19(1):1–11.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.
(2019). BERT: Pre-training of deep bidirectional
transformers for language understanding. In Proceed-
ings of the 2019 Conference of the North American
Chapter of the Association for Computational Lin-
guistics: Human Language Technologies, Volume 1
(Long and Short Papers), pages 4171–4186.
Divakaran, A. and Mohan, A. (2019). Temporal link pre-
diction: a survey. New Generation Computing, pages
1–46.
Goyal, P., Kamra, N., He, X., and Liu, Y. (2018). Dyngem:
Deep embedding method for dynamic graphs. arXiv
preprint arXiv:1805.11273.
Grover, A. and Leskovec, J. (2016). node2vec: Scal-
able feature learning for networks. In Proceedings
of the 22nd ACM SIGKDD international conference
on Knowledge discovery and data mining, pages 855–
864.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D.,
Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov,
V. (2019). Roberta: A robustly optimized bert pre-
training approach. arXiv preprint arXiv:1907.11692.
Nguyen, G. H., Lee, J. B., Rossi, R. A., Ahmed, N. K., Koh,
E., and Kim, S. (2018). Continuous-time dynamic net-
work embeddings. In Companion Proceedings of the
The Web Conference 2018, pages 969–976.
Perozzi, B., Al-Rfou, R., and Skiena, S. (2014). Deepwalk:
Online learning of social representations. In Proceed-
ings of the 20th ACM SIGKDD international confer-
ence on Knowledge discovery and data mining, pages
701–710.
Yang, Y., Lichtenwalter, R. N., and Chawla, N. V. (2015).
Evaluating link prediction methods. Knowledge and
Information Systems, 45(3):751–782.
Yue, X., Wang, Z., Huang, J., Parthasarathy, S., Moosav-
inasab, S., Huang, Y., Lin, S. M., Zhang, W., Zhang,
P., and Sun, H. (2020). Graph embedding on biomed-
ical networks: methods, applications and evaluations.
Bioinformatics, 36(4):1241–1251.
Zesch, T., Gurevych, I., and M
¨
uhlh
¨
auser, M. (2007). An-
alyzing and accessing wikipedia as a lexical seman-
tic resource. Data Structures for Linguistic Resources
and Applications, 197205.
Zhou, L., Yang, Y., Ren, X., Wu, F., and Zhuang, Y. (2018).
Dynamic network embedding by modeling triadic clo-
sure process. In Proceedings of the AAAI Conference
on Artificial Intelligence, volume 32.
KDIR 2021 - 13th International Conference on Knowledge Discovery and Information Retrieval
94