Towards the Use of AI-Based Tools for Systematic Literature Review

Lotﬁ Souiﬁ, Nesrine Khabou, Ismael Bouassida Rodriguez and Ahmed Hadj Kacem

ReDCAD Laboratory, ENIS, University of Sfax, Tunisia

ﬁ ﬁ

Keywords:

Artiﬁcial Intelligence, Systematic Literature Review Automation, GPT, Chatpdf, Pdf2gpt, Hipdf, SciSpace,

Easy-Peasy AI, DocAnalyzer AI.

Abstract:

The constant growth in the number of published research studies and their rapid rate of publication creates a

signiﬁcant challenge in identifying relevant studies for unbiased systematic reviews. To address this challenge,

artiﬁcial intelligence (AI) methods have been used since 2016 to improve the efﬁciency of scientiﬁc review

and synthesis. Nevertheless, the growth in the number of AI-powered tools dedicated to processing text-based

data has been remarkable since the introduction of generative pre-trained transformers by OpenAI in late 2022.

Moreover, alongside this development, ChatGPT, a language model that provides a user-friendly chatbot in-

terface, was introduced. The incorporation of this interactive feature has greatly enhanced the capability of

developers and end-users alike to effectively utilize and access ChatGPT. This study aims to investigate the

effectiveness of six AI-based tools namely Chatpdf, Pdf2gpt, Hipdf, SciSpace, Easy-peasy AI, and DocAna-

lyzer AI, developed utilizing ChatGPT technology. These tools will be evaluated in a speciﬁc scenario where

they are automated to carry out a particular step within a Systematic Literature Review. Furthermore, the

limitations associated with each tool will be analyzed, and strategies will be proposed to overcome them. Ad-

ditionally, this study aims to provide recommendations for researchers who intend to incorporate these tools

into their research processes.

1 INTRODUCTION

Artiﬁcial Intelligence (AI) is an expansive and inter-

disciplinary ﬁeld that integrates principles from com-

puter science and linguistics to develop computers ca-

pable of performing tasks typically reliant on human

intelligence (Sarker, 2022). Furthermore, in recent

studies conducted by various researchers (Yang et al.,

2023; Verma, 2023; Zhu et al., 2023), the increas-

ing signiﬁcance of utilizing AI in research has been

recognized. Researchers have come to acknowledge

the value and effectiveness of AI as a valuable tool

for data analysis and literature review. The system-

atic integration of AI into scientiﬁc research processes

can effectively enhance their efﬁciency. While still

nascent in development, AI has already showcased

considerable potential which could potentially rev-

olutionize research methodologies signiﬁcantly, par-

ticularly within the realm of non-coding applications

(Calo, 2017). As we explore the increasing capabil-

ities of artiﬁcial intelligence, it is evident that pos-

sessing deep technical skills is no longer a necessity

for leveraging AI to advance and enhance research.

One signiﬁcant advancement in natural language pro-

cessing is OpenAI’s Generative Pre-Trained Trans-

former (GPT), which demonstrates remarkable inno-

vation (Yenduri et al., 2023). GPT has been exten-

sively trained on large amounts of text data, allow-

ing it to effectively use ﬂexible language skills simi-

lar to human communication. By utilizing GPT’s core

abilities, tasks like chatbot programming and mod-

ern translation tools can greatly beneﬁt from its ex-

ceptional ability to create complex language nuances.

Moreover, this model can be ﬁne-tuned for various

language-related tasks, including but not limited to

language translation, text summarization, and text en-

hancement (Hassani and Silva, 2023). The most re-

cent iteration of the model, GPT-3, exhibits supe-

rior performance compared to its predecessors, mak-

ing it highly suitable for the dynamic ﬁeld of nat-

ural language processing. On November 30, 2022,

OpenAI introduced an AI-driven conversational agent

called ChatGPT (George and George, 2023). This an-

nouncement sparked great interest among experts and

researchers in artiﬁcial intelligence, leading them to

thoroughly assess and scrutinize the program’s abili-

ties. Furthermore, researchers have shown great inter-

est in the launch of ChatGPT because they are eager

to explore and experiment with this cutting-edge tech-

nology across various industries. As a result, there

Souiﬁ, L., Khabou, N., Rodriguez, I. and Kacem, A.

Towards the Use of AI-Based Tools for Systematic Literature Review.

DOI: 10.5220/0012467700003636

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 16th International Conference on Agents and Artiﬁcial Intelligence (ICAART 2024) - Volume 2, pages 595-603

ISBN: 978-989-758-680-4; ISSN: 2184-433X

595

has been signiﬁcant research conducted to examine

the wide array of potential applications where Chat-

GPT can be effectively employed (Ray, 2023; Sallam,

2023). In a study conducted by Patel et al. (Patel

and Lam, 2023), the researchers examined how Chat-

GPT could be utilized to produce hospital discharge

summaries in response to quick queries. The ﬁndings

showed that ChatGPT was proﬁcient at swiftly gener-

ating comprehensive summaries, offering the poten-

tial to decrease delays in patient discharges within

primary care settings while still preserving an appro-

priate level of detail. This automated process en-

ables physicians to allocate more time towards patient

care and education tasks. Furthermore, in the study

of (Jeblick et al., 2022) the researchers examined

ChatGPT’s effectiveness in streamlining radiology re-

ports with favorable outcomes. The generated reports

were highly technical and provided a comprehensive

overview with low perceived risks for patients. How-

ever, both studies also highlighted some instances of

inaccuracies within the system. In the case of the pa-

tient discharge summary, ChatGPT added additional

information that the authors haven’t requested (Pa-

tel and Lam, 2023). Similarly, the analysis of ra-

diology reports revealed potentially dangerous omis-

sions, such as the omission of important medical ﬁnd-

ings. These shortcomings suggest that a manual re-

view of the automated results would be necessary if

the system were to be implemented in clinical prac-

tice (Jeblick et al., 2022). Conversely, ﬁndings from a

study conducted by the European Patent Ofﬁce indi-

cate that around 30% of research and development in-

vestments are squandered as a result of reworking ex-

isting literature (Harhoff and Wagner, 2009). This un-

derscores the signiﬁcance of incorporating pertinent

scholarly papers when preparing grant proposals for

funding organizations like the National Science Foun-

dation and the National Institutes of Health. Failure

to provide pertinent literature can result in proposal

rejection. In traditional survey and review articles,

there is often a lack of systematic coverage of all

published work within a particular ﬁeld. Moreover,

basing new project concepts solely on these articles

can be misleading. To address these concerns, vari-

ous techniques have been developed, one being con-

ducting a systematic literature review (SLR) (Snyder,

2019). An SLR employs a methodology that identi-

ﬁes, evaluates, and synthesizes all available research

about a speciﬁc research question or topic area (Sny-

der, 2019). The goal of an SLR is to provide a trust-

worthy method for obtaining accurate, appropriate,

and unbiased information about a research topic (Gur-

buz and Tekinerdogan, 2018). The previously men-

tioned procedure provides a robust framework for the

Table 1: The steps of a systematic literature review (Keele

et al., 2007).

ID Category Step

SLR1

Need for a

review

Commissioning a

review

SLR2

Specifying the

research question(s)

SLR3

Developing a review

protocol

SLR4

Evaluating the review

protocol

SLR5

Conducting the

review

Identiﬁcation of

research

SLR6

Selection of primary

studies

SLR7

Study quality

assessment

SLR8

Data extraction and

monitoring

SLR9 Data synthesis

SLR10

Reporting the

review

Specifying dissemination

mechanisms

SLR11

Formatting the main

report

SLR12 Evaluating the report

methodical and unbiased examination of relevant lit-

erature, with a strong emphasis on accuracy. Since

2007, systematic reviews as introduced by keele et al.

(Keele et al., 2007) have been widely used in the area

of software engineering. Nevertheless, the process of

collecting, extracting, and synthesizing the data re-

quired for systematic reviews is recognized as chal-

lenging, error-prone, and labor-intensive in several

domains such as software engineering and medicine

(Marshall et al., 2016). It is generally known that it

takes more than one year from the last search to pub-

lication for an SLR study, and 2.5 − 6.5 years for a

primary study to be included in an SLR study (Jon-

nalagadda et al., 2015; Elliott et al., 2014). In addi-

tion, 23% of all SLR studies have become outdated

within 2 years of publication because reviewers fail

to include new evidence in their areas of interest (van

Dinter et al., 2021). However, The steps in the sys-

tematic review method are listed in Table 1 according

to (Keele et al., 2007).

According to Van Dinter et al. (van Dinter et al.,

2021), numerous research papers admit that one of the

main purposes behind automating systematic reviews

is to lessen the ﬁnancial burden linked with conduct-

ing these evaluations. Our research focuses on explor-

ing the application of AI-based methods to automate

the selection of primary studies during the SLR6 step,

as illustrated in Table 1.

ICAART 2024 - 16th International Conference on Agents and Artiﬁcial Intelligence

596

The subsequent sections of the paper are struc-

tured as follows. In Section 2, an examination of pre-

vious studies is presented. The tools used throughout

this research are deﬁned in Section 3. Section 4 out-

lines our conducted tests and presents the correspond-

ing results. Finally, in Section 5, we draw conclusions

from our ﬁndings and identify potential avenues for

further investigation in future research endeavors.

2 RELATED WORK

In this section, we will present some relevant stud-

ies that have been conducted to automate the steps of

SLR from a table 1. The procedure of selecting pri-

mary studies to conduct a systematic literature review,

commonly known as SLR6, has frequently been au-

tomated. This is primarily attributed to the consensus

among researchers that this step is exceedingly labori-

ous (Bannach-Brown et al., 2019; Sellak et al., 2015;

Tsafnat et al., 2018).

Several studies, such as (Mergel et al., 2015;

Scells et al., 2020; Scells et al., 2019), have

highlighted the automation of identifying research

(SLR5), particularly in creating the search query for

a systematic literature review, as one of the most au-

tomated steps in scholarly literature. This indicates

that formulating a search query for a systematic re-

view presents a considerable challenge.

To maximize the inclusion of relevant studies (Bi-

olchini et al., 2005) while excluding irrelevant ones

(Scells et al., 2019), researchers endeavor to establish

explicit criteria for their study. These criteria serve

as a basis for selecting articles that meet speciﬁc re-

quirements and are eligible for review. In his research,

(Felizardo et al., 2012) presents a novel method that

utilizes decision tree-based approach to automatically

generate queries in the ﬁeld of legal eDiscovery. Sim-

ilar to other conceptual and objective approaches, this

innovative technique relies on initial studies as ref-

erences to determine the keywords and their appro-

priate placement within the query. However, using

this methodology for literature searches during sys-

tematic reviews poses a challenge as it necessitates in-

cluding a considerably larger number of seed studies

than what is typically feasible. Conversely, leverag-

ing techniques like machine learning and natural lan-

guage processing can greatly enhance and automate

the systematic review process. furthermore, Ghafari

et al. (Ghafari et al., 2012) made a signiﬁcant con-

tribution by introducing a federated search tool that

offers an automated integrated search function across

major databases in the ﬁeld of Software Engineer-

ing. The ﬁndings of the case study evince that their

approach not only diminishes the time required to

perform SLR and simpliﬁes its search process, but

also enhances its dependability and leads to an up-

ward trend in the utilization of SLRs. In their re-

search, (Hannousse and Yahiouche, 2022) introduced

a unique strategy for creating a partially automated

system to reduce the manual labor required for paper

processing. This novel approach combines unsuper-

vised and semi-supervised machine learning models,

effectively using both approaches’ strengths. Addi-

tionally, this system makes use of a domain ontol-

ogy to improve accuracy and efﬁciency. Felizardo

et al. (Felizardo et al., 2012) conducted a study in

which they mechanized the assessment of the selec-

tion of primary studies. These studies were identiﬁed

as the sole studies carrying out a study quality assess-

ment, known as SLR7. The authors delineate that the

process of conducting a review comprises two steps,

namely selection execution and information extrac-

tion. The selection execution phase is further divided

into three sub-steps, with the last one, namely the se-

lection review step, being the primary focus of their

study, i.e., the study quality assessment step, SLR7.

The authors highlight that reviewers may perform this

step by employing quality criteria to ensure that rele-

vant studies are not excluded prematurely if required.

Finally, the automation of the Data extraction and

monitoring step (SLR8) has been implemented in ﬁve

studies. The underlying reason for automating this

step is that the data extraction process is commonly a

labor-intensive task (Aliyu et al., 2018; Elamin et al.,

2009). Studies have indicated a signiﬁcant incidence

of inaccuracies in the manual data extraction pro-

cess, which can be attributed to human-related aspects

such as insufﬁcient time and resources, inconsisten-

cies, and blunders resulting from monotony.

It is important to highlight that our investigation re-

vealed a lack of previous research on the utilization

of GPT-based tools for automating systematic litera-

ture reviews. As such, our study seeks to address this

gap by examining the feasibility and potential beneﬁts

of employing GPT-based tools in automating SLRs.

3 BASIC CONCEPTS

In this section, we provide a comprehensive deﬁnition

of the chosen tools along with an examination of their

functionalities and limitations. It should be noted that

a detailed technical analysis was not possible due to

the lack of information available on either the ofﬁcial

websites or in existing literature.

Towards the Use of AI-Based Tools for Systematic Literature Review

597

3.1 Chatpdf

Chatpdf is a platform that is powered by advanced ar-

tiﬁcial intelligence technology. It facilitates users to

effectively and proﬁciently extract information from

voluminous PDF ﬁles, which may include research

papers, books, etc. (ToolsPedia.io, 2023). The two

main access options for Chatpdf are:

• Free Access:

– 120 Pages/PDF

– 10 MB/PDF

– 3 PDFs/day

– 50 Questions/day

• Paid Access: ($5/month)

– 2,000 Pages/PDF

– 32 MB/PDF

– 50 PDFs/day

– 1000 Questions/day

3.2 Pdf2gpt

Pdf2gpt is a novel artiﬁcial intelligence tool speciﬁ-

cally designed to extract information from long PDF

documents using the Generative Pre-trained Trans-

former (GPT) model. It is designed to simplify the

process of extracting important data and key points

from long PDF documents, allowing users to under-

stand the core content without having to go through

the entire document. The interface is user-friendly

and allows users to either upload the PDF ﬁle or pro-

vide the URL for summarization, providing easy ac-

cess to the tool’s features (Theresanaiforthat, 2023).

Basically, Pdf2gpt offers two access options:

• Free Access:

– 15 Pages/PDF

– 40 MB/PDF

– 7500 Words/pdf

– The user can access two lengthy PDFs for free

by connecting to their account. Each account

has the option to obtain one instance of this of-

fer.

• Paid Access: ($5/month)

– 200 Pages/PDF

– 40 MB/PDF

– 75000 Words/pdf

3.3 Hipdf

Hipdf offers a convenient and cost-free method for

generating brief overviews of PDF documents. One

notable feature is ”Chat with PDF,” which employs

ChatGPT technology to efﬁciently condense a doc-

ument by producing synopses, highlighting key sec-

tions and keywords, fostering effortless comprehen-

sion. This presents an optimal approach towards en-

riching the educational process, elucidating intricate

ideas, acquiring fresh perspectives, and summarizing

lengthy textbooks. (wondershare, 2023). Basically,

there are two ways to access Hipdf:

• Free Access:

– 100 Pages/PDF

– 5 Batch Processing

– All PDF tools except OCR

• Paid Access: ($5/month)

– 2,000 Pages/PDF

– Desktop applications

– Access to all features, including OCR & AI

tools

– No Batch Processing limit

– No adverts.

3.4 SciSpace

The SciSpace platform, according to Khan et al.

(Khan et al., 2019),provides a comprehensive view

of data shared across many geographically dispersed

High-Performance Computing (HPC) data centers

through a single workspace that facilitates direct data

access to achieve optimal performance when read-

ing or writing data within the appropriate data cen-

ter namespace. The effectiveness of this approach is

determined by the use of real scientiﬁc datasets and

applications. The platform offers a comprehensive,

searchable database of more than 270 million scien-

tiﬁc papers, authors, subjects, journals, and confer-

ences (theresanaiforthat, 2023). There are no limita-

tions on the usage of SciSpace, except for a maximum

ﬁle size limit of 100 MB. Additionally, it is available

free of charge.

3.5 Easy-Peasy AI

Easy-peasy AI is an AI-powered content assistant

that helps users create original and polished content

quickly. With a signiﬁcant 10x increase in speed,

the software provides more than 80 AI copywriting

templates to assist in creating compelling and profes-

sional content. It also includes tools for generating

AI images and transcribing audio accurately and ef-

ﬁciently (theresanaiforthat, 2023). The platform fea-

tures a chatbot called ”Chat with MARKy” which of-

fers simple PDF manipulation capabilities. During

ICAART 2024 - 16th International Conference on Agents and Artiﬁcial Intelligence

598

Table 2: Comparison of all the tools in general.

Tools Limits Other function Payment

Chatpdf

Pages

-Pdfs

None monthly

Pdf2gpt Pages None monthly

Hipdf

Pages

-Tokens

Pdf and

Image

tools

monthly

Yearly

SciSpace None

Literature

review-

Paraphrase

None

Easy-peasy

Pages

AI Transcription-

Templates

monthly

Yearly

DocAnalyzer

number

of Pdfs

None

monthly

Yearly

our testing process, we found that there were limi-

tations when uploading large PDF ﬁles (e.g., a 400-

page PDF). However, there are no restrictions on the

number or size of PDFs other than exclusive access to

GPT4 for premium customers only.

3.6 DocAnalyzer AI

DocAnalyzer AI Is an intelligent tool that provides

interactive and contextually aware functionality when

working with PDF ﬁles. It utilizes cutting-edge

AI techniques to analyze documents effectively and

promptly respond to user inquiries. The system thor-

oughly understands the questions asked and delivers

accurate answers without any delay. Its user interface

is uncomplicated, private, and continuously improv-

ing.

• Free Access:

– 3 PDFs/day

– Automatically deleted documents after 7 days

of inactivity

• Paid Access: ($5/month)

– No limit on daily uploads

– Without daily question limitations (up to

10,000)

– 50 MB/PDF

– 1 GB storage

In summary, the key differences between the 6

tools are illustrated in Table 2.

Table 3: Research query result.

Database SLR1 SLR5

(1)-Springer 361 207

(2)-ScienceDirect 902 660

(3)-ACM 98 46

(4)-WebofScience 3 3

(5)-IEEE Xplore 74 70

Total 1438 986

4 EXPERIMENTS

4.1 Context

In the given context, we conducted a thorough investi-

gation called a Systematic Literature Review to exam-

ine how Mobile Edge Computing impacts Quality of

Service in the domain of 5G. Our research query was

thoughtfully devised prior to conducting an exten-

sive exploration using ﬁve well-regarded databases:

Springer, ScienceDirect, ACM, WebofScience, and

IEEE Xplore. The ﬁndings are succinctly displayed

in Table 3, revealing that a comprehensive evaluation

yielded 1438 articles. This thorough analysis seeks to

provide valuable perspectives and make a substantial

contribution to the current scholarly discourse on this

subject. Conducting these initial steps is crucial for

ensuring a comprehensive research process. Based on

the data presented in Table 3, there has been a notable

decrease in the number of articles from the initial step

(SLR1) outlined in Table 1 to SLR5, although it is still

signiﬁcant.

Once we ﬁnished the initial three stages of exclu-

sion (namely eliminating duplicates, surveys, and in-

accessible articles), our attention turned to creating

separate PDF ﬁles for each database. These doc-

uments contain all the abstracts that remained af-

ter undergoing previous elimination rounds. The

databases involved in this process are listed below

along with the number of abstracts and pages associ-

ated with them: Springer (207/127 pages), ScienceDi-

rect (660/440 pages), ACM (46/29 pages), Webof-

Science (2/2 pages), and IEEE Xplore (70/46). In

collaboration with an expert, we proceeded to execute

the fourth step and acquired results for each database.

Subsequently, during this phase, all the tools at our

disposal were utilized to compare their respective out-

comes with the ﬁndings of our expert collaborators.

Our initial testing involved utilizing a query that in-

corporates all the predetermined keywords from our

systematic literature review:

• Q1. Name all abstracts, without explanation,

that related to one of the following keywords:

Towards the Use of AI-Based Tools for Systematic Literature Review

599

”QoS AND 5G AND service deployment mod-

els AND energy efﬁciency constraints” OR ”QoS

AND 5G AND service orchestration models AND

energy efﬁciency constraints”

The selected tools yielded no results for the query.

It is possible that the lack of results is due to the difﬁ-

culty in ﬁnding a single abstract containing all speci-

ﬁed keywords. This suggests that these keyword com-

binations are not commonly found together, making it

challenging to ﬁnd relevant articles on this topic. To

increase our chances, we decided to divide the query

into two sub-queries focused on different keywords

using the ”OR” operation. The new queries are:

• Q2. Name all abstracts, without explanation,

that related to one of the following keywords: ”

5G AND QoS service deployment models AND

energy efﬁciency constraints”

• Q3. Name all abstracts, without explanation,

that related to one of the following keywords: ”

5G AND QoS service orchestration models AND

energy efﬁciency constraints”

4.2 Results and Discussion

In this section, we will explore the results obtained

from employing six selected tools to handle all of the

PDFs. Moreover, any signiﬁcant observations made

during this implementation stage will be highlighted.

Additionally, an assessment will be provided that de-

lineates both the advantages and disadvantages asso-

ciated with each of these six tools. Table 4 illustrates

the ﬁrst execution of all the queries. Please note that

if an article appears in both queries, it will be treated

as one instance and counted only once.

From Table 4 we can present some points:

• When it comes to the WebofScience database, all

the tools produce the same ﬁndings.

• In contrast to the ﬁndings in WebofScience, it is

evident that ScienceDirect presents a noticeably

wider gap in the results obtained from Pdf2gpt

and DocAnalyzer AI. This difference can be ex-

plained by the fact that Pdf2gpt beneﬁts from

smaller PDF documents, which meets its limited

requirements and allows for optimal performance.

We have compiled a few key points from the tables

above:

• The results from the expert and the tool were

mostly similar based on Table 4, with Springer be-

ing a notable exception. However, further analysis

in Table 5 for ACM and Table 6 for Springer re-

vealed differences between Pdf2gpt’s output and

the expert selection. For example, while Pdf2gpt

identiﬁed 34 articles from Springer according to

Table 4, the expert selected a total of only 69 ar-

ticles. The intersection between their selections

was even smaller at just 16 articles as shown in Ta-

ble 6. This ﬁnding suggests that while the search

yielded a large number of results, there is still a

notable difference between them. This empha-

sizes the importance of thorough evaluation and

validation when employing automated tools in re-

search.

• Next, we can now compare the results from each

database. Starting with ACM, Table 5 presents

the overlaps in our tool’s outcomes. From Ta-

ble 5, it becomes apparent that only three arti-

cles are present across all of the results. These

articles are: (Maleki et al., 2021; Sharma et al.,

2022; Sun and Naser, 2018). We move now to

Springer, Table 6 presents the overlaps in our

tool’s outcomes. Similar to ACM, Springer also

showed 4 articles in all search results. these arti-

cles are: (Patel et al., 2021; Velrajan and Ceron-

mani Sharmila, 2023; Thantharate and Beard,

2023; Kibalya et al., 2023).

4.2.1 Advantages

the main advantages of this approach are:

• To enhance the effectiveness of the expert’s task,

it is recommended to minimize the time consumed

during this phase. To be more precise, rather than

going through a total of 848 abstracts in our spe-

ciﬁc scenario, it would be sufﬁcient to examine

and validate only 640 abstracts instead. While this

difference may not appear substantial at ﬁrst when

conducting an initial examination, it becomes in-

creasingly evident as we advance toward the ﬁnal

evaluation.

• The results attained from the deployment of ar-

tiﬁcial intelligence (AI)-driven technologies have

demonstrated a signiﬁcant degree of efﬁcacy in re-

lation to precision and efﬁciency. These cutting-

edge technological solutions not only furnish

rapid outcomes but also guarantee a considerable

level of exactitude when conveying reliable infor-

mation or carrying out specialized assignments.

4.2.2 Limitations

the main problems of the use of AI-based tools are:

• The processing of PDF documents sometimes

consumes a signiﬁcant amount of time. One no-

table issue arose when we faced difﬁculties with

the page count, necessitating the need to divide

these PDFs into multiple sections. It was crucial

ICAART 2024 - 16th International Conference on Agents and Artiﬁcial Intelligence

600

Table 4: Result of the ﬁrst execution.

Tool/Database Springer ScienceDirect ACM WebofScience IEEE Xplore Total

Chatpdf 7 24 6 0 7 44

Pdf2gpt 34 115 11 0 12 172

Hipdf 14 32 5 0 10 61

SciSpace 14 99 9 0 8 130

Easy-peasy AI 11 51 7 0 7 76

DocAnalyzer AI 7 31 8 0 6 52

Human Expert 69 81 13 0 9 172

Table 5: Common selected paper for ACM.

Chatpdf Pdf2gpt Hipdf SciSpace

Easy-peasy

DocAnalyzer

Human

Expert

Chatpdf * 5 4 5 3 4 5

Pdf2gpt 5 * 5 7 5 6 8

Hipdf 4 5 * 4 4 3 4

SciSpace 5 7 4 * 6 7 8

Easy-peasy

3 5 4 6 * 5 6

DocAnalyzer

4 6 3 7 5 * 5

Human

Expert

5 8 4 8 6 5 *

Table 6: Common selected paper for Springer.

Chatpdf Pdf2gpt Hipdf SciSpace

Easy-peasy

DocAnalyzer

Human

Expert

Chatpdf * 6 5 7 6 6 5

Pdf2gpt 6 * 11 9 6 5 16

Hipdf 5 11 * 10 7 5 6

SciSpace 7 9 10 * 4 5 7

Easy-peasy

6 6 7 4 * 6 6

DocAnalyzer

6 5 5 5 6 * 5

Human

Expert

5 16 6 7 6 5 *

to ensure that no abstracts were inadvertently sep-

arated during this partitioning process. Moreover,

a substantial portion of time was expended while

subsequently searching through the documents,

particularly on platforms such as ScienceDirect

and Pdf2gpt.

• In response to the issue we faced regarding the

restricted daily PDF limit, we devised two alter-

native approaches for each tool. While attempting

to resolve the problem encountered with Chatpdf,

it became apparent that switching devices did not

rectify the persistent issue. Consequently, to over-

come this challenge, we opted to alter our network

connection by transitioning from one router to an-

other. Conversely, when confronted with a simi-

lar obstacle while using DocAnalyzer AI, we suc-

cessfully resolved it by simply logging into dif-

ferent accounts whenever we reached the prede-

termined PDF limit.

• When faced with a restriction, Hipdf employs

various strategies to address the issue. For in-

stance, when dealing speciﬁcally with ScienceDi-

rect PDFs, our approach involves dividing them

into smaller ﬁles through the process of splitting.

Additionally, in situations where users reach their

Token limit per user, we collaborate with another

account as an alternative solution.

• A recent observation has brought to light the fact

Towards the Use of AI-Based Tools for Systematic Literature Review

601

that several tools, including Hipdf, DocAnalyzer

AI, and SciSpace, frequently yield inaccurate re-

sults. This discrepancy is especially noticeable

when the title of the PDF document is missing.

5 CONCLUSIONS

In our research, we examined the utilization of

six artiﬁcial intelligence-based tools named Chatpdf,

Pdf2gpt, Hipdf, SciSpace, Easy-peasy AI and Doc-

Analyzer AI to automate a speciﬁc stage in compos-

ing the semantic literature review. We provide com-

prehensive results from each test conducted and high-

light both the advantages and disadvantages associ-

ated with utilizing these tools. Additionally, we dis-

cuss the limitations inherent in each tool and propose

effective approaches for overcoming them. The draw-

back of utilizing these methods is that they typically

necessitate pre-processing, like in our scenario, the

splitting of PDF ﬁles, etc.

In future investigations related to conducting

SLRs, our immediate goal is to complete the exam-

ination of IEEE and Science Direct databases. In ad-

dition, we will explore various writing and paraphras-

ing tools in future steps. Moreover, developers within

the community should introduce new features or alle-

viate existing constraints, such as restrictions on page

count or the number of PDFs processed per day. Fur-

thermore, it is essential for future studies to evaluate

the inﬂuence of AI-generated literature reviews on the

overall quality and integrity of academic research.

ACKNOWLEDGEMENTS

This work was partially supported by the LABEX-TA

project MeFoGL: ”m

ethodes Formelles pour le G

enie

Logiciel”

REFERENCES

Aliyu, M. B., Iqbal, R., and James, A. (2018). The canoni-

cal model of structure for data extraction in systematic

reviews of scientiﬁc research articles. In 2018 Fifth In-

ternational Conference on Social Networks Analysis,

Management and Security (SNAMS). IEEE.

Bannach-Brown, A., Przybyła, P., Thomas, J., Rice, A. S.,

Ananiadou, S., Liao, J., and Macleod, M. R. (2019).

Machine learning algorithms for systematic review:

reducing workload in a preclinical review of animal

studies and reducing human screening error. System-

atic reviews.

Biolchini, J., Mian, P. G., Natali, A. C. C., and Travassos,

G. H. (2005). Systematic review in software engineer-

ing. System engineering and computer science depart-

ment COPPE/UFRJ, Technical Report ES.

Calo, R. (2017). Artiﬁcial intelligence policy: a primer and

roadmap. UCDL Rev.

Elamin, M. B., Flynn, D. N., Bassler, D., Briel, M., Alonso-

Coello, P., Karanicolas, P. J., Guyatt, G. H., Malaga,

G., Furukawa, T. A., Kunz, R., et al. (2009). Choice

of data extraction tools for systematic reviews depends

on resources and review complexity. Journal of clini-

cal epidemiology.

Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins,

J. P., Mavergames, C., and Gruen, R. L. (2014). Liv-

ing systematic reviews: an emerging opportunity to

narrow the evidence-practice gap. PLoS medicine,

11(2):e1001603.

Felizardo, K. R., Andery, G. F., Paulovich, F. V., Minghim,

R., and Maldonado, J. C. (2012). A visual analysis

approach to validate the selection review of primary

studies in systematic reviews. Information and Soft-

ware Technology.

George, A. S. and George, A. H. (2023). A review of chat-

gpt ai’s impact on several business sectors. Partners

Universal International Innovation Journal, pages 9–

23.

Ghafari, M., Saleh, M., and Ebrahimi, T. (2012). A feder-

ated search approach to facilitate systematic literature

review in software engineering. International Journal

of Software Engineering & Applications (IJSEA).

Gurbuz, H. G. and Tekinerdogan, B. (2018). Model-based

testing for software safety: a systematic mapping

study. Software Quality Journal.

Hannousse, A. and Yahiouche, S. (2022). A semi-automatic

document screening system for computer science sys-

tematic reviews. In Mediterranean Conference on Pat-

tern Recognition and Artiﬁcial Intelligence. Springer.

Harhoff, D. and Wagner, S. (2009). The duration of patent

examination at the european patent ofﬁce. Manage-

ment Science, pages 1969–1984.

Hassani, H. and Silva, E. S. (2023). The role of chatgpt in

data science: how ai-assisted conversational interfaces

are revolutionizing the ﬁeld. Big data and cognitive

computing, page 62.

Jeblick, K., Schachtner, B. M., Dexl, J., Mittermeier, A.,

uber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel,

B. O., Ricke, J., and Ingrisch, M. (2022). Chat-

gpt makes medicine easy to swallow: An exploratory

case study on simpliﬁed radiology reports. ArXiv,

abs/2212.14882.

Jonnalagadda, S. R., Goyal, P., and Huffman, M. D. (2015).

Automating data extraction in systematic reviews: a

systematic review. Systematic reviews, 4(1):1–16.

Keele, S. et al. (2007). Guidelines for performing system-

atic literature reviews in software engineering.

Khan, A., Kim, T., Byun, H., and Kim, Y. (2019). Scis-

pace: A scientiﬁc collaboration workspace for geo-

distributed hpc data centers. Future Generation Com-

puter Systems.

ICAART 2024 - 16th International Conference on Agents and Artiﬁcial Intelligence

602

Kibalya, G., Serrat, J., Gorricho, J.-L., Okello, D., and

Zhang, P. (2023). A deep reinforcement learning-

based algorithm for reliability-aware multi-domain

service deployment in smart ecosystems. Neural

Computing and Applications, pages 23795–23817.

Maleki, E. F., Ma, W., Mashayekhy, L., and La Roche, H.

(2021). Qos-aware 5g component selection for con-

tent delivery in multi-access edge computing. In Pro-

ceedings of the 14th IEEE/ACM International Confer-

ence on Utility and Cloud Computing, pages 1–10.

Marshall, C. et al. (2016). Tool support for systematic re-

views in software engineering. PhD thesis, Keele Uni-

versity.

Mergel, G. D., Silveira, M. S., and da Silva, T. S. (2015).

A method to support search string building in system-

atic literature reviews through visual text mining. In

Proceedings of the 30th Annual ACM Symposium on

Applied Computing. Association for Computing Ma-

chinery.

Patel, S. B. and Lam, K. (2023). Chatgpt: the future of

discharge summaries? The Lancet Digital Health,

5(3):e107–e108.

Patel, Y. S., Reddy, M., and Misra, R. (2021). Energy and

cost trade-off for computational tasks ofﬂoading in

mobile multi-tenant clouds. Cluster Computing, pages

1–32.

Ray, P. P. (2023). Chatgpt: A comprehensive review on

background, applications, key challenges, bias, ethics,

limitations and future scope. Internet of Things and

Cyber-Physical Systems.

Sallam, M. (2023). Chatgpt utility in healthcare educa-

tion, research, and practice: systematic review on the

promising perspectives and valid concerns. Health-

care, page 887.

Sarker, I. H. (2022). Ai-based modeling: Techniques, ap-

plications and research issues towards automation, in-

telligent and smart systems. SN Computer Science.

Scells, H., Zuccon, G., and Koopman, B. (2019). Auto-

matic boolean query reﬁnement for systematic review

literature search. In The World Wide Web Conference.

Association for Computing Machinery.

Scells, H., Zuccon, G., Koopman, B., and Clark, J. (2020).

Automatic boolean query formulation for systematic

review literature search. In Proceedings of The Web

Conference 2020. Association for Computing Ma-

chinery.

Sellak, H., Ouhbi, B., and Frikh, B. (2015). Using rule-

based classiﬁers in systematic reviews: a semantic

class association rules approach. In Proceedings of

the 17th International Conference on Information In-

tegration and Web-based Applications & Services. As-

sociation for Computing Machinery.

Sharma, H., Budhiraja, I., Consul, P., Kumar, N., Garg, D.,

Zhao, L., and Liu, L. (2022). Federated learning based

energy efﬁcient scheme for mec with noma underlay-

ing uav. In Proceedings of the 5th international ACM

mobicom workshop on drone assisted wireless com-

munications for 5G and beyond, pages 73–78.

Snyder, H. (2019). Literature review as a research method-

ology: An overview and guidelines. Journal of busi-

ness research, pages 333–339.

Sun, P. and Naser, H. (2018). A service slicing strategy with

qos for lte-based cellular networks. In Proceedings of

the 14th ACM International Symposium on QoS and

Security for Wireless and Mobile Networks, pages 63–

69.

Thantharate, A. and Beard, C. (2023). Adaptive6g: adap-

tive resource management for network slicing archi-

tectures in current 5g and future 6g systems. Journal

of Network and Systems Management, page 9.

theresanaiforthat (2023). Easy-peasy ai accelerates content

creation. https://theresanaiforthat.com/ai/easy-peasy-

ai/?ref=search&term=Easy-peasy&ﬁd=50. (Accessed

on 07/05/2023).

Theresanaiforthat (2023). pdf2gpt - summarize a pdf.

https://theresanaiforthat.com/ai/pdf2gpt/?ref=search

&term=Pdf2GPT. (Accessed on 07/04/2023).

theresanaiforthat (2023). Scispace - your ai

aopilot to decode any research paper.

https://theresanaiforthat.com/ai/scispace/?ref=search

&term=scispace. (Accessed on 07/04/2023).

ToolsPedia.io (2023). Chatpdf - chat with any pdf

ﬁles for free using ai. https://www.toolspedia.io/ai-

tool/chatpdf. (Accessed on 07/04/2023).

Tsafnat, G., Glasziou, P., Karystianis, G., and Coiera, E.

(2018). Automated screening of research studies for

systematic reviews using study characteristics. Sys-

tematic reviews.

van Dinter, R., Tekinerdogan, B., and Catal, C. (2021). Au-

tomation of systematic literature reviews: A system-

atic literature review. Information and Software Tech-

nology.

Velrajan, S. and Ceronmani Sharmila, V. (2023). Qos-aware

service migration in multi-access edge compute using

closed-loop adaptive particle swarm optimization al-

gorithm. Journal of Network and Systems Manage-

ment, page 17.

Verma, M. (2023). Novel study on ai-based chatbot (chat-

gpt) impacts on the traditional library management.

International Journal of Trend in Scientiﬁc Research

and Development (IJTSRD).

wondershare (2023). Hipdf’s ’chat with pdf’.

https://www.hipdf.com/chat-with-pdf. (Accessed

on 07/04/2023).

Yang, X., Li, Y., Zhang, X., Chen, H., and Cheng, W.

(2023). Exploring the limits of chatgpt for query

or aspect-based text summarization. arXiv preprint

arXiv:2302.08081.

Yenduri, G., Srivastava, G., Maddikunta, P. K. R., Jhaveri,

R. H., Wang, W., Vasilakos, A. V., Gadekallu, T. R.,

et al. (2023). Generative pre-trained transformer: A

comprehensive review on enabling technologies, po-

tential applications, emerging challenges, and future

directions. arXiv preprint arXiv:2305.10435.

Zhu, J.-J., Jiang, J., Yang, M., and Ren, Z. J. (2023). Chat-

gpt and environmental research. Environmental Sci-

ence & Technology.

Towards the Use of AI-Based Tools for Systematic Literature Review

603