Authors:
Yang Zhang
and
James Pope
Affiliation:
Intelligent Systems Lab, University of Bristol, Bristol, U.K.
Keyword(s):
Natural Language Processing, Large Language Model, Prompting Engineering, Evidence-Based Policy-Making, Policy Analysis.
Abstract:
Policy analysis or formulation often requires evidence-based support to ensure the scientific rigor and rationality of the policy, increase public trust, and reduce risks and uncertainties. However, manually collecting policy-related evidence is a time-consuming and tedious process, making some automated collection methods necessary. This paper presents a novel approach for automating policy evidence collection through large language models (LLMs) combined with Reasoning and Acting (ReAct) prompting. The advantages of our approach lie in its minimal data requirements, while ReAct prompting enables the LLM to call external tools, such as search engines, ensuring real-time evidence collection. Since this is a novel problem without existing methods for comparison, we relied on human experts for ground truth and baseline comparison. In 50 experiments, our method successfully collected correct policy evidence 36 times using GPT-3.5. Furthermore, with more advanced models such as GPT-4o, t
he improved understanding of prompts and context enhances our method’s efficiency. Finally, our method using GPT-4o successfully gathered correct evidence 45 times in 50 experiments. Our results demonstrate that, using our method, policy researchers can effectively gather evidence to support policy-making.
(More)