A Generalized LLMs Framework to Support Public Health Financing through Probabilistic Predictions and Uncertainty Quantification

Our initial work using NLP to link short-text budget lines and outcome indicators showed promises and limits; for example, difficulties in correctly classifying data from country A when trained with information from country B. Here, we use large-language models to overcome such limitations and develop a model-agnostic framework to generate better mappings.

As a systemic problem, public health cannot be addressed without considering other policy dimensions. Hence, a holistic approach across public policy areas is necessary to incorporate Health-for-All values into decision-making. However, such multisectoral interventions require public budgets that are effectively mapped into public health outcomes and indicators of their wider determinants. This budget-tagging procedure is high-cost, given that it is often done manually by domain experts. In this paper, we propose Categorical Perplexity-based Uncertainty Quantification (CPUQ), a novel, cost-effective Large Language Models (LLMs) framework that can be leveraged by policymakers to build budget-to-indicator and indicator-to-indicator mappings. This model-agnostic method employs categorical style prompts to generate interpretable Bernoulli and categorical distributions for edges in a Text-attributed Graph, which is associated with the descriptions of the budget items and indicators. The prompting strategy proposed provides a novel way to incorporate models’ uncertainty within the final outputs, enhancing accuracy and safety, We find that the budget-to-indicator mapping predicted by the framework aligns effectively with expert annotations, while when prompted to infer indicator-to-indicator networks, CPUQ estimates more nuanced relationships compared to alternative LLMs-based methods. Through our work, we hope to provide valuable insights into the strengths and weaknesses of leveraging LLMs to support public health budget planning, with the aim of promoting the implementation of the Health-for-All agenda across diverse governments and institutions.