Understanding and Reasoning with Numerical Information in the Era of Large Language Models

<p dir="ltr">Numbers play a crucial role in textual communication, providing precision and clarity across diverse domains. While numerical reasoning has been widely explored in mathematical problem-solving, limited research has examined the broader challenges of processing numerical information expressed in natural language. Large Language Models (LLMs) have demonstrated impressive capabilities in understanding and generating text, yet their ability to interpret, extract, and reason with numerical information beyond mathematical contexts remains underexplored. This thesis addresses this gap by systematically analyzing how numerical information is conveyed in text and by evaluating LLMs’ capabilities in numerical information processing and reasoning.</p><p dir="ltr">This thesis is guided by four key questions that collectively examine the challenges of numerical information processing. First, it investigates how effectively Natural Language Processing (NLP) and information retrieval (IR) models can retrieve numerical information or information associated with numerical values. Based on two studies in the clinical domain, namely SemEval NLI4CT and TREC Clinical Trial, it is revealed that both IR models (which rely on term matching) and NLP models (which rely on semantic similarity) remain inadequate for the task of numerical information retrieval. Second, it explores how numerical information is expressed in natural language and the different types of meaning it conveys. This thesis introduces a taxonomy of numerical information that categorizes how numbers appear in text and their semantic interpretations, validated through a human annotation study. Third, it examines how numerical reasoning can be conceptualized and evaluated in the era of large language models. </p><p dir="ltr">It reviews recent research trends showing that most numerical reasoning studies focus narrowly on mathematical problem-solving. To extend this perspective, the thesis proposes a taxonomy of numerical reasoning that encompasses key reasoning components such as numerical operations, number sense, numerical knowledge, linguistic factors, and number representation. Finally, it assesses the extent to which LLMs demonstrate numeracy in understanding and reasoning with numerical information embedded in text. To address the fourth question, it explores LLMs’ numeracy by developing benchmarks that evaluate fundamental numerical skills (e.g., arithmetic, number comparison, normalization), a subset of the skills defined in the proposed taxonomy. These benchmarks are formulated as Natural Language Inference (NLI) tasks to assess whether LLMs can comprehend and reason about numerical information in text. The analysis reveals that while most LLMs can identify correct entailments, they often fail to correctly reject incorrect conclusions involving numerical information, misclassifying contradiction and neutral cases.</p><p dir="ltr">The contributions of this thesis include the introduction of structured taxonomies for numerical information and numerical reasoning, the development of new benchmarks for evaluating numerical reasoning in LLMs, and a comprehensive analysis of LLMs’ strengths and limitations in numeracy. By addressing these critical aspects, this work provides insights into how well modern NLP models understand and reason with numerical data and lays the foundation for future research on improving numeracy in LLMs.</p>