Reasoning is the process in which statistical models integrate knowledge from multiple sources, internal or external, to generate new information or new knowledge that is deemed reasonable thus providing values to downstream tasks. Needless to say, this very definition lends reasoning to a great presence in a wide range of applications and problems in statistics and natural language processing (NLP); although NLP has seen an impressive growth thanks to the recent advances in foundation and large language models (LLMs), only quite recently do the practitioners started to explore theoretical and principled aspects. These new advancements in LLMs present new academic, industrial, and societal challenges, including data safety, model fairness, and computational efficiency, that can be benefited from more principled and explainable reasoning systems involving natural languages. This dissertation investigates the synergistic interplay between statistical methods and natural language processing (NLP) techniques, aiming to advance the development of more principled and explainable reasoning systems for problems involving natural languages. Through concrete examples and applications, this dissertation argues that, to unleash the full potential of reasoning systems, adopting the dual approach that examines the interplay between statistical methods and NLP techniques not only elevates the analytical rigor and clarity of reasoning systems but also marks a significant stride towards achieving explainability and principled reasoning in artificial intelligence. We first present a series of work that is built on top of potential-outcomes causal inference framework to address causal reasoning in NLP, where we focus on the commonsense causality reasoning (CCR) task where the reasoning system reasons about causal relations between natural language descriptions. Adopting the formal potential-outcomes framework into reasoning helps addressing several challenges including the lack of a clear problem formulation, low data availability and coverage of causal signals, and zero-shot inference. We proposed a novel system named ROCK that gave our answers to these challenges. ROCK is the first known principled CCR system that is build on the potential-outcomes framework and its variant was shown to be performant even among systems that directly make use of LLMs such as ChatGPT. The formal statistical foundation within not only facilitate more principled reasoning about causal relationships in textual data but also enhance the explainability of reasoning systems as the reasoning process is done in a more transparent ``white-box'' fashion. Secondly, this thesis works the other way round to study how NLP techniques can be organically used to facilitate classical reasoning task, exemplified by causal reasoning in observation studies where the goal is the estimation of causal estimands. Through methods including extracting higher-level and semantically meaningful features that are not otherwise available, one may unlock deeper insights into the causal mechanisms expressed in textual narratives. The integration of these techniques exemplifies how modern NLP can be harnessed to enrich causal analyses, making the reasoning processes behind effect estimations more transparent and well-informed. Finally, towards improving explainability of reasoning systems, we study how influence functions can be used to perform training data attribution for text generation models. There are two challenges in this task -- first it is unclear how to use influence functions on the sequential output; second estimation of influence functions for even medium-sized models is almost insurmountable. We propose a new method that defines and approximate the influence function on sequential outputs, which is proven to be useful. This application directly objective by enhancing our ability to interpret complex reasoning tasks from training samples in a manner that is both grounded in statistical rigor and accessible for explainable reasoning. In doing so, the research directly addresses the challenge of constructing natural language reasoning systems that are not accurate but also more explainable. This dissertation concludes with an outlook into the future. With various societal challenges emerging with the advancement of LLM-as-a-service, based on the explorations in the dissertation, we argue that principled and explainable reasoning systems shall be a key component in the solution that eventually addresses these challenges.