The development, validation, and evaluation of prognostic systems: An application to the acquired immunodeficiency syndrome (AIDS)

Amy Caroline Justice, University of Pennsylvania

Abstract

Prognostic information is central to a wide variety of decision making by patients, physicians, researchers, health care managers, and policy makers. While all these decision makers require accurate prognostic information to identify optimal choices, their specific requirements vary: at times it is important to differentiate between who will live and who will die (discrimination); at other times the emphasis is upon the probability of mortality (calibration). The theoretical portion of the dissertation develops the roles that discrimination and calibration play in decision making processes, how these components of accuracy can be measured and compared, and how to determine whether the accuracy reported in one study may be generalized. It also extends these concepts to the problem of predicting mortality over a series of time intervals. The empiric portion begins with a meta-analysis of prognostic models that explores their accuracy in development and validation. This is followed by the major analytic focus in which a panel of prognostic models predicting mortality with AIDS are developed using logistic regression, recursive partitioning analysis, and neural networks and tested in a several data sets. The result is that a neural network developed to predict serial mortality, using base predictors and CD4 cell count, provided the greatest accuracy in development and the most consistent accuracy in external tests. A similar model, developed using logistic regression, approximated the overall accuracy of the neural network and was better calibrated. Models developed using recursive partitioning analysis did not perform as well. The conclusions are: (1) a consistent and systematic method can be used to evaluate and compare the performance of prognostic models across time intervals and data samples; (2) time exerts important effects upon accuracy of prognostic models and should be explicitly modeled; and (3) logistic regression and neural network models that predicted serial mortality and included baseline characteristics with CD4 cell count demonstrated a level of accuracy across validation samples that is superior to published models for mortality with AIDS.

Subject Area

Biostatistics|Surgery|Public health

Recommended Citation

Justice, Amy Caroline, "The development, validation, and evaluation of prognostic systems: An application to the acquired immunodeficiency syndrome (AIDS)" (1996). Dissertations available from ProQuest. AAI9627942.
https://repository.upenn.edu/dissertations/AAI9627942

Share

COinS