There are currently many good methods for solving sequential decision-making problems under certain assumptions on the structure of the incoming data. However, when the avail- able data fails to meet such assumptions, such as when it involves surrogate or proxy signals, involves measuring survival times, or is unstructured such as in the case of image data, these methods fall short. In this dissertation, we aim to address this discrepancy for specific types of complex data, inspired by healthcare settings in which we might encounter such data. We introduce new methods to better learn from incoming and limited data sources in order to make more efficient decisions. The goal of this work is to develop methods that medical professionals can use to better leverage data in order to determine efficacy or necessity of treatment. All three projects included in this dissertation focus on developing adaptive methods that leverage information as it becomes available to update predictions, to ensure that medical professionals can better balance providing effective healthcare with using available resources efficiently. We also focus on quantifying the performance of these algorithms, and identifying the conditions under which it is most beneficial to use these novel strategies instead of currently employed methods.