We study the performance of risk-controlling prediction sets (RCPS) and quantile risk control (QRC), two empirical risk minimization-based formulations of conformal prediction, with a single trajectory of temporally correlated data from an unknown stochastic dynamical system. First, we use the blocking technique to show that RCPS attains performance guarantees similar to those enjoyed in the iid setting whenever data is generated by asymptotically stationary and contractive dynamics. Next, we use the decoupling technique to characterize the graceful degradation in RCPS guarantees when the data generating process deviates from stationarity and contractivity. Finally, we use the stitching technique to propose a novel online QRC algorithm that attains non-trivial guarantees against nearly arbitrary data generating processes. We conclude by discussing how these tools could be used toward a unified analysis of offline and online conformal prediction algorithms, which are currently treated with very different tools.