This thesis extends and improves methods for estimating key quantities of hidden Markov models through spectral method-of-moments estimation. Unlike traditional estimation methods like EM and Gibbs sampling, the set of estimation methods, which we call spectral HMMs (sHMMs), are incredibly fast, do not require multiple restarts, and come with provable guarantees. Our first result improves upon the original spectral estimation of hidden Markov models algorithm by estimating the parameters from fully reduced data. We also show that the parameters developed in the fully reduced dimensional version can be estimated using various forms of regression, which can lead to major speed gains, as well as allowing flexibility in the estimation scheme. We then extend the algorithm beyond basic hidden Markov models to latent variable tree structures that have linguistic applications, especially dependency pars- ing, and finally to hidden Markov models in which the output is a high-dimensional, continuously distributed variable. We show that spectral estimation of hidden Markov models can be factored into two major components- estimation of the hidden state space dynamics, and estimation of the observation probability distributions. This leads to extremely flexible estimation procedures that can be tailored precisely for the task of interest. These tools are all simple to implement, fast, and naturally incorporate dimension reduction, which allows them to scale gracefully as the dimension of the data increases.