Human motion recognition has many important applications, such as improved human-computer interaction and surveillance. A big problem that plagues this research area is that human movements can be very complex. Managing this complexity is difficult. We turn to American Sign Language (ASL) recognition to identify general methods that reduce the complexity of human motion recognition. In this paper we present a framework for continuous 30 ASL recognition based on linguistic principles, especially the phonology of ASL. This framework is based on parallel Hidden Markov Models (HMMs), which are able to capture both the sequential and the simultaneous aspects of the language. Each HMM is based on a single phoneme of ASL. Because the phonemes are limited in number, as opposed to the virtually unlimited number of signs that can be composed from them, we expect this framework to scale well to larger applications. We then demonstrate the general applicability of this framework to other human motion recognition tasks by extending it to gait recognition.