Tau is a microtubule-associated protein, which promotes neuronal microtubule assembly and stability. Accumulation of tau into insoluble aggregates known as neurofibrillary tangles (NFTs) is a pathological hallmark of several neurodegenerative diseases, known as tauopathies. Aggregated proteins are normally degraded by the cell’s protein degradation mechanisms, autophagy or the ubiquitin-proteasome system (UPS). In tauopathies, however, the efficiency of these degradation pathways becomes challenged by the abnormal accumulation of the tau protein, which consequently, does not get fully degraded. The current hypothesis is that small, soluble oligomeric tau species preceding NFT formation cause toxicity. However, thus far, visualizing the spatial distribution of tau monomers and oligomers inside cells under physiological or pathological conditions has not been possible. Moreover, it is unclear whether certain tau aggregate species are more resistant to degradation. Here, using single-molecule localization microscopy, we show that tau forms small oligomers on microtubules ex vivo. These oligomers are distinct from those found in cells exhibiting tau aggregation and could be precursors of aggregated tau in pathology. Furthermore, using an unsupervised shape classification algorithm that we developed, we show that different tau phosphorylation states are associated with distinct tau aggregate species. Using machine learning, we also show that autophagy and UPS target distinct classes of tau aggregates for degradation. More specifically, we propose a model where tau fibrils are targeted by UPS for degradation and NFTs are mostly degraded by autophagy, generating more tau monomers and oligomers as well as small fibrils. Our work elucidates tau’s nanoscale composition under nonaggregated and aggregated conditions ex vivo and further informs our understanding of how tau aggregates become degraded by the cell’s degradation pathways.