The $SL_2(\mathbb{C})$ character variety $X(M)$ has emerged as an important tool in studying the topology of hyperbolic 3-manifolds. \cite{ChinburgReidStover} constructed arithmetic invariants stemming from a canonical quaternion algebra over the normalization of an irreducible component of $X(M)$ containing a lift of the holonomy representation of $M$. We provide an explicit topological criterion for extending the canonical quaternion algebra over an ideal point, potentially leading to finer arithmetic invariants than those derived in \cite{ChinburgReidStover}. This topological criterion involves Culler-Shalen theory (\cite{CullerShalen}) and, in some cases, JSJ decompositions of toroidal Dehn fillings of knot complements in the three-sphere. Inspired by the work of \cite{PaoluzziPorti} and \cite{TillmannDegenerations}, we provide examples of several cases where these refined invariants exist. Along the way, we show that certain families of once- and twice-punctured tori in hyperbolic knot complements can be associated with ideal points of character varieties.