As robots move beyond controlled environments into everyday settings, they require a robust, human-level understanding of scenes. This thesis introduces a unified mathematical framework and a set of algorithms for representing, fusing, and exploiting scene semantics in robotics. We first formalize semantic observations as partial interpretations of a first-order language and as local sec- tions of a predicate sheaf, thereby unifying model-theoretic and sheaf-theoretic perspectives. Next, we present an incremental Boolean factor-graph formulation G = (V,F), where weighted predi- cate assignments carry confidence scores. To achieve real-time performance, we develop subclique decomposition, sliding-window factor merging, variable quotienting, and overlap factors, together with a localized Iterated Conditional Modes (ICM) inference that updates only a small “focus” of recent variables in under 0.05 s per scan.

Building on this pipeline, we address two fundamental tasks: (i) Semantic Localization & Denoising, in which noisy new scans are embedded into the union-graph via factor similarity and corrected by focused ICM; and (ii) Completion & Generation, in which a Greedy Cluster-Growth algorithm extends a seed predicate set into a maximally consistent partial scene. We further introduce a complementary semantic-manifold formulation: CLIP-based embeddings are organized via Isomap into a Riemannian semantic manifold, enabling goal localization and path planning that respect a quasi-isometry to the physical world. Finally, we demonstrate an online method for semantic Gaussian splatting that incrementally builds a hierarchical spatial model.