This thesis presents a study of van der Waals antiferromagnetic materials and topological semimetals, examining their quantum behavior through optical nonlinear harmonic generation and terahertz emission techniques. We specifically investigated spin behaviors at low dimension in transition metal trichalcogenides MPX$_3$ and explored nonlinear optical responses and quantized photogalvanic effects in multi-fold semimetals RhSi and CoSi. We developed an ultra-high sensitivity second-harmonic generation microscope, capable of detecting 0.1 photon per second, which allowed us to characterize antiferromagnetic orders at the single-layer limit. Additionally, we established a broadband terahertz emission setup with an excitation energy range of 0.2 eV to 1.5 eV at cryogetic temperature. This mid-IR capacity facilitated the detection of topological responses within the Weyl cones of the materials. Utilizing the second-harmonic generation microscope, we imaged antiferromagnetic orders in two-dimensional magnetic materials, such as Néel-type MnPSe$_3$ and MnPS$_3$, and zigzag-type FePS$_3$. MnPSe$_3$ is a van der Waals antiferromagnet with in-plane Néel orders, and we detected its antiferromagnetic long-range ordering down to the monolayer limit through optical second-harmonic generation. MnPS$_3$ is another van der Waals antiferromagnet with out-of-plane Néel orders and exhibits nearly isotropic magnetic anisotropy, closely resembling a Heisenberg magnet. We demonstrated long-range order in bilayer samples but observed suppressed magnetism in monolayer samples. By exploiting the interference between two distinct second-harmonic terms, we successfully imaged the different domains in both materials. Unlike MnPSe$_3$ and MnPS$_3$, the zigzag orders in FePS$_3$ did not break inversion symmetry; however, its antiferromagnetism was strongly coupled to the surface second-harmonic generation, enabling detection of zigzag orders in few-layer samples. We investigated the unique properties of the XY magnet MnPSe$_3$ with in-plane spins, where polarization-resolved measurements resolved the in-plane spin direction of the samples. We discovered that the spin direction was pinned by small, arbitrary local strain, and it could be further manipulated by additional magnetic anisotropy induced by strain control or in-plane magnetic fields. Our findings highlight the XY nature of intrinsic MnPSe$_3$ samples and the stabilization of long-range order in 2D due to external anisotropy from the environment. We examined the potential for noncentrosymmetric spin structures to generate bulk photogalvanic effects in crystals. We observed terahertz emission from intrinsic MnPSe$_3$ below the Néel temperature, and the relationship between magnetism and photocurrent generation was clearly demonstrated by opposite signs of the photocurrent under different Néel type domains. Excitation energies both above and below the band gap generated terahertz emission, indicating the observation of magnetic photogalvanic effects and magnetic optical rectification. Lastly, we investigated topological materials with noncentrosymmetric structures. We detected topological photogalvanic effects near a single topological node through terahertz emission, extending down to mid-IR photon excitation. We studied the relationship between the magnitude of nonlinear optical responses and topological electronic structures, and with the assistance of density functional theory, we examined the topological origins of photogalvanic effects and the proposed quantized circular photogalvanic effects.