There's a great interest in studying particle assembly on fluid interfaces for their properties to stabilize droplets in food, cosmetics, and oil industry, and to form permeable capsules in pharmaceutical industry. Capillary interaction holds tremendous promise as a tool to orient and assemble microparticles on interfaces between two immiscible fluids. Microparticles trapped at an interface deform the interface around them due to particle geometry, surface roughness, external fields and body forces. When particle deformations overlap, capillary interaction arise to minimize the interfacial deformation, hence minimizing energy. In literature, anisotropic shaped particles has been reported that their interactions depend strongly on the particle shape and curvature fields where particles adsorb can orient and attract particles to high curvature regions. However, less is known about particles of other shapes that don't disturb the interface as much as those anisotropic particles. In my thesis, I aim to use curvature field as a means to direct assemblies of particles with less pronounced deformation on interfaces. On curved oil-water interfaces, I study capillary attraction of microdisks and spheres that are pinned at the three-phase contact line due to surface roughness. These particles have radius much smaller comparing to the radius of curvature; they induce nanometric deformation on the interface and still create significant interface distortion and capillary interactions. To understand structure formation, I develop theory for pair interactions of particles on curved interfaces where capillary curvature interaction competes with particle pair interaction. Furthermore, I delve into interactions of large microdisks with curvature fields in both experiments and theory to explore capillary repulsion owing to the interaction of higher order modes with curvature fields. Lastly, I show the elastocapillary interactions of microcylinders on a thin film of uniform nematic liquid crystals. In all the studies described above, interface curvature, in particular, finite deviatoric curvature plays a central role in assembly and in guiding structure formation. The concepts studied here are fundamental and provide guidance in understanding, building, and designing colloidal structures with functionalities that have potential applications in various fields.