Omnidirectional video enables direct surround immersive viewing of a scene by warping the original image into the correct perspective given a viewing direction. However, novel views from viewpoints off the camera path can only be obtained if we solve the 3D motion and calibration problem. In this paper we address the case of a parabolic catadioptric camera – a paraboloidal mirror in front of an orthographic lens – and we introduce a new representation, called the circle space, for points and lines in such images. In this circle space, we formulate an epipolar constraint involving a 4x4 fundamental matrix. We prove that the intrinsic parameters can be inferred in closed form from the 2D subspace of the new fundamental matrix from two views if they are constant or from three views if they vary. Three dimensional motion and structure can then be estimated from the decomposition of the fundamental matrix.