Epipolar Geometry

The geometry that relates the cameras, points in 3D, and the corresponding observations is referred to as the epipolar geometry of a stereo pair. It is independent of the scene structure and only depends on the internal and external parameters of the camera.

The general setup of epipolar geometry. The gray region is the epipolar plane. The orange line is the baseline, while the two blue lines are the epipolar lines.

The standard epipolar geometry setup involves two cameras observing the same 3D point P, whose projection in each of the image planes is located at p and p’ respectively. The camera centers are located at O1 and O2, and the line between them is referred to as the baseline. We call the plane defined by the two camera centers and P the epipolar plane. The locations of where the baseline intersects the two image planes are known as the epipoles e and e’. Finally, the lines defined by the intersection of the epipolar plane and the two image planes are known as the epipolar lines. The epipolar lines have the property that they intersect the baseline at the respective epipoles in the image plane.

When the two image planes are parallel, then the epipoles e and e’ are located at infinity. Notice that the epipolar lines are parallel to u axis of each image plane.

An interesting case of epipolar geometry is shown in Figure 4, which occurs when the image planes are parallel to each other. When the image planes are parallel to each other, then the epipoles e and e’ will be located at infinity since the baseline joining the centers O1, O2 is parallel to the image planes. Another important byproduct of this case is that the epipolar lines are parallel to an axis of each image plane.

Fundamental Matrix and Essential Matrix

Since we have now covered the basics of epipolar geometry we can now address two more ingredients that are needed in order to find the epipole and epiline, that is; the fundamental and essentials matrices.

Essential Matrix contains the information regarding translation and rotation that maps 3D real-world coordinates to 3D image coordinates. Fundamental matrix on the other hand, contains the same information as the essential matrix but also includes intrinsic information about both cameras which maps 3D image coordinates to 2D pixel coordinates.

Fundamental Matrix is a multiplication of 2 other matrices i.e. extrinsic and intrinsic camera matrices. These matrices store the extrinsic and the intrinsic parameters of the camera respectively.

The Extrinisic matrix stores information in the form of a rotation matrix as well as a translation vector. The rotation matrix stores the camera’s 3D orientation while the translation vector stores its position in 3D space.

Three types of rotations are required as extrinsic parameters which are pitch, roll and yaw.

· Rotation around the front-to-back axis is called roll.

· Rotation around the side-to-side axis is called pitch.

· Rotation around the vertical axis is called yaw.

The intrinsic matrix stores the camera intrinsic such as focal length and the principal point.

P = [R|T] K

If we have the 2D coordinates, then using calibration parameters, we can map to 3D and vice versa using the following equation:

Epipolar Geometry — Feature Correspondence

Epiline’s significance is that when we are searching for the matching point of X in the right sided camera’s image we already know that it will appear along this epiline therefore greatly reducing our search effort. Furthermore we can assume every point that appears in the left handed image will always have an accompanying epiline found in the right sided image. This is known as the epipolar constraint.

Here’s how the epipolar search works:

• Take a pixel in the left image
• Now, to find this pixel in the right image, simply search it on the epipolar line. There is no need for a 2D search, the point should be located on this line and the search is narrowed to 1D.

Epipolar Geometry — Stereo Vision

Stereo Vision is about finding depth based on two images. Stereo vision system consist of two cameras, a left and a right one. These two cameras are aligned in the same Y and Z axis. Basically, the only difference is their X value

Since a stereo camera assumes the epipolar geometry, instead of searching for a point in the whole image, we search for a particular point only along the horizontal-x axis of the stereo images. In simpler words, all we have to do is to find the correspondence of a point, in the left image, to its position in the right image along the same x axis, usually with an offset and range. This drastically reduces the space complexity of our algorithm and reduces the multi-dimensional problem to a single dimension.

It works because the stereo cameras are aligned along the same Y and Z axis and differ only in the X-value.

This article covered the concepts of epipolar geometry, epilines, decomposition of fundamental matrix into extrinsic and intrinsic matrices, use of epipolar constraint in feature correspondence and stereo vision.