Bundle Adjustments & Reprojection Error

Bundle adjustment is a joint state estimation technique to estimate the pose of 3D coordinate points describing the scene geometry, through a set of images taken from multiple perspective views while simultaneously estimating the position of cameras when the images were taken.

It is performed to estimate the 6DOF camera trajectory and 3D map (3D point cloud) from the input feature tracks. It is almost always used as the last step of every feature-based 3D reconstruction algorithm.

Methodology:

Once the pose of the camera and the 3D real world coordinate are estimated and the scene is captured through those cameras as image points, the shift or the geometric error corresponding to the image distance between a projected point and a measured one can be effectively measured. This shift in the image distance between the projected image points and measured image points is known as reprojection error.

Bundle Adjustment tries to minimize the reprojection error between the locations of projected and predicted image points, which is expressed as the sum of squares of a large number of nonlinear, real-valued functions.

Reprojection Error:

When a point is computed automatically or marked by the user on at least two images, the 3D coordinates of this point are computed using the camera’s internal and external parameters as well as the position of the point in the images.

External and internal camera positions help defining the correlation between 2D keypoints on different images and calculate its’ 3D position.

Once the 3D coordinates of the point are computed, the 3D point is reprojected on all the images that it appears. This error depends on the quality of the camera calibration (position and orientation), as well as on the quality of the marked point on the images (position and zoom level at which the point is marked).

The distance between the marked and the reprojected point on one image is the reprojection error.

Drawbacks in using Bundle Adjustments in SLAM:

However, two fundamental weaknesses plague SLAM systems based on bundle adjustment.

1. Need to carefully initialize bundle adjustment i.e. All variables, in particular the map, must be estimated as accurately as possible and maintained over time, which makes the overall algorithm cumbersome.

2. Since estimating the 3D structure (which requires sufficient baseline) is inherent in bundle adjustment, the SLAM algorithm will encounter difficulties during periods of slow motion or pure rotational motion.