The Mobile Robot Programming Toolkit

1. Image management in the MRPT

Images themselves are stored in CImage objects, which hold OpenCV's IplImage structures, plus some further functionality. When managing rawlogs or datasets gathered by a robot, images are packaged as observations into CSensoryFrame's as CObservationImage objects. Note that this article applies to monocular cameras, while images from stereo cameras are stored in a different class (see CObservationStereoImages). However, an arbitrary number of monocular cameras can be grabbed simultaneously in a given rawlog. These CObservationImage objects keep the images among all the important information of the camera required for any posterior post-processing of the images:

• The camera intrinsic parameters matrix (Discussed below).
• The distortion parameters vector (Discussed below).
• The focal length of the camera, in meters.
• The pose of the camera reference system (the focal point), relative to the robot (the extrinsic parameters). See the following schematic description:
  
  

  Note how the +Z axis points forward in the camera frame, while the standard for all other sensors and robot frames in MRPT is the +X axis pointing forward.

2. Camera parameters

Each image observation has a data member of type mrpt::utils::TCamera which maintains all the camera parameters. In the case of stereo cameras, two instances of this structure exist, one for each camera. 

A TCamera structure has methods to dump and read a complete set of camera calibration parameters to configuration files, following this format:

[SECTION_NAME]
  resolution = [NCOLS NROWS]
  cx         = CX
  cy         = CY
  fx         = FX
  fy         = FY
  dist       = [K1 K2 T1 T2 K3]
  focal_length = FOCAL_LENGTH   // Optional

To avoid mistakes, TCamera provides inline functions to access the distortion parameters by their names instead of their indices in the vector of distortion parameters.

2.1. Intrinsic parameters  (or "calibration matrix")

 A TCamera structure includes the 3x3 matrix of intrinsic parameters that define how 3D points in local coordinates wrt the camera projects into the image plane. The matrix is:

Where:

• f_x and f_y: The focal length, in distance (pixels).
• c_x and c_y: The image central point (in pixels).

2.2. Distortion parameters

A TCamera structure also includes the vector of distortion parameters:

Where:

• k₁, k₂ and k₃ are radial distortion parameters.
• p₁ and p₂ are tangential distortion parameters.

References

• Camera Calibration Toolbox for Matlab.
• The book "Multiple View Geometry in Computer Vision", Hartley, R. I. and Zisserman, A.
• The paper A four-step camera calibration procedure with implicit imagecorrection, Heikkila, J. and Silven, O., 1997.