Camera calibration guide

In order to obtain the camera matrix and distortion coefficients that determine how the camera projects a scene into an image (see this page for more detailed information), we need to do a process known as camera calibration. The camera calibration will use a set of features (usually the corners of a checkerboard) whose real-world positions relative to each other are well-known, and determine the camera parameters based on their relative locations in the image. We have a calibration program taken from the OpenCV example code, which will open the camera in question, take pictures of the checkerboard, and use the locations of the corners to output camera parameters.

= Obtaining a Calibration Target =

Before doing calibration, you will need a printout of a calibration target, which can be obtained at https://calib.io/pages/camera-calibration-pattern-generator.
 * 1) Select &quot;Checkerboard&quot; for target type, as we don't currently have support for ChARUco boards.
 * 2) Set &quot;height&quot; to 215mm and &quot;width&quot; to 279mm - these are roughly the dimensions of an 8.5x11 inch sheet of paper.
 * 3) Try to change the square size so that the squares are as big as possible without going off the page or requiring you to decrease the number of rows or columns. This will make them easier to detect.

Note: Be careful not to bend or fold the target - it is recommended to tape it to a piece of cardboard or to a table. If you do so, make sure that the paper is as flat as possible. With the latest version of the calibration program, it is okay if the paper is not perfectly flat or has some wrinkles, but it MUST stay rigid!



A Note on Inner Corners and Size
Note that the size of this example target is written as 9x12 - this is the number of rows and columns of squares, respectively. However, since the calibration program is looking for well defined points that it can use to determine the projection of the camera, the notion of inner corners will be used frequently - these are the corners where two black squares and two white squares join. The number of inner corners in a row and column will be 1 less than the number of squares, so the size of this target in inner corners is 8x11.

= Running the Calibration Program =

Compiling
We should have the calibration program in the codebase, under. First, configure the project like you normally do by running CMake from the build directory (usually ) By default, we exclude the calibration program from the compilation process so that it isn't compiled every time you want to build the main rover program. To compile it, we have to explicitly specify the `calibration` target by running.

'''Note that you MUST have OpenCV 4 to build the calibration program. Our other code is (at the time of writing) backwards compatible with OpenCV 3, but it is recommended to upgrade to OpenCV 4 as soon as possible. See Upgrading to OpenCV 4 for more details.'''

Command Line Options
The calibration program can be run by executing a command of the form `ar/calibration `. There are many options, and running `ar/calibration` without any arguments will print them all, but the most important ones are below.


 * - The board width, in inner corners. This is the number of squares on the width of the checkerboard minus one. For the target shown above, this is 10.
 * - The board height, in inner corners. This is the number of squares on the height of the checkerboard minus one. For the target shown above, this is 7.
 * - The size of one of the squares on the board, in user-defined units. Using just the number of millimeters is fine here - the scale does not actually matter. Also, make sure you actually measure the board with a ruler instead of relying on the size on the printout! Often printer settings can cause the size of the squares to be different than they are in the PDF. It does not matter what the actual size of the square is as long as you input that value accurately into the program. Accuracy and precision are very important here! The more accurate your measurement, the better the calibration.
 * - The distance between the top left inner corner and bottom right inner corner, in user-defined units. Supplying this information in addition to the square size will use a new camera calibration method in OpenCV that is more accurate with the printed paper targets we are using. Note that the units are user-defined, but they must be in the same scale as the square size - if you give the square size in millimeters, for example, give this distance in millimeters as well. Like with square size, accuracy and precision are also very important.
 * - The number of frames to capture. Generally more frames is better as it gives the program more information about the camera to use. I recommend around 50 or 60.
 * - The minimum delay (in milliseconds) between attempts to capture a frame. The program will not capture a picture until it can fully detect the corners, but it might be nice to have it wait for you to move the checkerboard before trying to take a picture. I recommend around 1000-2000 but you can use whatever you want.
 * - The output file where the camera parameters should be written. The file name has to end in,  , or  , as these are the file formats supported by OpenCV.   is recommended as that is what most of the OpenCV tutorial code uses, and what we have used already. It is recommended that you include the date or some kind of unique name for this file to avoid overwriting previous calibration data.

An example command for the target shown above would be ar/calibration -w=11 -h=8 -s=20 -dt=244.13 -n=50 -d=1000 -o=params.yml Please note that you must have the  signs - the program does not take space-separated flag/argument pairs like most other command-line programs. The program may also silently do the wrong thing if they are not supplied - the OpenCV argument parser isn't the best.

Performing the Calibration
Once the program starts you should get a window showing you a live feed from the camera. If you don't have a piece of cardboard on the calibration target, put it flat on a table. The target should be as flat as possible during the calibration process, but it is much more important that it stays rigid. It is highly recommended to tape the target to some flat surface. Then, make sure the target is visible in the camera, and a set of colored circles should appear around the corners of the checkerboard. If you do not see the circles, move the target closer or rotate it so it is more visible.

Once you are ready to begin, press the &quot;G&quot; key. The program should start capturing a series of images, and will flash the image window every time it captures an image. During this time you should move either the calibration target (if you have it on a piece of cardboard) or the camera to get as many different angles of the calibration target as possible. Try getting some images where the calibration target is near the corners or edges of the image (this will help give some information about the lens distortion), some images where the target is rotated in different angles relative to the camera, and some images where the target is a different distance away from the camera (these will give more information about the projection). There should be a counter in the corner telling you how many images have been captured. Once you have captured all of the images, the program will freeze for a while as it computes the parameters, and then it will display a success message. At this time you can hit ESC to exit. The parameters should be saved in the  file you indicated with. They will look like this:

%YAML:1.0 --- calibration_time: &quot;Sun 03 Jan 2021 02:04:23 PM PST&quot; image_width: 640 image_height: 480 flags: 0 camera_matrix: !!opencv-matrix rows: 3 cols: 3 dt: d  data: [ 2.8163054058138857e+02, 0., 3.1986287258182375e+02, 0., 6.5147998226038069e+01, 2.4050723303632392e+02, 0., 0., 1. ] distortion_coefficients: !!opencv-matrix rows: 1 cols: 5 dt: d  data: [ -3.6842153449902861e-03, -8.3861698098247113e-05, -7.1088419024984209e-03, -4.1784556258417084e-04, 3.0386330418980679e-07 ] avg_reprojection_error: 1.9535680203954037e+01 The important parts are the  arrays under   and. The names are fairly self explanatory. Also note the average reprojection error. This should ideally be as low as possible. As a general rule of thumb, you should try to shoot for numbers that are less than 1. If the number is fairly high (the one in the example is very high) you should probably try recalibrating and getting more images or better angles of the calibration target. Again, try to get shots where the board is near the edges and where the board is tilted relative to the camera. More information on reprojection error is here: https://en.wikipedia.org/wiki/Reprojection_error

= Finishing Up = You should now be finished with the calibration process! Be sure to save the `.yml` file that was created as a result. Once the new camera system is implemented, we will add more to this page about what to do with your `.yml` file - for now, just save it someplace you can find it.

A Note on Image Size
Camera parameters are generally not the same for different image sizes, even for the same camera. This is because they include data like the focal length and image center in pixels - if the image size changes, these measurements will be wrong. They also often cannot simply be scaled to fix this problem; many cameras, especially cheap or commonly available webcams, will have entirely different fields of view at different resolutions, rendering either their camera matrices or distortion coefficients (or both) incorrect for different image sizes. For now, keep your camera parameters separated by image size even if they are for the same camera. The `.yml` file should have the image size stored in it, so this should make it easier to keep track.