This is a step-by-step guide to running an OrthoCam stereo camera calibration.
The MATLAB Camera Calibrator App should do single-camera calibrations pretty well, except the (z-depth, focal length) parameter pair will not be well-resolved. Often it helps to initialize this optimization by clicking the 'Optimization Options' button and entering something in the 'initial intrinsics' box that is close to what you expect e.g. in the case of Stephen's rig: [100000 0 0;0 100000 0;384 255 1] Check that the calibration looks good (low reprojection error). Then do a "Save Session". This will save the calibration images you used with their detected corners, along with the calibration results.
Run/save single-camera calibrations for both cameras. These will be used by OrthoCam. You may need to intialize the optimization for this to work.
SH rig: Go to 'optimization options' and enter [100000 0 0;0 100000 0;384 255 1].
Step 2: Open the MATLAB Stereo Camera Calibrator App (stereoCameraCalibrator) and Create a new Stereo Project.
Add the calibration image pairs you would like to use and verify that the corners are detected appropriately etc.
Important: All calibration images included/used in the stereo project need to have been included in the single-camera calibrations in Step 1.
Step 3: Click Calibrate. If your MATLAB path is configured correctly, this will run the OrthoCam stereo calibration instead of the MATLAB calibration.
Step 3a. Obtain single-camera Orthocam calibrations for each camera.
First, you will be prompted to load your saved single-camera calibration sessions from the MATLAB App, first for camera 1, then camera 2.
Single-camera OrthoCam calibrations will be done on each camera, using the MATLAB results as a guideline/starting point. Currently, the MATLAB optimizer lsqnonlin is used to run these optimizations. In the progress display, the third column of numbers indicates the current optimizer residual. A "good" value for this residual is say 1000 or less. This residual may decrease slowly at times but will often make large quantum leaps.
Verify that the reproduction error is good with these single-camera calibrations.
After both single Orthocam calibrations are done, Save the results. You will now have two Orthocam single-camera calibrations. Now, do NOT continue with the stereo calibration. Instead, save your Stereo Calibration session and run the next step (3b).**
Step 3b. (recommended, possibly optional in some cases) Resolve Mirror-Image Ambiguity for single-camera Orthocam calibrations.
A single OrthoCam camera has a mirror-image ambiguity or symmetry. Suppose an object A is being viewed from an OrthoCamera. Let A' be the mirror-image of this object generated by reflection in the frontal/camera plane, but with the same center-of-mass position. By the nature of Weak Perspective projection, A and A' look identical to and cannot be distinguished by an OrthoCam.
In particular, an OrthoCam cannot tell whether a rectangular calibration pattern (eg checkerboard) is oriented at a certain 3D angle phi or its mirror image phi'.
This issue becomes more serious when a stereo calibration is performed on a pair of OrthoCams. In the case of the SH rig, the cameras are oriented at precise right angles relative to each other. As a result, there is a rig-wide systemic symmetry where two mirror-image states are very nearly identical (perfectly identical in a perfect world) and indistinguishable:
(all calibration patterns reflected in frontal plane) AND (Camera2 positioned to left of Camera1 vs Camera2 positioned to right of Camera1, ie Camera2 flipped 180degrees on its optical axis and repositioned).
It is possible that this issue may affect any two-camera rig (I am not sure right now).
Our Solution to this dilemma for now is very practical. We manually check and correct each calibration pattern in each OrthoCam single-camera calibration, "mirror-flipping" any patterns as necessary so that they match the physical rig configuration. Run the following line-by-line.
cal = load('/path/to/yourorthocam/step3a/orthocam_mono_calib_cam1.mat','-mat');
[mx,my,u0,v0,k1,k2,r2vecs,t2vecs] = OrthoCam.unpack1cam(cal.pOpt,cal.nPat);
% This line will show each calibration pattern in the object sequentially. Use the 3D rotation
% tool in the figure toolbar to rotate the figure and determine whether the pattern is
% "flipped" in the frontal plane relative to its correct orientation. If it is, enter '-1' and
% this will flip the pattern to its correct orientation. Go through and do this for every
% pattern.
%
% TODO: precise flipping (keeping center-of-mass fixed) requires knowing the center of pattern
% in pat-coords. Hardcoded Default 20170605: 8x5 pattern, .1mm checkboard size. It is OK if
% this is slightly incorrect, see below.
[hFig,tfflipped,r2vecs,t2vecs] = OrthoCam.viewExtrinsics1cam(cal.worldPts,...
r2vecs,t2vecs,'patByPat',true);
%
calflipped = rmfield(cal,'p0');
calflipped.tsManuallyFlipped = now;
calflipped.pOpt = OrthoCam.pack1cam(mx,my,u0,v0,k1,k2,r2vecs,t2vecs);
%
save('/path/to/yourorthocam/step3a/orthocam_mono_calib_cam1_manually_flipped_maybeapproximate.mat','-struct','calflipped');
Do this for both cameras, saving new Ortho-mono-calibrations for each camera.
Important. The flipping procedure may only be approximate, eg if the center of the calibration patterns is not precisely specified. Thus the two manually-flipped Ortho-mono-calibrations are approximations only. However, these are only starting points/inputs into the Ortho-stereo-calibration! So far, we have found that using these improved starting points
- Improves convergence of the stereo calibration, because the patterns are starting in roughly correct positions. If roughly 50% of the calibration patterns are 'flipped', then the optimizer has a tough battle to gradually flip all of the patterns to the same side.
- The final, converged stereo calibration, is nearly certain to be the "correct" choice out of any system-wide symmetry. Again, the calibration patterns are starting off with correct/consistent orientations, and the optimizer will have a much easier time converging to the correct solution.
When this step is complete, you will have 4 single-camera Orthocam calibrations! Each camera will have one original Ortho-calibration of step 3a, and one manually-flipped one from step 3b. The one generated in step 3b may be approximate, but all of its calibration patterns will be aligned correctly. We are on the home stretch.
Step 3c. Go back to the MATLAB Stereo-Calibration App. Load your stereo session from Step 3a. Click Calibrate, and load the FLIPPED Orthocam single-cam calibrations from step 3b.
Step 3d. Optionally, select a "base" pattern for the stereo calibration.
With the new step 3b, I bet that this step is mostly unnecessary now. The optimizer will have a much easier time converging. When the listbox pops up, you can just pick the first/default pattern. For reference, here is the old blurb:
This is a technical quirk with the current implementation. One calibration-pattern-pair is
currently selected to serve as a common coordinate system for the optimization. Pattern-pairs
that are "unusual" (eg upside-down, far from image center(s) etc) should not be chosen as they
can lead to convergence problems. Select a very "normal" pattern-pair for this step (X to
right; Y down).
Step 3e. Run the stereo optimization. If you are still reading this, you are a tough cookie. Your perseverance will soon pay off!
Again, lsqnonlin is used for the optimization. This optimization is tougher, and usually requires restarts. Rather than just letting the optimizer run continuously for a long time, using restarts seems to shock/randomize the optimizer and provides for a faster overall optimization. The residual will decrease slowly at times and will make large leaps at other times.
The good news is, with the new Step 3b, these optimizer runs usually run much more smoothly now. The optimizer doesn't have to fight getting patterns that are randomly mirrored back into alignment.
You can restart/repeat the optimization until the final optimizer message is "Possible minimum found" etc rather than "Maximum iterations/evaluations exceeded". Again, a "good" residual value (3rd column of numbers in display) is say in the range of 1000 or less.
Verify that the reproduction error is good with the stereo calibration.
VERY IMPORTANT: Verify the extrinsics using the calObj.viewExtrinsics(); function. Make sure the entire rig is as physically expected! Use the Rotate3D tool to convince yourself.
You will be prompted to save your Orthocam calibration.
At this point you are done! the final results are not integrated with the MATLAB Stereo Camera Calibration App, so you may not see the usual MATLAB App display after completion.
To check your saved calibration, load the MAT-file containing your saved calibrations and run these methods:
calObj.viewExtrinsics
calObj.viewRPerr
calObj.viewRPerrStroTri % different form of RP err