Monday, December 1, 2014

Adding stereo rendering to Metaio for Google Cardboard.

Adding stereo rendering to Metaio for Google Cardboard.

We will do this w the Metaio creator

 http://dev.metaio.com/sdk/tutorials/see-through-glasses/

See-through Glasses

General

In order to support see-through glasses, such as the Epson Moverio BT-200 glasses, the Metaio SDK has stereo rendering support built in with version 5.2 and newer. If the stereo rendering mode is enabled, the scene is rendered twice from two slightly different perspectives. The output of these two renderings is displayed in a so called split screen mode that is supported by most 3D output devices such as phones with 3D capable displays or stereo glasses. It might be necessary to use some device vendor specific API to enable the 3D display or glass to work in split screen mode together with the Metaio SDK.
To test the stereo rendering, you can enable it on any device. You will just see the screen split in the middle, showing one rendering on the left and another one on the right side.
The Metaio SDK is optimized for stereo mode, which is available on the Epson Moverio BT-200 glasses, but will run perfectly well in mono mode on devices such as Google Glass, Vuzix M100, etc. There is a dedicated tutorial explaining how to run the Metaio SDK on Google Glass.

Enabling Stereo Rendering

There is only one method that needs to be called to initially enable stereo rendering

Android (Java)

  Copy Code 
metaioSDK.setStereoRendering(true)

Adjusting the Rendering Angles [SDK 5.3 and older]

To adjust the position of the virtual cameras which render the scene from two different angles, two more methods are provided by the SDK.

Android (Java)

  Copy Code 
//left camera
Vector3d positionLeft = new Vector3d(0.0f, 0.0f, 0.0f);
Vector3d upVectorLeft = new Vector3d(0.0f, 1.0f, 0.0f);
Vector3d lookAtLeft = new Vector3d(0.0f, 0.0f, -100.0f);
metaioSDK.setStereoRenderingLeftCameraTransformation(positionLeft,upVectorLeft,lookAtLeft);
 
//right camera
Vector3d positionRight = new Vector3d(-10.0f, 0.0f, 0.0f);
Vector3d upVectorRight = new Vector3d(0.0f, 1.0f, 0.0f);
Vector3d lookAtRight = new Vector3d(0.0f, 0.0f, -100.0f);
metaioSDK.setStereoRenderingRightCameraTransformation(positionRight,upVectorRight,lookAtRight);

Adjusting the Rendering Angles [SDK 5.5 to 6.0]

With SDK 5.5, we superseded the old methods setStereoRendering{Left,Right}CameraTransformation by setHandEyeCalibration, which allows you to influence the hand-eye calibration (and thus the viewing angle of left/right eye on 3D-capable devices) using a translation and a rotation component.
The "hand-eye calibration" defines the transformation from camera space to eye space, which is necessary to render AR correctly e.g. on devices like smart glasses where the camera and eye projector positions and orientations (viewing directions) are distinct. So if for instance, a device has one projective display for each human eye, and a camera positioned on the right temple of the glasses, the translation part for each eye's calibration would most probably be positive (e.g. 10mm for right eye, and 70mm for left eye).
The setStereoRendering(true) call automatically applies an initial hand-eye calibration for known devices. For example, SDK 5.5 introduced a calibration for the Epson Moverio BT-200 glasses. Mind however that SDK 6.0 does not automatically apply a calibration anymore, since we found that BT-200 devices are not manufactured equally and hence there is a need for per-device calibration. Please check the changelog for more updates on this.
In the example code, we use contrived values. They should be replaced by the values for the actual device in a real world scenario, or even better: use SDK 6.0.1 or newer, together with the recommended approach as documented below.

Android (Java)

  Copy Code 
// Adjust stereo calibration (i.e. difference in view of camera vs. left/right eye).
// These are contrived example values. Real values should be gathered by an exact
// calibration. Note that for typical scenarios, e.g. AR/VR glasses where the camera has
// a translation to left/right eye, the camera image is still rendered as for the mono
// case (it is not transformed by the hand-eye calibration to look correct). Therefore
// on glasses, see-through mode should be enabled (see below).
// This is just a contrived example. The values are overwritten below, using the recommended
// approach.
metaioSDK.setHandEyeCalibration(
 new Vector3d(70f, 0f, 0f),
 new Rotation(0f, -18f * (float)Math.PI/180f, 0f),
 ECAMERA_TYPE.ECT_RENDERING_STEREO_LEFT);
metaioSDK.setHandEyeCalibration(
 new Vector3d(10f, 0f, 0f),
 new Rotation(0f, 7f * (float)Math.PI/180f, 0f),
 ECAMERA_TYPE.ECT_RENDERING_STEREO_RIGHT);

Adjusting the Rendering Angles [SDK 6.0.1 and newer]

The "hand-eye calibration" defines the transformation from camera space to eye space, which is necessary to render AR correctly e.g. on devices like smart glasses where the camera and eye projector positions and orientations (viewing directions) are distinct. So if for instance, a device has one projective display for each human eye, and a camera positioned on the right temple of the glasses, the translation part for each eye's calibration would most probably be positive (e.g. 10mm for right eye, and 70mm for left eye).
SDK version 6.0.1, together with Toolbox for Android 6.0.1, introduce a new and recommended approach to set the hand-eye calibration.
Toolbox allows creation of calibrations directly on the device, e.g. supported for the Epson Moverio BT-200. After you've finished calibrating a device, Toolbox stores a calibration file at a fixed path on the device (typically "/sdcard/metaio/hec/hec.xml"), where the SDK can load it by simply calling setHandEyeCalibrationFromFile() without parameters. Since stereo devices may need per-device calibration (such as the BT-200, since they are not manufactured identically), we recommend the following approach to set the hand-eye calibration (in this order, exit if a step succeeds):
  1. Load your own, exact calibration (calibration XML file created with Toolbox 6.0.1 or newer), i.e. you as developer provide a calibration file. Note that the path to "hec.xml" doesn't actually exist in this example; it's only there to show how to apply a custom calibration file.
  2. Load calibration XML file from default path, i.e. in case the user has used Toolbox to calibrate (result file always stored at same path).
  3. Load calibration built into Metaio SDK for known devices (may not give perfect result because stereo glasses can vary).
Depending on your use case, you may want to skip step 1, or extend it with a mapping between device identifier and calibration file. Check the stereo calibration article for more ideas.

Android (Java)

  Copy Code 
final File calibrationFile = AssetsManager.getAssetPathAsFile(this, "TutorialStereoRendering/Assets/hec.xml");
 
if ((calibrationFile == null || !metaioSDK.setHandEyeCalibrationFromFile(calibrationFile)) &&
 !metaioSDK.setHandEyeCalibrationFromFile())
{
 metaioSDK.setHandEyeCalibrationByDevice();
}

Importance of adjustments for stereo devices

A hand-eye calibration ensures the augmentation, e.g. a 3D model, is displayed at the correct position and scale so that the human eye recognizes the left/right half images (imagine the screenshot below without the camera image) as one "real" object. Therefore, in contrast to mono rendering (AR for mobile phones or desktops), it is absolutely important to configure tracking in a way that physical scale is correct. This means if you are for instance tracking a magazine cover of physical size 210x297mm, but forget to specify the size of the reference image, the Metaio SDK will take the dimensions in pixels of the reference image as size in millimeters, which is in most cases not correct. Instead, you must specify the physical size in the tracking configuration - for markerless tracking, it would look like so:
<ReferenceImage WidthMM="114" HeightMM="114">metaioman_target.png</ReferenceImage>
For ID marker tracking, like this:
<MarkerParameters><MatrixID>1</MatrixID><Size>80</Size></MarkerParameters>
For 3D map tracking, you should use Metaio Toolbox to record, and then adapt the map to millimeter coordinates ("Align map" feature).
If you develop for a device that requires a hand-eye calibration, e.g. the BT-200, and tracking is not configured to match physical millimeter coordinates, you will see a wrongly scaled and shifted pose in the best case, or in the worst case the human eye will recognize e.g. an augmented 3D model as two distinct objects.
The reason is that, if the distance between camera and trackable image is not recognized correctly (e.g. SDK thinks it's 300mm when it's actually 700mm), the hand-eye calibration will influence the position of the augmented 3D objects more or less than it should, which leads to the effects just described.

See Through

In stereo rendering mode, displaying the camera image does not make much sense. Therefore it is recommended to enable the see through mode with setSeeThrough(true). For debugging purposes, see through can be disabled with setSeeThrough(false) in order to always render the camera image – which will however not be aligned to the augmentation in case a hand-eye calibration is set.

Android (Java)

  Copy Code 
// Enable see through mode (e.g. on glasses)
metaioSDK.setSeeThrough(true);
metaioSDK.setSeeThroughColor(0, 0, 0, 255);



Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)