Kinect and MRPT

1. Overview

MRPT implements a common C++ interface to Xbox Kinect, a new RGB+D sensor with an immense potential in mobile robotics. The implementation offers the user only one unified interface. Behind the scenes, MRPT uses one of these libraries to actually access the sensor:

• OpenKinect's libfreenect on Unix/Linux and Windows.
• CL NUI on Windows.

Intensity (RGB) images are automatically converted to the OpenCV's format (mrpt::utils::CImage), 10bit or 11bit raw depth is converted to meters, and methods are provided to handle the two matrices of camera parameters (one for the RGB camera and one for the "range image").

Observations grabbed from MRPT's Kinect C++ class can be directly saved, among other types, in rawlogs for grabbing robotics datasets, or can be used online for any real-time application. On performance, full speed 30fps is achieved without problems under both Linux and Windows with libfreenect, but a multi-core processor is highly recommended if something else has to be done in real-time with the huge amount of received data. 

Note: Kinect is supported since MRPT 0.9.3. Also, check out the Ubuntu PPA repository for possible often updates.
 

2. Example code and documentation

• Tutorials:
  • Tutorial: Grabbing your own Kinect datasets.
  • Tutorial explaining the different pathways for generating 3D point clouds from RGB+D images.
  • Kinect calibration tutorial (New: Jun-2012).
• Examples:
  • Example application: kinect_3d_view. Sample program demonstrating grabbing from Kinect and live 3D point cloud rendering.
  • kinect-3d-slam: A demo application for building small 3D maps by moving a Kinect.
  • Example code of how to switch between grabbing from a Kinect (online) and from a previously recorded dataset (offline).
  • How to convert Kinect 3D ranges to 2D "fake" laser scans?
• Related C++ API:
  • mrpt::hwdrivers::CKinect: The doxygen documentation of the CKinect C++ class provides details on usage.
  • mrpt::slam::CObservation3DRangeScan: The data-type of the observations (readings) from the CKinect class. The doxygen docs contain drawings of the convention used for the cameras axes orientation.

3. Installing in Windows

To enable Xbox Kinect in MRPT for Windows, you'll need either OpenKinect's libfreenect or CL NUI SDK: http://codelaboratories.com/nui.
Before January 2011 (MRPT 0.9.3), the only choice in Windows was CL NUI. That SDK is much easier to install than freenect, but lacks some important features like grabbing the IR channel (needed for a proper calibration), so I would rather recommend freenect.

3.1. With OpenKinect's libfreenect (RECOMMENDED)

A) Driver installation (Required either if you install MRPT precompiled packages or build it from sources)

1. Download and extract the latest libusb-win32-bin-x.x.x.x.zip (it has binaries for both 32 and 64bit Windows). MRPT programs will need some DLLs in that package to be in the system PATH, so add libusb-win32-bin-x.x.x.x/bin/{x86 or amd64}/ to your system PATH.
2. Download the libfreenect sources (for example: from github). You'll only need a few files under the /platform subdirectory there.
3. Follow these "Driver installation" instructions (copied from the libfreenect step-by-step installation guide for Windows):

Windows 7: step-by-step walkthrough (should also work with Windows XP!) Plug in your Kinect. Windows will warn that no device driver was found for the plugged device (the LED of Kinect will not turn on). If Windows presents a dialog asking to search for drivers, simply cancel it. Open the Device Manager: Start >> Control Panel >> System and Security >> System >> Device Manager A device called "Xbox NUI Motor" should be somewhere there (most probably be under "Other devices") with a small yellow warning symbol "!" on top of its icon. Right click on it, and select "Update Driver Software...", then click on "Browse my computer for driver software". "Browse" and select the folder where the "XBox_NUI_Motor.inf" is located (/platform/windows/inf inside your libfreenect source folder). Click "Next" and if warned by Windows that a non-certified driver is about to be installed, just order it to install it anyway. After that, the Kinect LED should start blinking green. Now there will be two additional devices in the Device Manager list: "Xbox NUI Camera" and "Xbox NUI Audio". Repeat the above instructions for them as well.

B) CMake configuration (Required only if you're building MRPT from sources)

Notice that MRPT 0.9.4+ does NOT require the sources of the "freenect" project, since there exists an embedded copy of them.

In MRPT CMake configuration (from cmake-gui), enable BUILD_KINECT, click "Configure" and verify that "BUILD_KINECT_USE_FREENECT" is marked. Two new variables will appear:

• LIBUSB_1_INCLUDE_DIR: Select the path to the libusb header folder named "/include".
• LIBUSB_1_LIBRARY: For instance, to configure for MS Visual Studio 2010, use /lib/msvc/libusb.lib or if you plan to use mingw, use /lib/gcc/libusb.a as the libusb library path.

Click configure again and no errors should appear now.

If at some point you have errors in MSVC 2008 due to a missing stdint.h file, download this file pstdint.h , rename it to stdint.h and save it, for example, in OPENKINECT_LIBFREENECT_DIR/include. You can also replace those #include <stdint.h> with #include <mrpt/utils/mrpt_stdint.h> which is assured to always work disregarding of the compiler version.

Notes on libusb for Windows:

• If you build for win32 (not win64), at least in the present version of libusb I had to rename libusb-win32-bin-1.x.x.x\bin\x86\libusb0_x86.dll to libusb-win32-bin-1.x.x.x\bin\x86\libusb0.dll .
• Since libusb-win32-bin-1.2.5.0 they have replaced usb.h by lusb0_usb.h, which will break existing code:

    fatal error C1083: Cannot open include file: 'usb.h': No such file or directory
  A very easy solution is to create a new header in libusb-win32-bin-1.x.x.0\include\usb.h containing just one single line:
    #include "lusb0_usb.h"

3.2. With CL NUI SDK

After downloading and installing, CMake should find it while you configure MRPT. If it's not done automatically, enable BUILD_KINECT, click "Configure", then "BUILD_KINECT_USE_CLNUI", then "Configure" again and set the variable CLNUI_SDK_DIR pointing to the installation directory. Remember to add the "CL NUI Platform\SDK\Bin" directory to the system PATH.

4. Installing in Unix/Linux

Just install libusb-1.0-0-dev. In Ubuntu:

 sudo apt-get install libusb-1.0-0-dev

OpenKinect's libfreenect is currently embedded into MRPT sources.

Accessing the device requires root privileges (executing as "sudo"). To avoid that, install this file: 

 http://mrpt.googlecode.com/svn/trunk/scripts/51-kinect.rules

5. Data sets 

• A collection of 33 Kinect datasets with 6D ground truth, available in the Rawlog format (how to read it), by the CVPR team at TUM.
• See also: The RGBD_dataset2rawlog converter tool.