The magic behind Kinect

Today at ETH, Jamie Shotton talked about Kinect, previously know as Project Natal, the new Xbox 360 add-on which turns your body into a game controller.

The device consists of two cameras, one of which is an infra-red depth camera. Using some clever processing (done using the Xbox 360's GPU mainly), it is able to create 3D models of multiple players in real time. No training or calibration is required.

Based on the depth image from the infra-red camera, an algorithm which was trained with one million images of different people in different poses, does the body part recognition. For each pixel, the corresponding body part is determined by processing the region surrounding it. From that the joint positions are estimated and the 3D skeleton constructed. Even if the algorithm should fail to give correct results in a special situation, it can recover instantly as it doesn't rely on any previous processing as is the case for systems based on tracking movement.