Making Sense of RealSense

At CES 2014, Intel showed off its RealSense camera which will be integrated into Intel PC’s, laptops and tablets. The RealSense camera is a full 1080 camera with infrared sensors that allow for depth information to be processed. We’ve seen this before in other tech like the Kinect or the Leap Motion sensor, but the RealSense is the first to be integrated into devices people are likely to buy anyway. It’s also more than just a depth camera. RealSense technology is a full package that presents a new way to interact with our devices. With our relationships at Intel, we’ve been involved in some early R&D with the RealSense and are blown away by its quality. Here are some of our observations.

Gesture Recognition

Interact with your device more like you interact with people.”

Gesture

The RealSense 3D camera accurately tracks the joints in your hand in order to figure out exactly what motions you’re making. That’s 4 joints per finger, plus palm and wrist joints. There are a few familiar gestures that it recognizes by default, such as a “thumbs-up/down, a peace sign, open/closed fist, a wave motion, etc. Of course, developers are not limited to these gestures and are free to program their applications to recognize custom motions.

Gesture-based interaction can be a lot of fun for games or educational kids apps, but many are skeptical of whether it’ll ever replace the speed and accuracy of a mouse or a touch-based interaction. By integrating this technology into their devices, Intel are hoping to make this a more mainstream approach.

Face Tracking

The RealSense camera knows how you’re feeling.

Facial analysis is one of the things RealSense does very well. It tracks 78 points on the face and calculates the roll, pitch and yaw of the head. From those feature points, it’s able to tell what emotion you’re displaying - whether you’re happy, sad, surprised or angry. It can also recognize one face from another. So the technology can be used for things like user verification or cataloguing your photos based on the people in them.

Another cool application of facial tracking is in the 3D animation world. Since there is enough data to know what your face is doing, we can now apply that to a 3D model and create a dynamic avatar that mimics your facial movements. It works really well too! Check out the demo in the video.

Although RealSense has accurate hand and face tracking, it falls short from other depth cameras like the Kinect with its inability to track a full body. While a Kinect can map an entire skeleton and immerse your whole body in an experience, the RealSense camera is more tailored for “shoulders and up” tracking. This makes sense though in a way, since RealSense cameras are integrated cameras being marketed more for a more up close and personal use of your tablet or laptop. 

Speech Recognition & Synthesis

RealSense technology also adds voice recognition capabilities to applications. You can program your application to recognize certain commands, and you can also use it as a dictation tool. The reverse use is also possible with speech synthesis, which allows you to type words that the software will speak for you. For this component, Intel uses Dragon Assistant technology made by Nuance. 

Snapshot (3D Photos)

Another cool feature of RealSense adds a depth component to regular photos you take with your device. Photo editing apps are one of the most popular categories in application stores, so I think people will enjoy this one. By storing 3-dimensional information about your photo, RealSense allows you to edit certain features of your app after you’ve taken it. For example, you change the focus point in a photo, making the background more blurred and bringing out the part of the photo you want to stand out. 

Snapshot

Photos can also tell you measurements of objects in the pictures, since the depth camera is able to understand the environment in more dimensions than a regular camera.

Detecting which objects are background and which are foreground also makes background replacement possible. This feature is also available on its own and can be used in other applications in real time. For example, if used in a video chat app, you can eliminate the background of your environment and replace it with something more fun (or something more professional perhaps). I’ve tested some tools that attempt to do this, and I have to say, RealSense does it more accurately than anything else I’ve seen. 

3D Scanning & Printing

3D printers are becoming more and more popular, and Intel are hoping to make it more available to the average consumer. The RealSense camera lets you scan any 3D object in the real world and it will create a 3D model matching its shape. What you do with the model is up to you; have it printed, share it  with someone, use it in a game you’re developing (or playing). You could even scan your face and generate a 3D version of yourself.

Metaio

Intel also uses the 3D object tracking technology developed by Metaio, a leading provider for Augmented Reality developer tools. Developers can write applications that recognize a 3D object when scanned and interact with it virtually.

In the end, Intel is a good balance between accuracy and good user experience. You get good results without having to place the user under many restrictions (ie. environment, distance from camera, restriction of movement, etc). It’ll also be the first to integrate this type of camera into an everyday device. If Intel’s devices sell well, it’ll definitely contribute to the exposure of this tech and to making it more mainstream. The first device housing a RealSense camera is set to be in the market in the next couple of months.