MIT develops low-cost, high-speed hand motion recognition system with colorful gloves

It is becoming common to operate devices with gestures like Nintendo's "Wii" shaking the remote control to play games.

And a research team at MIT has developed a system that recognizes hand movements that can be handled very quickly. Since the only things required for operation are colorful gloves and webcams, it is a much cheaper system than before.

Details are as below.
Gesture-based computing on the cheap

In the system developed by MIT's Robert Wang and Jovan Popović, using only one webcam that is standard in colorful gloves and laptops, the image captured by the camera can be displayed without any delay on the hands in three-dimensional space It can convert it to data, even if the location or user changes, it can be calibrated.

Gloves and webcams necessary for recognition.


I tried wearing gloves. A three-dimensional object from which the left is the camera image and the right is obtained from it. It is properly in accordance with the image of the camera. The gloves consist of twenty complicated shapes of pieces using ten colors, and colors and shapes can be distinguished even if they are influenced by the state behind the hand or the light source seen from the camera, and furthermore, It seems that they are carefully selected so that the colors do not overlap as much as possible.


As a procedure to acquire hand information, first remove the background from the image acquired by the web camera, cut out only the glove, and then reduce the size to 40 x 40 pixel image data. All that is left is to retrieve the image data from the pre-registered database. Since the retrieved image has detailed information on the shape of the hand, it is not necessary to calculate the position of the finger etc. by calculation and it can be obtained at high speed.

The problem at the moment is that the image database is somewhat large. The image itself is very small, 40 × 40 pixels, but it is hundreds of megabytes because the number is very large.

Similar gesture-based user interface research required the installation of multiple cameras in the room, but with this approach only one webcam is required. In addition, we can only take information on fingertips in research that we wear color tapes on another fingertip and recognize with camera, but with this method you can also obtain the position of the back of the hand.

Since it can be realized only with gloves and webcams, games using this method may come out from now on. In the block piling up demonstration in the linked movie, this method is used to manipulate three dimensional objects in real time.


The result of pinching the block and raising it.


Demonstration of disassembling operation of tire parts. It seems that rotation operation can be done by using both hands.


I grabbed the part and took it out.