General purpose sensor "Synthetic Sensor" that implements "smart environment" by installing various sensors on a terminal without camera

International conference in an important position in the field of human-computer interaction (human-computer interaction)CHI2017And Google also has invested "Synthetic Sensors Project"Was announced. Although smart home appliances that connect to the Internet and work with smartphones and the like are increasing, the feature of this sensor is that the camera is not installed in consideration of privacy. Instead, it has become possible to realize a "smart environment" with fulfilling sensors.

Gierad Laput | Synthetic Sensors
http://www.gierad.com/projects/supersensor/

Synthetic Sensors Project Presented at CHI 2017 | Human-Computer Interaction Institute
https://www.hcii.cmu.edu/news/2017/synthetic-sensors-project-presented-chi-2017

You can see what you can do with Synthetic Sensor by watching the following movie.

Synthetic Sensors: Towards General-Purpose Sensing - YouTube


"Have you kept the refrigerator door unopened?" "Did you have a coffee?" "There was a fire in the stove properly erased ..." There are lots of points to watch out for around the kitchen.


That helps smart home appliances that work with smartphones. However, we can not replace all kitchen appliances.


Even if you think "If you use a sensor", it seems that you will be healthy if you think of the number of sensors to install ... ....


The Synthetic Sensors Project solves this problem by placing infrared sensors, color sensors, magnetometers, temperature / humidity / barometer, 6 axis inertial measurement, Wi - Fi, approach detection, analog microphone etc on one terminal.


You can see that there are many items that can be detected even in comparison with similar sensors. However, no camera is installed. When thinking about use in a wide range of places such as home, school, hospital and office, privacy problem comes out and it is based on the idea of ​​"impractical".


There is a USB terminal on the back of the "Synthetic Sensor" in which the board is exposed, so insert it into an outlet using the USB-AC adapter that is also used with iPhone or Android charger.


Preparation is complete. Response of various sensors is displayed on the right side of the screen.


When a man standing in the sink starts to drain water ......


The sensor detects its motion.


When moving the garbage disposal machine installed under the sink, the reaction appears obviously on the accelerometer (the upper three on the right side of the screen).


Men who move mixers, coffee blenders and more. When using a gas stove the temperature sensor reacted.


However, this does not understand what the sensor is reacting to because of what. Therefore, we used human input and machine learning in the project so that we can determine what is responsive to the sensor. As the result is like this, as soon as water is released, a message saying "a faucet is in use" appears.


The garbage disposal machine is at an invisible angle from the camera, but you can see that it is operating with the notebook message.


It is convenient that on and off of a stove that is hard to see if it is far is displayed with a message.


The place of use is not limited to the kitchen. In the washroom it is also distinguishing whether it is the left faucet or the right faucet.


Since the machine room learns the operating sound of each machine tool, you can see which machine is moving now.


Also, knock sound is detected.


"I am writing something on the whiteboard" ... ...


I detected even the operation "I am erasing the whiteboard".


If further evolving, count how many paper towels you used ... ...


You can also announce that "the remaining amount is low."


If you count the water usage ... ....


Everyone can compete "how much water can be saved".


In the case of a microwave oven, you can simply notify you of the status of "in use" and "warming completion", not just "opening and closing the door".


The one that illustrates the mechanism is Kore. "Usable" if the door is closed, "in use" if the microwave comes out next. It is "interrupted" if it could be opened before the sound notifying that it warmed up, "warming done" if the sound sounded.


Even if multiple home appliances operate at the same time this way.


Professor Chris Harrison of Carnegie Mellon University Human-Computer Interaction Institute (HCII) and PhD Dr. Gierad Laput and Dr. Yang Zhang. After prototype development, we learned the operation sound of 38 kinds of different terminals. According to Professor Harrison, it seems to be "artificial intelligence", the best estimate was made based on the data even if there was new behavior, the detection accuracy was close to 100% in the initial test is.