I tried 'kbd-audio' which can judge which character was input from the keyboard tap sound



If the key entered by the keyboard is detected, you will be victimized such as parsing the password. Software " kbd - audio " which predicts the input key in real time by detecting the key typing sound using the PC microphone is released on GitHub.

GitHub - ggerganov / kbd - audio: Tools for capturing and analyzing keyboard input paired with microphone capture
https://github.com/ggerganov/kbd-audio

kbd-audio is software that lets you learn the key input sound by machine learning, and guesses the key you entered in real time. If you have a PC with a microphone that can pick up the sound of keyboard input, you can predict the key. Since "live demo" version which can be used with the browser is published in kbd - audio, I tried it immediately.

Because kbd-audio needs Threads of WebAssembly to be enabled, browser used " Google Chrome Canary " for Live demo.

First of all, in order to enable Threads of WebAssembly, enter "chrome: // flags / # enable - webassembly - threads" in the address bar of Chrome Canary and change the highlighted "WebAssembly threads support. Enabled "and restart the browser.



Next, open the following Live demo page.

Keytap: acoustic keyboard eavesdropping | C ++ and stuff
https://ggerganov.github.io/jekyll/update/2018/11/24/keytap.html

You are asked to use the microphone, so click "Permit".



It is OK if "Demo" window which was not displayed until WebAssembly Threads is activated is displayed.



In order to predict key input with kbd-audio, it is necessary to let you first learn the type sound. Keep the keys one by one firmly as you type and press "Predict" when you finish pressing most of the keys.



During logging in, if you see a log on the "Standard output" under the Demo window, you can get the key sound. If there is no response, you can not recognize the key sound, so you need to reconsider the setting of the microphone etc.



Since the training work is done using the input data, wait for a while.



When the virtual keyboard is displayed in the Demo window, it becomes testable.



If you key in with the keyboard as a test, the candidate key will respond in green on the virtual keyboard. Among the several candidates, the one with the key displayed in red seems to be the "most likely" prediction.



Below "Main", the waveform of the detected sound is displayed.



Since the waveform is slightly different depending on the input key, it seems to predict the key input from here.



As for the accuracy of the key point, there are many cases that you can not recognize the key input, and even if you recognize it well, the frequency of correctly correcting the key was not so high. The notebook PC ( Let's Note CF - SZ 6 ) used in the test, the key input sound is not so big as it is a pantograph type keyboard, and that the cooling fan raised the beat when picking up Live Demo and picking up the sound There may also be factors due to the environment, such as becoming difficult, and the surrounding noise is not zero.

Depending on the environment, it seems that the following key input guessing becomes possible as well.



However, since we predict the keys in the vicinity where the key you have entered, you can detect the sound better and repeat the training more, you will make it more likely that the accuracy will increase. It is hard to say that kbd - audio has reached the "practical" level, such as not supporting continuous key input, but confirm that there is a risk that "information will be read from the keyboard input sound" It was soft that let me do.

Related Posts:

in Software,   Security, Posted by darkhorse_log