Hackers could possibly use your computer’s microphone to “see” your screen

We’ve all heard about the dangers of webcams and why we should cover them when not in use, but now there’s a new attack that can reliably know what’s on your computer’s screen by using its microphone.

In what seems like witchcraft, a group of researchers has figured out how to reveal the contents of your screen by listening to “content-dependent acoustic leakage.” They’ve named the side-channel attack “Synesthesia” and how it works is pretty darn nifty.

Here’s how it works:

The side-channel attack leverages what’s known as “coil whine,” which is the audio emissions from transformers and other electronic components that power the device’s LCD display. Due to how a computer renders a display, with signals being sent to each pixel of a line with varying intensity levels for each sub-pixel, as the monitor goes through its refresh scans the power sent to each pixel fluctuates.

That fluctuation changes the sounds created by the power supply for the screen, which inadvertently leaks data about the image being refreshed, through the microphone.

If that audio is captured by an attacker and fed into a machine learning trained model – the model can accurately recreate what’s on the screen. Just having the audio alone won’t cut it. The researchers applied machine learning to three different types of attacks, and demonstrated that a surprising amount of data can be reconstructed.

Examples of accuracy:

For example, in one attack they managed to reliably identify (96.5% accuracy) which of the Alexa top 10 websites was on a screen based on audio captured during a Google Hangouts call. Typed keystrokes were also able to be captured in another attack, with a 96.4% accuracy while a device was in portrait orientation. The last attack tried to deduce what text was shown on the remote screen, which again resulted in a scarily high level of accuracy.

The per-character validation set accuracy (containing 10% of our 10,000 trace collection) ranges from 88% to 98%, except for the last character where the accuracy was 75%. Out of 100 recordings of test words, for two of them preprocessing returned an error. For 56 of them, the most probable word on the list was the correct one. For 72 of them, the correct word appeared in the list of top-five most probable words.

While the researchers only used a single monitor type in their testing, they also demonstrated that a “cross screen” attack is possible by calibrating a baseline for an unknown screen type. Pretty scary stuff.

With more and more reliance on our mobile devices, which already come equipped with a pretty sensitive microphone, I’m sure this isn’t the last we’ll hear about this type of attack. Mitigating it would require re-engineering the display technology we currently use.

For most of us, the risk of this attack is fairly slim. For anyone that works with sensitive data, maybe just don’t look at anything while you’re on a Hangouts call, k?