71

For instance, would it be possible for an app to determine what pixel range on a smartphone display a user is looking at by analysing their eyes with the front facing camera? If so, with what kind of precision?

It would be very discomforting to know that apps could collect data in the background on how your eyes respond to displaying certain advertisements.

Laurent
  • 713
  • 1
  • 5
  • 7
  • 3
    A spyware don't need to go through those trouble. It can silently make a screen shot of what you are looking at. – mootmoot Nov 16 '17 at 11:35
  • 41
    I dont think that's what the OP means @mootmoot . They are wanting to know could it know which parts of the screen you were looking at - so they could for example know which adverts attracted your eye – ISMSDEV Nov 16 '17 at 11:41
  • 2
    Indeed, I'm not talking about copying the user's screen but rather analysing the specific content they are looking at in real time which is a whole other thing. – Laurent Nov 16 '17 at 11:45
  • 2
    I wonder whether this kind of technology would be accurate enough to see your password when using an on screen keyboard – User1 Nov 16 '17 at 14:36
  • @user1 Probably not today but it might become an interesting issue in the future when accuracy improves. – Laurent Nov 16 '17 at 18:15
  • 1
    I don't find that discomforting in the context of advertisements. Let's say they gather data on what catches my eye. What's the worst that I've to fear? (I'm actually wondering if I'm missing something here). Knowing my location, contacts, or message data is one thing, but my eyeball rotation is the least of my concerns unless someone is indeed able to learn my passwords or other actually sensitive data. – Bort Nov 16 '17 at 18:20
  • 2
    @user1 Using secondary sensors as a means of determining user input with fair accuracy is definitely not an abstract risk. The accelerometers in smartphones can theoretically be used to reconstruct typing on phone keyboards and even keyboards the phone is adjacent to on a desk. See https://www.wired.com/2011/10/iphone-keylogger-spying/ https://www.cs.swarthmore.edu/~aviv/papers/aviv-acsac12-accel.pdf and http://www.techradar.com/news/scientists-find-a-way-to-crack-your-phones-password-using-just-the-accelerometer – pwdst Nov 16 '17 at 18:25
  • @Bort - see my comment on my answer regarding a political activist. Where you look in adverts isn't a major risk but this kind of access can be for multiple other reasons. – Hector Nov 17 '17 at 09:49
  • 2
    I hope they count how many times I roll my eyes at their ads. – Octopus Nov 17 '17 at 21:47
  • @Laurent Youv'e got plenty to worry about, buddy: https://www.wired.com/story/the-dark-side-of-replay-sessions-that-record-your-every-move-online/ – Craig Tullis Nov 18 '17 at 22:05
  • @pwdst History repeats itself, they were using bugs to detect changes in the magnetic field caused by moving components in typewriters back in the 70s and 80s. The Center for Cryptologic History has a good write-up [here](https://permanent.access.gpo.gov/gpo58656/Learning_From_the_Enemy_The_GUNMAN_Project.pdf). – Kelly Thomas Nov 19 '17 at 15:55

3 Answers3

84

Eye Tracking for Everyone. 2176-2184. 10.1109/CVPR.2016.239. (2016) - Krafka, Khosla, Kellnhofer et al

Our model achieves a prediction error of 1.71cm and 2.53cm without calibration on mobile phones and tablets respectively. With calibration, this is reduced to 1.34cm and 2.12cm.

So yes - it is possible. This particular study was performed using iOS and achieved a read rate at 10–15fps.

There are several companies selling products with similar technology - UMoove for example. It would not surprise me if a higher precision than in the mentioned paper could be achieved.

If you are paticularly concerned a number of smartphone camera covers are available - here is one example.

Hector
  • 10,893
  • 3
  • 41
  • 44
  • Good answer! It would be nice though, if you could add some more context from the paper and the names and title of the paper itself in case the link goes dead. – Tom K. Nov 16 '17 at 11:41
  • Very interesting stuff, this indicates that advertisements can be tailored to how the user responds to them physically using trail/error instead of trying to predict what they might respond to based on online behaviour tracking. – Laurent Nov 16 '17 at 11:48
  • 34
    @Laurent - Personally if ad providers used this technology i'd be far more concerned about them taking pictures of your face if they have access to the camera. Facial recognition would be a possible way for them to unify your data between devices when you have taken explicit steps to prevent this otherwise. – Hector Nov 16 '17 at 12:07
  • 1
    @Hector That's a valid argument. However, don't you think there are easier ways to identify a single person using multiple devices without using any camera? I reckon somebody who really wants to stay anonymous will allow camera access to apps. But seeing as most people don't know about any of this (or simply don't care), advertisement networks will have plenty of pictures on them already. The only way for them to improve is by gathering new kinds of data. – Laurent Nov 16 '17 at 12:30
  • 18
    @Laurent - online political activist uses a different device to avoid tying to real life identity. Add nothing from their personal life to it, tunnel all connections through a VPN. They have facebook & twitter apps installed to post to social media - both request access to the camera as they allow sharing of photos. Maybe our user has shared a photo from a political rally. If these companies used facial recognition during normal use then unknown to the user facebook/twitter could have paired their alias with their real name. And this could be requested by state authorities. – Hector Nov 16 '17 at 12:56
  • 2
    Note I: For this to happen the app needs to be able to access the camera. Note II: I assume a higher permission can be achieved on iOS than on Android because Android devices come in way more different form factors and thus makes it more difficult to optimize the algoritmes to work on all those devices. – Rolf ツ Nov 16 '17 at 14:28
3

Some 20 years ago Canon implemented in their yet analogue SLR cameras the autofocus system, that focuses on the object the photographer was looking at. The user had feedback what direction was chosen.

So yes, it is possible to determine it, even if the accuracy may be questionable. There can be adviced to deny camera permission to all applications​ but those that really need it.

Poutnik
  • 131
  • 4
  • An eye on a viewfinder is a different thing than a pair of eyes glancing at a screen from distance, though. – Bergi Nov 16 '17 at 18:58
  • 1
    Yes, they are different. But it is not a contradiction. – Poutnik Nov 16 '17 at 19:55
  • The autofocus does not track the photographer's eyes. In the past it would simply use the autofocus point selected by the photographer, usually the one in the middle of the viewfinder. More recently the cameras are able to choose whichever autofocus point they think gives the best result, using a computer-based algorithm. I suggest posting on [photography.stackexchange.com](https://photography.stackexchange.com/) for more information. – Micheal Johnson Nov 17 '17 at 08:40
  • 1
    What you have written is generally true, but not in this case. This system explicitly chose the point selected by the eye. – Poutnik Nov 17 '17 at 08:44
  • 1
    Just a link to verify that this is actually true https://www.dpreview.com/articles/6531126959/looking-back-canons-eye-controlled-focus – qwazix Nov 17 '17 at 18:35
2

As Hector answered it is certainly possible, however iOS and latest Android allow you to disable permissions for an app. This (in the absence of system vulnerabilities) would prevent any such app using the camera.

As a general rule giving an app's requested permissions a once over is good practice. If your free card game needs access to your camera then it's probably not the right one to install.

ste-fu
  • 1,092
  • 6
  • 9
  • While true, some of the apps that would be able to most directly use this eye-ad info would be things like facebook, etc, which most people would naturally grant camera access to, since it does also include camera features. – Geobits Nov 16 '17 at 20:23
  • Facebook doesn't need eye tracking on mobile...they can just record the amount of time each post spends on screen. – ste-fu Nov 16 '17 at 20:43
  • Being on screen is not the same as being looked at. Facebook and its advertisers would be *very* interested in knowing which ads people actually look at. – Stig Hemmer Nov 17 '17 at 08:22
  • 2
    @StigHemmer Correct on a monitor, but on even a decent sized mobile there is normally only one post that is properly visible. Plus they can - and do - (it's data they give to their paying customers) record the scrolling speed and any variations to get a very accurate picture of how long you spend looking at any one post – ste-fu Nov 17 '17 at 09:49
  • @ste-fu surely it shows both posts and ads at the same time? If so, how does it know which one you're looking at? – Kat Nov 18 '17 at 18:29
  • @kat ...not at all..on mobile it is literally one mostly visible post at a time – ste-fu Nov 18 '17 at 22:21
  • @Kat most of the content on facebook is ads in one form or another, and the rest of it is useful for ad targeting. – hobbs Nov 19 '17 at 00:05
  • I see using fb as an example might have been misguided. I simply meant that there are plenty of apps that would naturally have both camera features and ads. – Geobits Nov 20 '17 at 14:25