Canvas Based Authentication Methods with Deep Learning

In the last years, canvases have shown promising results with regard to device fingerprinting. A canvas is an HTML5 element that can be manipulated by JavaScript to draw anything. An interesting fact is that the browsers draw the same canvas with small differences from each other depending on OS/graphic card/drivers version/browser version etc. A sample canvas is shown below.


In this project, we develop new authentication methods that are seamless and based on canvases and deep learning.

The goal is to have a system that can identify passively a device without user interaction. The way a device draws a particular image will be secret in our authentication protocol. It is a two-phase protocol, the first phase is to learn the secret: a device is asked (passively) to draw some canvas. The results are sent back to the server where machine learning is used to "learn" the secret. 

The second phase is the challenge-response protocol to authenticate the user where a new canvas is drawn by the user and sent back to the server. Then, the server decides if it is a valid drawing for that particular user. 

The methods are developed with the intent of weak authentication. This means that the developed authentication methods are intended for non-sensitive services. They can be part of a larger authentication chain, where they are used as a light authentication test and have the other methods in the chain as fallback for stricter authentication.



For this we employ deep learning applied to the images generated by the devices drawing on the canvas. We explore two distinct approaches; convolutional neural networks for classification and deep convolutional auto-encoders.

The main idea behind using classification is not so much to actually classify a user, but to find the probability that a user is who they claim they are. The output of a trained classifier can be the probability that an instance is a member of a class. That probability can be used to decide how authentication occurs. The output of the classifier can be thresholded so that users with low probability will be referred to a secondary authentication system for further authentication.

A similar effect can be achieved by the use of auto-encoders. An auto-encoder learns a mapping of an input to a lower dimensional space. In a nutshell it is able to extract useful features so an input can be encoded using only the learned features. In our case, we can use an auto-encoder to learn useful features of canvases and then encode the input to a lower dimension. We expect canvases coming from the same user to have similar encodings. In this case when a user tries to authenticate by drawing a new canvas, what we are interested is the "distance" of the new canvas encoding compared against what the system knows for that particular user. Specifically, if the encoding "distance" is large, it is likely that the user is not who they claim to be and should be referred for secondary authentication.

It should be noted that for both approaches, classification-based or auto-encoders, the final decision on authentication is performed by a separate authentication system. The ML component does not decide which users should be referred for extra authentication steps. The output of both those approaches will denote how likely it is that a user is who they say they are. Based on this the authentication system will make its decision.



For data collection we set up an internal website where students could register their devices. We totaled 80 devices from 49 users which resulted in 187387 canvases.



The convolutional neural network achieves very good results at classifying users. It reaches an accuracy of ~95% and lays good foundations for further research in that direction. 


Auto-Encoders for Hashing

This idea is based on resarch by Salakhutdinov & Hinton on semantic hashing. By forcing a binary representation we expect canvases coming from the same users to have similar encodings.

This approach will do away with one limitation of the deep convnet; every time a device is added to the system the network needs to be retrained. With auto-encoders we only need to save the device encoding.

The results plot the accuracy vs group size i.e. how many canvases are drawn during the authentication step. The threshold for accepting or not a device is set based on the variance of the drawn batch vs the variance of the saved encoding. 

The results of this approach however did not match our expectations and further research is required in setting a threshold/architecture etc. to bring it on par with the convnet.


Future Work

  • Tuning of the architectures: The autoencoders and convolutional networks require a lot of decision making with regard to the hyperparameters. All the deep learning methods in our experiments have the same architecture and configuration.
  • Advanced canvases: In this research we considered only a very simple drawing technique to represent the canvases. It would be interesting to see how our methods would perform with more complex canvases. A nice addition to the canvases are for example shapes or emoijs. Emoijs introduce a point of variation, since they are drawn differently in each operating system. This could help the algorithms make a more accurate decision.
  • Security of the authentication method: The security of the authentication method is not discussed in this project. Nevertheless, it is important to make an assessment of the security of the method. How can the authentication be evaded and is it possible to fake someone's identity? These are questions that could be asked in future research.
  • Large scale evaluation: We evaluated this method with a rather limited dataset. In the real world there is much more variation. A large scale evaluation of the authentication method could give us more insight on the entropy of the canvases and the performance of our system.

For more in depth information regarding this early stage of this work look here.

Involved Members:
Kyriakos Efthymiadis
Yannick Merckx(previously)