A zero-shot deep metric learning approach to Brain–Computer Interfaces for image retrieval

摘要

In this paper we propose a deep learning based approach for image retrieval using EEG. Our approach makes use of a multi-modal deep neural network based on metric learning, where the EEG signal from a user observing an image is mapped together with visual information extracted from the image. The inspiration behind this work is the vision of a system which allows the user to navigate their image catalogue just by thinking about the image they want to see. Thanks to our metric learning approach, the system is scalable in that it can operate with new images that have never been used in training, resulting in a zero-shot image retrieval system. This framework is tested in two different standard EEG image-viewing datasets, where we demonstrate state-of-the-art results in this complex scenario.