Using eye tracked virtual reality to classify understanding of vocabulary in recall tasks

Jason Orlosky, Brandon Huynh, Tobias Hollerer

Research output: Chapter in Book/Report/Conference proceedingConference contribution

11 Scopus citations

Abstract

In recent years, augmented and virtual reality (AR/VR) have started to take a foothold in markets such as training and education. Although AR and VR have tremendous potential, current interfaces and applications are still limited in their ability to recognize context, user understanding, and intention, which can limit the options for customized individual user support and the ease of automation. This paper addresses the problem of automatically recognizing whether or not a user has an understanding of a certain term, which is directly applicable to AR/VR interfaces for language and concept learning. To do so, we first designed an interactive word recall task in VR that required non-native English speakers to assess their knowledge of English words, many of which were difficult or uncommon. Using an eye tracker integrated into the VR Display, we collected a variety of eye movement metrics that might correspond to the user's knowledge or memory of a particular word. Through experimentation, we show that both eye movement and pupil radius have a high correlation to user memory, and that several other metrics can also be used to help classify the state of word understanding. This allowed us to build a support vector machine (SVM) that can predict a user's knowledge with an accuracy of 62% in the general case and and 75% for easy versus medium words, which was tested using cross-fold validation. We discuss these results in the context of in-situ learning applications.

Original languageEnglish (US)
Title of host publicationProceedings - 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages66-73
Number of pages8
ISBN (Electronic)9781728156040
DOIs
StatePublished - Dec 2019
Externally publishedYes
Event2nd IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2019 - San Diego, United States
Duration: Dec 9 2019Dec 11 2019

Publication series

NameProceedings - 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2019

Conference

Conference2nd IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2019
Country/TerritoryUnited States
CitySan Diego
Period12/9/1912/11/19

Keywords

  • Classification
  • Cognition
  • Eye Tracking
  • Memory
  • Pupillometry
  • Virtual Reality

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Science Applications
  • Human-Computer Interaction
  • Media Technology
  • Modeling and Simulation

Fingerprint

Dive into the research topics of 'Using eye tracked virtual reality to classify understanding of vocabulary in recall tasks'. Together they form a unique fingerprint.

Cite this