Using AI-based NiCATS System to Evaluate Student Comprehension in Introductory Computer Programming Courses

Bradley Boswell, Andrew Sanders, Andrew Allen, Gursimran Singh Walia, Md Shakil Hossain

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This Research to Practice Full Paper presents the use of data collected by our Non-Intrusive Classroom Attention Tracking System (NiCATS) to evaluate student comprehension. Quantifying students' cognitive processes in classrooms in a non-intrusive way is challenging. By analyzing various aspects of the eye metrics against defined regions of interest (ROI), instructors can better understand students' cognitive processes as they acquire new knowledge. Eye-tracking studies primarily define ROIs based on commonly used metrics (source code complexity, significant fixation durations, etc.). While helpful, these metrics, when used independently, do not accurately represent their comprehension patterns. This paper contributes an alternative, multilayered approach for calculating gaze metrics against automatically defined ROIs. The work utilizes the AI-based Non-Intrusive Classroom Attention Tracking System (NiCATS - developed by the researchers), collecting raw-gaze data in real-time as information is presented on a computer screen. This paper reports the results of a study in which undergraduate students in a CS programming course were asked to identify defects seeded in Java programs. Each JAVA program included its own unique sets of ROIS defined using two different granularities: lexer-based and line-based. The ROI sets were then used to calculate relevant eye metrics in the context of each ROI layout. The results of the eye metric analysis at specific ROIs w.r.t their code review task provide insights into the cognitive processes students undergo when trying to comprehend new material. Subdividing this region into lexer-based regions, we determined 'content topics' students struggled with (e.g., using complex data types) in a specific area. This feedback is valuable to the instructor as it enables the ability to identify hard-to-comprehend content topics post-hoc and gives the ability to validate student learning in the classroom. While this experiment focused on students in introductory programming courses, we intend to conduct experiments in other learning settings where students are expected to read material on a computer screen or solve actual problems. To summarize, the analysis of these eye metrics using more fine-grained ROIs (lexer-based, line-based) as an extension of complexity-based ROIs provides instructors with deeper insights into the cognitive processes used by students when compared to the current state-of-the-art techniques.

Original languageEnglish (US)
Title of host publication2022 IEEE Frontiers in Education Conference, FIE 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781665462440
DOIs
StatePublished - 2022
Event2022 IEEE Frontiers in Education Conference, FIE 2022 - Uppsala, Sweden
Duration: Oct 8 2022Oct 11 2022

Publication series

NameProceedings - Frontiers in Education Conference, FIE
Volume2022-October
ISSN (Print)1539-4565

Conference

Conference2022 IEEE Frontiers in Education Conference, FIE 2022
Country/TerritorySweden
CityUppsala
Period10/8/2210/11/22

Keywords

  • Code Comprehension
  • Gaze Tracking
  • Knowledge Gain

ASJC Scopus subject areas

  • Software
  • Education
  • Computer Science Applications

Fingerprint

Dive into the research topics of 'Using AI-based NiCATS System to Evaluate Student Comprehension in Introductory Computer Programming Courses'. Together they form a unique fingerprint.

Cite this