James Prather, Brent Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S. Randrianasolo, Brett Becker, Bailey Kimmel, Jared Wright, Ben Briggs

Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Previous research has shown that novices can encounter multiple metacognitive difficulties while programming. Novices are typically unaware of how these difficulties are hindering their progress.

Meanwhile, many novices are now programming with generative AI (GenAI), which can provide complete solutions to most introductory programming problems, code suggestions, hints for next steps when stuck, and explain cryptic error messages. Its impact on novice metacognition has only started to be explored.

Here we replicate a previous study that examined novice programming problem solving behavior and extend it by incorporating GenAI tools. Through 21 lab sessions consisting of participant observation, interview, and eye tracking, we explore how novices are coding with GenAI tools.

Although 20 of 21 students completed the assigned programming problem, our findings show an unfortunate divide in the use of GenAI tools between students who accelerated and students who struggled.

Students who accelerated were able to use GenAI to create code they already intended to make and were able to ignore unhelpful or incorrect inline code suggestions.

But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties.

Furthermore, struggling students often expressed cognitive dissonance about their problem solving ability, thought they performed better than they did, and finished with an illusion of competence. Based on our observations from both groups, we propose ways to scaffold the novice GenAI experience and make suggestions for future work.

Abstract

Generative Artificial Intelligence (GenAI) tools (e.g., ChatGPT, Calude) have rapidly become integral to software development.

These tools are especially attractive to students, as they can reduce cognitive load. However, their adoption also introduces a socio-cognitive risk: the accumulation of Comprehension Debt (CD).

CD refers to the growing gap between what a development team knows about its codebase and what it actually needs to understand in order to maintain and modify it effectively. This qualitative study investigate how GenAI tools contribute to CD in the context of an undergraduate software engineering project. Our study is based on 621 reflective diaries from 207 students over eight weeks. We identify four CD accumulation patterns and one mitigating pattern in students’ use of GenAI tools.

The four accumulation patterns include:

• (1) AI-as-black-box code acceptance,
• (2) context-mismatch debt,
• (3) dependency-induced atrophy, and
• (4) verification-bypass.

In contrast, the mitigating pattern involves students using GenAI as a comprehension scaffold, allowing them to build a deeper understanding of the code.

We argue that CD is distinct from traditional technical debt because it resides in the collective cognition of development teams rather than in the codebase itself. Our findings highlight the need for explicit pedagogical strategies to mitigate CD in software engineering education, emphasizing verification practices, structured retrospectives, and active learning assessments.

Comprehension Debt, Generative AI, Software Engineering Education, Agile, Cognitive Load, Technical Debt

Conclusion

This paper examined how GenAI tools influence understanding in student software engineering projects. We introduced CD as a socio-cognitive construct describing gap between codebase demands and collective team understanding. We identified four CD accumulating patterns (black-box acceptance, context-mismatch, dependency atrophy, and verification bypass) and one CD mitigating pattern (AI as comprehension scaffold). We also articulate conceptual model of linking epistemic orientation, germane cognitive load investment, and CD accumulation.

GenAI tools do not inherently undermine or enhance learning. Our study shows that they act as amplifiers of students existing orientation toward acceleration or exploration.

Additionally, shows that verification competence must be intentionally cultivated through courses or module design. Otherwise students may find themselves in competence trap, where they lack domain knowledge required to safely use the tools they rely on for code generation.

By integrating structured retrospectives, active learning assessments, and cognitive apprenticeship models, educators can promote comprehension oriented.

CD offers a vital lens to ensure that next generation of software engineers possesses not just speed to generate code, but the depth to sustain it.