Researchers link longer AI feedback to weaker next attempts
A study of AI hints in an online web development course found longer messages went with fewer correct next submissions. The authors say this does not prove the hints caused it.
Six computing education researchers posted a study to arXiv on 18 October 2026 on AI feedback for students in a web development course.
Three of the authors work at Aalto University in Finland and three at universities in the United States. The paper covers an online introductory course from September 2025 to June 2026. After a wrong submission, a large language model, the kind of AI behind chatbots, wrote a hint for the student. The course used o3-mini, then GPT-5-mini, then GPT-5.5.
These comparisons do not identify the causal effect of feedback.
The authors said feedback went with better next attempts in several comparisons, but the pattern varied. Longer messages went with fewer correct next submissions, more attempts and longer gaps before students tried again. Students who had feedback earlier in a course part were less likely to solve the related final-project step. From January to May 2026 the course had 525 students who made 83,374 submissions.
For teachers who add AI hints to programming exercises, the authors suggest testing two changes. One is a limit on the length of each message. The other is letting students choose between a short hint and a fuller explanation. They said both ideas need proper trials.
The paper is listed for 6 November 2026 at Koli Calling, a computing education conference in Finland, the conference schedule shows. The study measured later performance, not learning itself. It is not known whether shorter hints would help students learn more.