Study finds AI marking misses student misconceptions in computing
Three AI models marked 3,041 university computer science answers. Their scores were broadly in line with a teacher's, but their feedback rarely pointed out what students had misunderstood.
Researchers at Columbia University and New York University posted a study on 7 October 2026 showing AI marking rarely spots students' misconceptions.
Xi Zhao, Xinyue Jiao and Zhen Xu wrote the paper. It uses 3,041 answers to 50 open questions from computer science courses at the Universidad Francisco de Vitoria in Spain in 2025 to 2026. A teacher marked each answer out of 10. So did three large language models (LLMs), the AI models behind chatbots: DeepSeek, Qwen and Gemini.
The models' scores agreed moderately to strongly with the teacher's, the full text says. Their written feedback differed more. Teachers flagged a misconception, an error in understanding, in 15.0% of comments and the models in 4.8% to 7.4%. For answers the teacher scored 0 to 2, the figures were 31.5% and 5% to 7%.
For lecturers and teachers who use AI to mark, the authors suggest it may suit procedural topics such as software testing. They say topics that need judgement should still be marked by people. In England, Department for Education guidance already says "Teachers, leaders and staff must use their professional judgement when using these tools."
The paper's header says it appears in the proceedings of the 34th International Conference on Computers in Education. The authors say the data comes from one institution and one software engineering course. They did not test newer models, so it is not known whether those give better feedback.