Using automated essay scoring and feedback systems for the development of ethical reasoning

Principal Investigator: Andres Luco, Assistant Professor, School of Humanities and Social Sciences
Co-PI: Kevin Hartman, Research Scientist, Centre for Research and Development in Learning (CRADLE@NTU)
Research Associates: Lee Vwen Yen Alwyn, CRADLE@NTU
Funding Agency: CRADLE Start-up Grant

As NTU continues its push to move more course content online so its students can access it anytime from anywhere, the freed up face-to-face time has been used by instructors to focus on meaningful interactions in the classroom rather than giving lectures and reviewing readings. While instructors are delivering more materials online, their students are submitting more of their assignments digitally. From quizzes, to problem sets, to reflective essays, NTU’s learning solutions process and archive students’ course-related activities. What if these archives could be used to provide students targeted and timely feedback about the competencies needed for life-long success?
Assistant Professor Andres Luco of the School of Humanities and Social Sciences has taken the challenge of answering this question within the context of his online course: “Ethical Reasoning and Moral Dilemmas.” According to NTU’s general education requirements for undergraduate education, every undergraduate should have at least a basic understanding of ethical reasoning and moral philosophies. As NTU annually takes in more than 6000 new undergraduates, that yields a lot of students engaging in activities related to ethical reasoning, Prof Luco’s areas of expertise. It also yields approximately 3000 ethics-related essays per semester. 
After his first semester running the course and reading his students’ essays, Prof Luco noticed that although every essay was unique, they could be clustered into common categories. Some essays contained too many grammatical errors to fairly assess their content. Other essays, although grammatically flawless, exhibited logical errors. A number of essays skirted around the ethical dilemmas at the heart of the assignments. Prof Luco sought a way to cluster the essays with the type of personalised feedback he would give them if he had the resources. When no ready-made solution fit to his requirements, he sought to develop a solution of his own.
At the time, the Centre for Research and Development in Learning (CRADLE@NTU) researchers were already experimenting with automatic scoring and feedback systems to improve formative assessment at NTU. However, they needed a real use case to test the systems’ practicality. Together, the team designed a research plan spanning four iterations of the course cycle, starting with identifying categories of student responses and feedback messages informed by research, and finally testing it in a live course.
As the system develops, Prof Luco intends to scale it to similar courses that wish to use essay writing as a form of formative feedback to their students.