Skip to main content

Are you trying to trick me? Insights in LLMs based education

 LLTeacher allows creating homeworks and assignments that wouldn't be possible in classical education and classroom settings.

One of the most interesting kind of assignment it can create is the "tricky question" one.

The LLM is prompted to use a false or wrong information useful in the completion of the homework. The student is supposed to recognize that one of the information provided by the LLMs is wrong and call it out - completing the exercise correctly.

 This kind of exercise was tested in winter 2025 during a course of Statistical Computing, STAT302 at University of Washington (UW).

I am glad to report that the exercise succeeded in stimulating critical thinking in the students as well as in making sure that the students deeply understand the material.

We followed up interviewing one of the student that took the exercises.

The student reported that he tend to trust what the LLM answers. So at the beginning it was hard for him to recognicile the answer with his understanding of the subject.

 After a few rounds of interactions, the student decided to check online. At that point he was clear that his understanding was correct and that the information provided by the LLM was wrong.

He eventually asked to the LLM: were you program to say me the wrong answer?

At that point the LLM drop its charade and  admitted that it was indeed prompted to provide wrong information.

Today more than ever, we believe that it is fundamental for students to develop and use critical thinking. Not trusting blindly information even when they are from reputable sources. 

We are glad that LLTeacher helps in achieving this objective. 

 

Popular posts from this blog

How LLTeacher Works: From Assignment Design to Review

LLTeacher is designed to change the way students interact with AI during their coursework. Rather than using an AI to generate a quick answer, the platform creates a structured environment where the AI acts as a tutor.  The process follows a simple three-step cycle: the instructor creates the homework, the student works through the problem conversationally, and finally the instructor reviews the journey. Preparing the Lesson The process begins in the Instructor View , where the educator sets the foundation for the assignment. Instead of simply providing a question, the instructor uploads a problem statement and a detailed reference solution that serves as the "gold standard."  Crucially, the instructor also select a set of instructions, or a prompt, that defines the teaching style for that specific task.  This allows the instructor to direct the AI to: Act as a nudge for recall, asking students to remember concepts from a previous lecture, or  Act as a guide for disc...
Read more

Why We Developed LLteacher

Generative AI is reshaping education, and homework is at the center of this change. Tools like ChatGPT make it very tempting to bypass essential cognitive steps such as recalling concepts, re-elaborating ideas, and applying knowledge. These steps are not just formalities; they are the foundation of learning. When they disappear, so does much of the educational value of assignments. Some institutions have responded by banning AI, but this approach is difficult to enforce and ignores the reality that AI will be part of students’ future professional environments. Instead, we believe the challenge is to integrate AI thoughtfully, in ways that preserve learning objectives, enhance pedagogical tools, and promote equity. LLteacher is our attempt to do this. It is an open-source tool designed for homework assignments in statistics courses. Rather than providing direct answers, LLteacher guides students through problem-solving, encouraging active engagement and critical thinking. Instructors re...
Read more

Preventing AI use is not the answer

When ChatGPT first arrived, it fundamentally changed the landscape of how our students complete their homework. Traditionally, homework was a space where students engaged in their learning process: they had to recall information, re-elaborate concepts, and apply their knowledge independently. Today, that process is at risk.   We’ve seen firsthand how the widespread availability of LLMs has disrupted this cycle. Students can now bypass the entire learning process by asking an AI for direct solutions.  This isn’t just about "cheating", it’s a qualitatively different challenge.  AI is easily accessible, harder to detect, and increasingly socially acceptable among students.  Most critically, when students use AI this way, they miss out on the essential cognitive processes that are central to genuine learning.  While some instructors have moved to ban these tools, we believe that is not the right path for two major reasons:   Enforcement is nearly impo...
Read more