AI for Informal Student Feedback

Using AI tools to improve student learning

Artificial intelligence (AI) can improve learning when used appropriately, but we are just in the infancy of realizing this potential. AI tools can transform how instructors engage with students, but it requires changes in how we approach designing, delivering, and assessing learning. AI's potential as a tireless teaching assistant is vast, but its limitations necessitate a thoughtful approach to its integration. They can serve as valuable teaching assistants by generating helpful feedback. This post provides an example of using AI to give students helpful, informal feedback while maintaining transparency, ethical standards, and a focus on student learning.

AI's Limitations in Grading: Consistency Matters

For instructors, the allure of using AI for grading is undeniable—promising future efficiency, flexibility, and objectivity. However, the current generation of generative AI models, such as ChatGPT or Claude, are not sufficiently consistent for assessing student work. AI models rely on probabilistic algorithms that produce outputs based on patterns in training data and randomly change each response subtly. This approach often results in inconsistencies and hallucinations (fabricated information), making AI unsuitable for assessments. Slight variations in prompts or input details can lead to drastically different grades for the same assignment, making AI tools unsuitable for fair and reliable assessments.

AI as a Teaching Assistant: Practical Applications

While grading is off the table, AI can excel as a teaching assistant by providing formative feedback. Here are some practical ways to integrate AI into your teaching workflow:

• Personalized Feedback: AI tools can analyze student submissions and offer tailored suggestions for improvement. For instance, they might highlight areas where arguments lack evidence or writing could be more concise.

• Prompt Coaching: By evaluating the quality of prompts students use to interact with AI tools, instructors can guide them toward better practices in framing questions—a skill increasingly valuable in the workplace.

• Classroom Engagement: AI tools can quickly summarize student submissions before a class discussion, allowing instructors to identify common themes and misunderstandings and invite students to share their ideas. 

Transparency and Ethical Considerations

Transparency is crucial. Students must understand how and why these tools are employed to build trust and encourage ethical use. Some best practices:

1. Be Open About AI Use: Communicate to students that AI is used for informal feedback but not for formal grading. Explain the reasons behind this distinction to avoid misconceptions.

2. Anonymize Data: To protect privacy, remove all personally identifiable information when inputting student work into AI systems.

3. Encourage Student Exploration: Motivate students to use AI tools for self-assessment and learning, empowering them to develop skills they'll need in their careers.

Below is an example of how I recently used AI tools to generate personalized, individual feedback for undergraduate marketing students on an AI prompt they submitted as part of an assignment.

Step-by-Step Approach to Using AI for Feedback: A Detailed Guide

Integrating AI into the classroom requires a thoughtful, structured approach to ensure it enhances learning without compromising standards or student engagement.

1. Introduce Students to AI

Early in the semester, I dedicated a class session in this course to a hands-on workshop exploring the fundamentals of generative AI, including its strengths, limitations, and good practices for developing prompts that will get the most out of AI tools. This session sets the stage for students to understand how AI can support their learning while emphasizing the importance of critical evaluation.

The workshop included content such as the concepts summarized in my post “Well-Crafted Prompts Make AI Better” with this framework:

• Goal: a clear objective of what you want the AI to accomplish

• Instructions: Detailed instructions setting context, clarifying expectations, and specifying any limitations

• Response Format: specifying formatting such as bullet points, tables, or paragraphs of text

• Tags: Tags such as <GUIDELINES> in the example below act as variables within the prompt, referencing specific content without ambiguity

You can download the PowerPoint from the workshop for this class here: AI Tools Workshop

2. Integrate AI Into Assignments

I have modified many of my exercises to prompt students to use an AI tool of their choice in completing portions of the exercise. This introduces them to practical applications of AI and encourages critical thinking about its outputs.

Example:
In a recent assignment, I asked students to use "good prompting practices to ask an AI tool of your choice to forecast three likely benefits Walmart might gain by enhancing its omnichannel capabilities over the next year." The exercise required students to submit:

• The exact prompt they used.

• A 2–3 sentence summary of the AI's response in their own words.

• A short critical evaluation of one potential limitation in the AI’s response.

• How useful they found the AI tool

Overall feedback was positive, with nearly two-thirds of students rating their use of the AI tool as “Very useful”:

3. Evaluate Prompts Using AI

Some of the student prompts were quite good, while others lacked some of the best practices or were just far too general, along the lines of:

What are three likely benefits Walmart might achieve by enhancing its omnichannel capabilities over the next year?

Other student prompts were stronger but could still be better, such as:

Forecast three likely benefits Walmart might gain by enhancing its omnichannel capabilities over the next year. Use simple language and the articles I provided along with reliable articles from the last 3 years. Include descriptive examples of each benefit. In your response include a concise explanation of the three benefits with clear description of how each relate to the benefits of using an omnichannel strategy.

Providing informal, individual feedback on AI prompts is a task where AI tools can excel. In this case, I used Perplexity.ai with four different AI models: DeepSeek R1, OpenAI O3 Mini, OpenAI GPT-4o, and Claude Sonnet 3.5, allowing me to compare model performance and identify which tools provide the most actionable insights.

Example:

• I created a CSV format file with a random student ID code to anonymize student identification and the prompts submitted.

• With the prompt, I included a rubric for evaluating the student prompts, <GUIDELINES>, which I pasted at the bottom of the prompt (not shown) drawn from the AI workshop materials the students saw previously.

• This prompt was used with each of the four AI models to generate a CSV file output with the student ID and the AI-generated feedback.

Evaluate the prompts in the second column of the attached CSV file, written by undergraduate business students who attended a workshop on AI prompting best practices in the context of the assignment instructions: "Use good prompting practices to ask an AI tool to forecast three likely benefits Walmart might gain by enhancing its omnichannel capabilities over the next year." 

Write a paragraph of feedback for each student, recommending improvements to prompt using best practices in AI prompting and my <GUIDELINES> below. 

Output a CSV with student IDs and feedback. Then, list suggestions to enhance student feedback and propose an improved version of this prompt based on your recommendations.

In subjectively evaluating the prompt feedback each of these AI models generated for the students, OpenAI O3 Mini was the clear winner.

1. OpenAI O3 Mini: most detailed and consistent feedback, shared this with students

2. OpenAI GPT-4o: student feedback was nearly as detailed as OpenAI O3 Mini, and I would have used response if I didn't have access to O3 Mini

3. Claude 3.5 Sonnet: results were than DeepSeek, but not as detailed as the OpenAI models

4. DeepSeek R1: results were vague, lacking in detail, and did not follow my best practices in prompting even after three follow-up prompts

I was surprised by DeepSeek R1's poor performance, given that I have found it to be the strongest of the four at summarizing documents. When I first tried DeepSeek R1 on Perplexity, it only generated feedback for two out of nineteen students due to vague instructions in my initial prompt. Note that I was using an instance of DeepSeek R1 on Perplexity.ai, so all of my data remained on US servers.

Example feedback for a student:

• "Your prompt lacks sufficient detail to guide an AI tool effectively toward actionable insights. Adding specific instructions about data sources, analytical frameworks, and output format would improve it significantly. Suggested revision: 'What are three likely benefits Walmart might achieve within the next year through enhanced omnichannel capabilities? Please use reliable business journals published after January 2023 as references; apply SWOT analysis to identify opportunities for growth; present findings as bullet points (50-75 words per point), supported by at least one statistic or example per point."

The AI tool generated feedback for each student customized for the quality of their prompt. Feedback ranged from the prompt being “highly detailed with excellent structure but could be further enhanced by…” to “includes useful source criteria but could be improved by…”

4. Deliver Feedback Transparently

In sharing AI-generated feedback with students, I make clear that AI is involved. This is important for transparency and as an example that encourages students to seek feedback from AI tools.

The anonymized IDs allowed me to re-associate the AI-generated prompt feedback with student names and emails. I then used Microsoft Word mail merge to send personalized emails to each student. I could have also just copied and pasted each prompt into assignment feedback. Note I did not use the AI feedback to score the student assignment.

Integrating AI Into Assignments Can Enhance Learning

By thoughtfully integrating AI into assignments, instructors can unlock their potential as powerful teaching assistants while maintaining transparency and ethical standards. Although AI tools are not yet reliable for grading, they excel at providing personalized, formative feedback that enhances student learning. The process fosters skill development and engagement, from introducing students to AI's capabilities and limitations in workshops to designing assignments that encourage critical thinking about AI outputs and leveraging multiple models to evaluate and refine student prompts. Experiment with your examples, adapting these strategies to your field and teaching contexts.

Please share your examples with me at ben.bentzin@mccombs.utexas.edu!