Analyzing course discussions using AI
I am teaching a course on data storytelling in a strategic brand communication degree program this summer, and I have integrated Chatbot support into almost every assignment, while coaching students on how to use these tools for brainstorming, help learning tools such as Tableau, and as a partner in discussions during their project.
I often share examples of how I use AI in analyzing their discussion posts for summarizing and pattern recognition and directly for activities in the course. I recently shared an analysis of their end of semester reflections and sent them a note reflecting on that analysis. I think it will be of value to all the readers here, so replicating it here with minor edits.
“I decided to try an AI tool to find patterns in the course reflection forum, and try some visualizations as well. I have a setup where a desktop chat tool can 'talk' to canvas, and find information about specific assignments and pull information and make it available to an AI tool for analysis and chat. This is a common capability, so it does not matter which tool I use. Rather it matters that I know this is possible, and then I can spend effort to get it working. But the output needs to be critically evaluated, so I'll use this use case to show what is possible, and what the issues are.
This is similar to what I mentioned a few times in the live sessions, that you don’t need to worry about perfecting your Tableau skills. It is more important to know what kind of problems can be solved, and the general frameworks/techniques that can be applied, and then pick an appropriate problem, and then learn the tool.
AI can help you explore more problems, ideate more on solutions to a selected problem, and learn how to use tools to solve the problem. However, it cant find a problem for you, or collect the context (Prep/Talk process) , and cant really tell you if the solution is any good(in most domains), unless you actually apply it(which can get expensive, and that is why you need judgement/taste which comes with experience).
In this case, I used AI to help summarize 23 discussion posts, with a view to find patterns(which AI is good at). It is also good at generating code and creating artifacts to display information, create visuals, and make it all interactive. Take a look at https://claude.ai/public/artifacts/ec427353-3ae5-46c6-b451-56a82e6651b2 . I don’t share real student names with AI tools, so you will see cryptic identifiers, but you might find yourself in the summaries.
I spent about an hour on fine tuning this output, deleting some useless analysis/visualizations, changing some words, but the AI was a tool, being directed based on my judgement/taste which comes from experience and understanding of the context( what the input text meant, what the assignment was, how long was the class, what we studied etc). The AI can be given all that context, but there are limitations and it could do a lot better with more thought on what context we provide it. This practice now has a name, it is called Context Engineering, and is relevant for all of us to learn, beyond the traditional 'prompt engineering'. (If you want to learn more, feed the aforementioned terms into, guess what..... a chatbot)
Now to critique the artifact. It has use in understanding the breadth of experiences you all had, and your 'sentiment' as extracted and visualized by the language model operating the chatbot. However, the labels it provides for the individual summary (Joy, Trust, Anticipation, Surprise) were auto generated, the percentages it allocated are based on an algorithm that is not clear so I have no way of checking its consistency.
Lastly, while interpreting this, I could feel all warm and fuzzy inside that average sentiment is 0.85 , with 20 out of 23 responses being highly positive. The reality is that you wont dunk on the course or me in a public forum, which is not anonymous, especially when grades are pending :)”
I have done similar exercises, trying to tie the analysis to the course topic, and in this case, we were focused on storytelling with data visualizations, and this output was useful to show the power of AI tools, while coaching them to be skeptical of the outputs.
In case you are wondering, my current tool of choice is Claude Code and Claude Desktop, though I give Gemini and ChatGPT some opportunity to impress me as well. I am working up to Gemini 2.5 Pro, and with the freebies they are doling out to students, it becomes a useful tool to try in class.