Redesigning an Assessment Using the Student AI Literacy Competency Framework

Context

For a session at Summer Institute at Southwest Tennessee Community College, the Course Design Support Coordinator and Director of Assessment collaborated with the TBR AI Learning Collab Team to facilitate a session on redesigning an assessment using the Student AI Literacy Competency Framework. An intentionally selected course assessment served as the redesign component for participants.

The assessment asked students to critically evaluate AI-generated academic writing. Since many students often accept AI outputs at face value without recognizing limitations such as incomplete information, vague language, or hallucinated content, this activity was designed to address this gap by guiding students to compare an AI-generated abstract with an instructor-provided model abstract for the same research article.

Application

As a low-stakes, scaffolding micro-activity, AI was used to generate a sample abstract based on a scholarly source. A common practice among students is to ask AI to write an abstract for an article by providing the source link or the name of the article. The generated abstract intentionally included common issues such as inaccuracies in the methods section, vague information, and generalized claims.

Students, represented by the workshop participants during the session, engaged in a short micro-activity in which they used a checklist to evaluate the AI-generated abstract by comparing it to the instructor-provided model abstract. They identified inaccuracies, missing elements, and vague language, then reflected on the strengths, weaknesses, and necessary revisions during group discussions.

Outcomes (Student AI Literacy Competency Framework)

• Describe common limitations of AI systems, such as incomplete information and hallucinated content.
• Compare tasks completed by humans with those supported by AI systems.
• Evaluate AI-generated content using an instructor-provided sample.

The activity supported the development of transferable academic skills that are commonly expected of students across a variety of disciplines, including summarizing and analyzing scholarly articles and evaluating the credibility of information sources.