Getting Started with LLM Evaluation: A Primer for Psychometricians.

A 2-hour workshop to be presented by Jodi Casabianca.

Abstract: This session introduces psychometricians and assessment scientists to evaluation methods for large language model (LLM) applications in education. Through a combination of lecture and hands-on activities, participants will explore key evaluation techniques—including error analysis, human review, and the use of LLMs as evaluators ("LLM-as-a-judge"). The session will emphasize how psychometric principles can strengthen the rigor, validity, and interpretability of LLM application pipelines.