_edited.jpg)
Where Assessment, Humans, and AI Converge
BroadMetrics integrates assessment, psychometrics, and artificial intelligence (AI) to assess humans and AI.
We help education, AI, and healthcare organizations with their assessment problems by employing modern psychometric methods. What is psychometrics? It is the science of measuring abilities, traits, and mental/cognitive processes in humans.
What sets us apart is that we are a combination of an assessment consultancy and an AI evaluation firm. That is, in addition to measuring constructs in humans, we use assessment and psychometric methods to evaluate AI. We design rating scales and human rating systems to measure constructs such as accuracy, safety, bias and more. We help you measure what matters - from scoring essays to evaluate writing ability to auditing AI to determine accuracy of a large language model (LLM).
How AI and Humans Work Together in Assessment
We may think of the combination of AI + Humans + Assessment as cyclical or symbiotic...
​
-
In the modernized assessment of individuals, we rely on AI to develop tasks and score them. We use AI to evaluate humans.
-
The AI behind those processes needs to be evaluated (by humans).
-
When we develop AI for any purpose (not necessarily assessment of humans) we rely on humans to evaluate the AI using different psychometric approaches. We assess AI.
All possible combinations exist. Humans evaluating humans, AI evaluating humans, humans evaluating AI, and AI evaluating AI.
Get to Know Us
BroadMetrics is a boutique consulting firm offering a broad range of analytic services. We specialize in applied assessment and psychometrics aimed at measuring both individuals and artificial intelligence.
With 20+ years of experience in the assessment and psychometrics industry, we assist in solving all types of assessment problems whether in education, healthcare, user experience surveys, people analytics, or sales. We also understand the unique capabilities psychometrics offers in the AI sciences. We provide services to ensure your AI is ethical, responsible, and accurate.
​
We are available to help as advisors or implementation partners. Reach out for a complimentary consultation call to discuss your needs.
Types of Services
From assistance with research and test design to AI strategy and grant proposals, we provide expert guidance and support every step of the way. We can maintain a long-term contributory role or train and mentor your team to take the reins.
Areas of Expertise
The field of psychometrics and measurement is evolving. We'll guide you through recent developments in the field and design and/or implement an assessment solution to suit your needs.​​ Our services include operational support, special studies, and research projects. Our solutions strongly adhere to industry standards for assessment as defined by the American Psychological Association, the American Educational Association, and the National Council for Measurement in Education.​
Let us assist you in creating and maintaining ethical AI systems using psychometric principles and methods. Our services in the applied AI sciences include statistical evaluation of AI using state-of-the-art metrics, expert rating systems to evaluate and assess AI for accuracy, safety, and more, and automated scoring methodologies and evaluation for assessments scoring humans or AI. Whether collecting data for training or for evaluation, our background in psychometrics ensures that the human data used for your AI will be of the highest quality.
We have a very unique expertise in the design, monitoring, and the evaluation of scoring mechanisms for open-ended tasks and likert scales. We develop systems using human raters, AI, or a combination of the two depending on the tasks and use context. Our services include the creation and review of likert scales and rubrics, development of training guides and human rater qualification instruments, ​real-time monitoring of the rater pool for consistency and accuracy, and development of automated scoring models with traditional NLP features or using generative AI.
Publications & Speaking Events
Studies & Papers
Our recent studies delve into cutting-edge AI topics, including generative AI for essay scoring.
Our latest publication explores the differences in validity argumentation for LLM-based AI scoring and feature-based AI scoring.
Conference Presentations
We recently gave paper presentations at the annual meeting of the National Council for Measurement in Education in Denver CO in April 2025.
​​
Paper 1: Evaluating Rationales: A Comparative Study of LLMs and Human Raters in Assessing Language Learners’ Essays
​​
Paper 2: Validity Evidence for Use and Interpretation of Scores from Generative AI
Workshops
Our next workshop is scheduled for October 27, 2025. Join us the Artificial Intelligence in Measurement and Education Conference (AIME-Con) in Pittsburgh, PA. What will be be talking about? Jodi will give a 2-hour workshop: Getting Started with LLM Evaluation: A Primer for Psychometricians
​​
Can't make it? Contact us to schedule a workshop for your team.