_edited.jpg)
Psychometric and evaluation support and advising, for all of your assessment needs.
to Measure Humans and Machines
principled
assessment
BroadMetrics integrates assessment, psychometrics, and artificial intelligence (AI) to assess humans and AI.
We help education, AI, and healthcare organizations with their assessment problems by employing modern psychometric methods. What is psychometrics? It is the science of measuring abilities, traits, and mental/cognitive processes in humans.
What sets us apart is that we are a combination of an assessment consultancy and an AI evaluation firm. That is, in addition to measuring constructs in humans, we use assessment and psychometric methods to evaluate AI. We design rating scales and human rating systems to measure constructs such as accuracy, safety, bias and more. We help you measure what matters - from scoring essays to evaluate writing ability to auditing AI to determine accuracy of a large language model (LLM).
Get to Know Us
BroadMetrics is a boutique consulting firm offering a broad range of analytic services. We specialize in applied assessment and psychometrics aimed at measuring both individuals and artificial intelligence.
With 20+ years of experience in the assessment and psychometrics industry, we assist in solving all types of assessment problems whether in education, healthcare, user experience surveys, people analytics, or sales. We also understand the unique capabilities psychometrics offers in the AI sciences. We provide services to ensure your AI is ethical, responsible, and accurate.
​
We are available to help as advisors or implementation partners. Reach out for a complimentary consultation call to discuss your needs.
What We Do
Psychometrics &
Assessment
The field of psychometrics and measurement is evolving. We'll guide you through recent developments in the field and design and/or implement an assessment solution to suit your needs.​​ Our services include operational support, special studies, and research projects. Our solutions strongly adhere to industry standards for assessment as defined by the American Psychological Association, the American Educational Association, and the National Council for Measurement in Education.​
AI Evaluation & Applied AI Sciences
Let us assist you in creating and maintaining ethical AI systems using psychometric principles and methods. Our services in the applied AI sciences include statistical evaluation of AI using state-of-the-art metrics, expert rating systems to evaluate and assess AI for accuracy, safety, and more, and automated scoring methodologies and evaluation for assessments scoring humans or AI. Whether collecting data for training or for evaluation, our background in psychometrics ensures that the human data used for your AI will be of the highest quality.
Rating Scales &
Scoring
We have a very unique expertise in the design, monitoring, and the evaluation of scoring mechanisms for open-ended tasks and likert scales. We develop systems using human raters, AI, or a combination of the two depending on the tasks and use context. Our services include the creation and review of Likert scales and rubrics, development of training guides and human rater qualification instruments, ​real-time monitoring of the rater pool, and development of automated scoring models.
Types of Services
From assistance with research and test design to AI strategy and grant proposals, we provide expert guidance and support every step of the way. We can maintain a long-term contributory role or train and mentor your team to take the reins.
Publications & Speaking Events
Studies & Papers
Our recent studies delve into cutting-edge AI topics. Some recent papers include:
​
-
Psychometrics is all you need is a position paper that details our approach to AI evaluation. It's not just about identifying errors and quantifying trustworthiness. Learn how to apply evidence-centered design and collect validity evidence for your LLM application. Read it here!
-
Another paper explores the differences in validity argumentation for LLM-based AI scoring and feature-based AI scoring.
​
Both papers are under peer review and coming to a journal near you!
Conference Presentations
Can AI-generated rationales provide evidence that AI scores are valid?
If you're curious about this question, contact us for more information on the paper presentation ​​given at the Artificial Intelligence in Measurement and Education Conference (AIME-Con) in Pittsburgh, PA in October!
​
Workshops
We recently gave a workshop at the Artificial Intelligence in Measurement and Education Conference (AIME-Con) in Pittsburgh, PA. What did we talk about? Jodi gave a 2-hour workshop: Getting Started with LLM Evaluation: A Primer for Psychometricians
​​
Couldn't make it? Contact us to schedule a workshop for your team.