_edited.jpg)
Where Assessment, Humans, and AI Converge
PSYCHOMetrics
Psychometrics & Assessment
Developing an assessment? Need to score an existing assessment?
Psychometric analyses encompass a variety of methods aimed at measuring psychological attributes such as intelligence, personality traits, and attitudes. You might be interested in user experience and creating surveys. You might be evaluating students' learning within an online tutoring system. You might be evaluating patient reported outcomes. You might be evaluating people skills and employer satisfaction. We can help with all of it!
There are three broad approaches to psychometrics. The traditional era introduced classical test theory, which is still in use today. Another era came with the introduction of latent variable modeling and item response theory, which is more statistically sophisticated and generally requires more data. We recently have ventured into the modern era which relies heavily on AI and relatively more computational solutions. Which approach is best? It really depends. While we start with modern solutions, we might suggest a mixture of these approaches depending on the specific context, data limitations, and goals. No matter the approach, we are dedicated to ensuring that industry standards for validity, reliability, and fairness are met.
​
Our services range from item analysis, to test construction and validation, to comprehensive psychometric reviews and audits, and everything in between. We also excel at technical communication and can assist with technical reports, presentations, manuscripts, RFPs/proposals​.
Task Design
Item design, rubrics, piloting, and item analysis in principled design approach​
Equity & Fairness
Fairness reviews, subgroup analyses, differential item functioning
Scoring System Design
Design of assessment scoring systems using item response theory, classical test theory, or a hybrid
AI in Assessment
AI-driven item development, scoring, and analysis​​​​
Psychometric Research
Conduct special studies for making principled psychometric decisions​
Technical Communication
Technical advisory committee meetings, presentations, memoranda, reports and more
Test Construction
Factor analyses, item response theory modeling, validity studies
Scaling & Linking
Design equating studies and develop a plan for linking tests
​​​
Adaptive Testing
Computerized adaptive testing or multi-stage testing, including design and monitoring
​
Psychometric Research
AI-driven item development, scoring, and analysis
Assessment Audits
Full evaluation of assessment system to verify adherence to industry standards​
Example Case Studies
Example Use Cases
1
Fairness Evaluation for a New Assessment
Suppose you're a state department of education and you want to ensure fairness and equity in the development of a new assessment. Consult with us to determine how to approach bias training, item and task fairness reviews, and piloting items. We can also discuss personalization and culturally responsive assessment.
​
We will develop an analysis plan for fairness analyses, including differential item functioning (DIF) analyses and other methods depending on the items and test structure. We can instruct your team or conduct the analyses.
2
Collaborate on Technical Aspects of Education Research
Suppose you are a researcher with a large grant to conduct research on an educational intervention that uses assessment scores as an outcome variable. Consult with us to develop and/or implement a scoring plan using item response theory (IRT). We will first take steps to understand the assessment and your goals, and then propose a plan that will produce scores that reflect the construct under measurement. Depending on the study we might propose an equating plan. We serve as key personnel or consultant, depending on your needs and the grant program.
3
Refine an Existing Survey to Measure Employee Satisfaction
Suppose you're a human resources specialist who has an existing survey instrument to measure employee satisfaction. Consult with us to refine the scale by conducting a qualitative review of the survey content, conducting literature reviews to establish the state-of-the-art in measuring the construct, propose refinements to the scale, and collect data to assess the scale.
We offer research-backed suggestions and also assist with the technical communication so that your stakeholders are confident in implemented changes to the survey.
​
4
Present at a Technical Advisory Committee Meeting
Suppose you're an assessment contractor or vendor who is required to present at an upcoming technical advisory committee meeting on a very technical topic. You don't have time to prepare a polished slide deck or the comfort level to deliver the message.
Consult with us on your presentation materials or have us deliver the message. We are excellent technical communicators, possibly even "storytellers" and will be able to break down the information into a digestible format for your audience.
5
Analyze Data from Digital Learning Platform
Suppose you are a learning scientist or owner of an intelligent tutoring system. You recently made some changes and need help with a special study to understand some aspects of the data in the system and with deriving an assessment approach using the data. Consult with us to develop a plan using a variety of techniques including latent variable modeling and machine learning models.
We can serve as a consultant or we can implement the plan. We can also partner with you write up the study.​