AQP Evaluator Calibration
- University of Central Florida
- Instructional Design
- Instructional Design 101
- AQP Evaluator Calibration
Last Modified: 2011-06-21
- Inter Rater Reliability (IRR): Do raters rate consistently with one another?
- Referent Rater Reliability (RRR) Do raters reate consistently with ("gold standard") some standard (e.g., industry standard).
- Leniency/Severity (Grading everyone as good or poor)
- Central Tendency (Grading everyone as average)
- Halo effect (When we consider a person good (or bad) in one category, we are likely to make a similar evaluation in other categories.(pigeon holding) ,e.g., judging a good looking person as more intelligent.)
- Contrast effect (Norm-reference grading)
- Alternate test
- Scoring agreement (inter rater, percentage agreement)
- Internal consistency (Split half, Kuder-Richarson, Cronbach Alpha)
- Content (Face)
- Construct (Discriminant, Convergent)
- Criterion (Predictive, Concurrent)
- Agreement (Inter rater agreement)
- Systematic differences (how similar are average scores? Ex. R1 rates person low, R2 rates high, and their average cancel out each others ratings)
- Congruency (how many 1,2,3,4,'s compared to the group
- Consistency (Do I/E's shift up and down together?)
- Sensitivity (How well doe I/E respond to small changes)
- Identifies problems early
- Delivers objective data
- Identifies trends and causes
- Facilities confirmation of problem
Words From Our Students
"StudyBlue is great for studying. I love the study guides, flashcards, and quizzes. So extremely helpful for all of my classes!"
Alice, Arizona State University
"I'm a student using StudyBlue, and I can 100% say that it helps me so much. Study materials for almost every subject in school are available in StudyBlue. It is so helpful for my education!"
Tim, University of Florida
"StudyBlue provides way more features than other studying apps, and thus allows me to learn very quickly! I actually feel much more comfortable taking my exams after I study with this app. It's amazing!"
Jennifer, Rutgers University
"I love flashcards but carrying around physical flashcards is cumbersome and simply outdated. StudyBlue is exactly what I was looking for!"