Inter Rater Reliability (IRR): Do raters rate consistently with one another?
Referent Rater Reliability (RRR) Do raters reate consistently with ("gold standard") some standard (e.g., industry standard).
Instructors/Evaluators are subject to standard judgment errors. name these.
Leniency/Severity (Grading everyone as good or poor)
Central Tendency (Grading everyone as average)
Halo effect (When we consider a person good (or bad) in one category, we are likely to make a similar evaluation in other categories.(pigeon holding) ,e.g., judging a good looking person as more intelligent.)
Contrast effect (Norm-reference grading)
List the Methods for Demonstrating Reliability & Validity
Systematic differences (how similar are average scores? Ex. R1 rates person low, R2 rates high, and their average cancel out each others ratings)
Congruency (how many 1,2,3,4,'s compared to the group
Consistency (Do I/E's shift up and down together?)
Sensitivity (How well doe I/E respond to small changes)
Second Generation Measures: History shows that _____ (Measured using) &____(Measured using) are the most salient measures for US air carrier I/Es.
Second Generation Measures: History shows that Sensitivity (measured using IRR & RRR) & Accuracy (measured using MAD Mean Average Deviation) are the most salient measures for US air carrier I/Es.
Flight Operations Quality Assurance.
is a method of capturing and analyzing the data generated by an aircraft moving through the air from one point to another. Applying the information learned from this analysis helps to find new ways to improve flight safety and increase overall operational efficiency.
The goal is to improve the organization or unit's overall safety, increase maintenance effectiveness and reduce operational costs.