Beginning with the newly validated Physical Therapy and Occupational Therapy assessments, the assessment results page will provide additional information for an employer to make an informed decision regarding the applicants assessment score.
Angoff Cutoff Score
One of the most commonly used approaches to establishing a cutoff score is the method detailed by Angoff (1971)1. This technique is particularly attractive in applied personnel selection settings because it is easily understood by subject matter experts (SMEs), it has high face validity to job applicants, the courts regularly endorse the technique, and it stands up to academic scrutiny.2
Although there are many variants, the classic Angoff method for setting cutoff scores
- Select and train SMEs who are familiar with the job and have them review each test item;
- Obtain ratings from SMEs on the percent of minimally qualified applicants who would be expected to answer each item correctly;
- Analyze SME ratings to obtain an unmodified Angoff cutoff score;
- Administer the test, and
- Facilitate a discussion with SMEs on the impact/validity of the unmodified cutoff score.3
With each new NurseTesting clinical assessment, the initial Unmodified Angoff Cutoff Score will be displayed next to the applicants overall score. Based on the applicants overall score, the system will automatically compare this score to the Unmodified Angoff Cutoff Score and display either PASS or FAIL on the results page. This enables the recruiter the ability to quickly determine if the applicant passed that particular assessment.
It is important to note, that each assessment will have its own, unique Angoff Cutoff Score. NurseTesting does not recommend the use of Arbitrary Cutoffs such as the 80% rule. Arbitrary cutoffs do not make academic or practical sense.
Furthermore, they can incense applicants who may realize that a meaningless standard in the selection process was used to make very meaningful decisions about their lives and careers.2 As recently as this past April (2009), the United States Supreme Court heard the case Ricci v. DeStefano (07-1428 & 08-328) in which the use of arbitrary cutoffs was an issue brought before the Court.
Modified Angoff Cutoff Score
Initially, when an assessment is administered, the Unmodified Angoff Cutoff Score is used until a sufficient sample size of assessment data is available for analysis. Once NurseTesting has obtained a sufficient sample size for each assessment, a comprehensive analysis will be conducted to determine if a modified Angoff Cutoff Score is necessary. When it comes to best practices for developing cutoffs, there is perhaps none better than the modified Angoff method.2
The cutoff score will be refined / modified based on the data collected during the initial launch of the assessment to account for assessment unreliability and actual test-taker performance. NurseTesting, using the Test Validation and Analysis Program (TVAP), will refine the assessment and identify test items that might be:
- Too difficult
- Too easy
- Negative or low Point Biserials – showing how correlated the item is to the overall test score.
- Cronbachs Alpha review once flawed items are removed
- Differential Item Functioning test items function in different ways for different groups of test-takers
Once the modified Angoff Cutoff Score is calculated using the Conditional Standard Error of Measurement (CSEM), the assessment results page will display the modified Angoff Cutoff Score instead of the initial unmodified Angoff Cutoff Score. Based on statistical analysis, including an Adverse Impact analysis, the recommended modified Angoff Cutoff Score will be used for all future administered assessments. As stated before, each assessment will have its own, unique modified Angoff Cutoff Score.
NurseTesting implemented these processes to continue providing validated content for new exams that meet the Uniform Guidelines for Employee Selection Procedures.
2Biddle, D. (2005). Adverse Impact and Test Validation. Burlington, England: Gower.
3Kuang, D. C.Y., & Higgins, J. (April 2008). Application of CSEM-Based Methods in Estimating Modified Angoff Cutoff Scores. In Hurtz, G. (Chair), Integrating Conditional Standard Errors of Measurement into Personnel Selection Practices. Symposium conducted at the meeting of the Society for Industrial and Organizational Psychology, San Francisco, CA.