Learn more

New Assessment Data Allows Employers to Make Informed Scoring Decisions

Online Healthcare Competency Validation SealBeginning with the newly validated Physical Therapy and Occupational Therapy assessments, the assessment results page will provide additional information for an employer to make an informed decision regarding the applicant’s assessment score.

Angoff Cutoff Score

One of the most commonly used approaches to establishing a cutoff score is the method detailed by Angoff (1971)1. This technique is particularly attractive in applied personnel selection settings because it is easily understood by subject matter experts (SMEs), it has high face validity to job applicants, the courts regularly endorse the technique, and it stands up to academic scrutiny.2

Although there are many variants, the classic Angoff method for setting cutoff scores
involves:

  1. Select and train SMEs who are familiar with the job and have them review each test item;
  2. Obtain ratings from SMEs on the percent of minimally qualified applicants who would be expected to answer each item correctly;
  3. Analyze SME ratings to obtain an “unmodified” Angoff cutoff score;
  4. Administer the test, and
  5. Facilitate a discussion with SMEs on the impact/validity of the unmodified cutoff score.3

Seals for NT ValidationWith each new NurseTesting clinical assessment, the initial “Unmodified” Angoff Cutoff Score will be displayed next to the applicants overall score. Based on the applicants overall score, the system will automatically compare this score to the “Unmodified” Angoff Cutoff Score and display either “PASS” or “FAIL” on the results page. This enables the recruiter the ability to quickly determine if the applicant passed that particular assessment.

It is important to note, that each assessment will have its own, unique Angoff Cutoff Score. NurseTesting does not recommend the use of “Arbitrary Cutoffs” such as the “80% rule.” Arbitrary cutoffs do not make academic or practical sense.

Furthermore, they can incense applicants who may realize that a meaningless standard in the selection process was used to make very meaningful decisions about their lives and careers.2 As recently as this past April (2009), the United States Supreme Court heard the case Ricci v. DeStefano (07-1428 & 08-328) in which the use of “arbitrary cutoffs” was an issue brought before the Court.

Modified Angoff Cutoff Score

Initially, when an assessment is administered, the “Unmodified” Angoff Cutoff Score is used until a sufficient sample size of assessment data is available for analysis. Once NurseTesting has obtained a sufficient sample size for each assessment, a comprehensive analysis will be conducted to determine if a modified Angoff Cutoff Score is necessary. When it comes to best practices for developing cutoffs, there is perhaps none better than the modified Angoff method.2

The “cutoff score” will be refined / modified based on the data collected during the initial launch of the assessment to account for assessment unreliability and actual test-taker performance. NurseTesting, using the Test Validation and Analysis Program (TVAP), will refine the assessment and identify test items that might be:

  • Too difficult
  • Too easy
  • Negative or low Point Biserials – showing how correlated the item is to the overall test score.
  • Cronbach’s Alpha review once flawed items are removed
  • Differential Item Functioning –test items function in different ways for different groups of test-takers

Once the modified Angoff Cutoff Score is calculated using the Conditional Standard Error of Measurement (CSEM), the assessment results page will display the “modified Angoff Cutoff Score” instead of the initial unmodified Angoff Cutoff Score. Based on statistical analysis, including an Adverse Impact analysis, the recommended modified Angoff Cutoff Score will be used for all future administered assessments. As stated before, each assessment will have its own, unique modified Angoff Cutoff Score.

NurseTesting implemented these processes to continue providing validated content for new exams that meet the Uniform Guidelines for Employee Selection Procedures.

1Angoff, W. J. (1971). Scales, Norms, and Equivalent Scores. In Thorndike, R. L., Educational Measurement (508-600). Washington, DC: American Council on Education.
2Biddle, D. (2005). Adverse Impact and Test Validation. Burlington, England: Gower.
3Kuang, D. C.Y., & Higgins, J. (April 2008). Application of CSEM-Based Methods in Estimating Modified Angoff Cutoff Scores. In Hurtz, G. (Chair), Integrating Conditional Standard Errors of Measurement into Personnel Selection Practices. Symposium conducted at the meeting of the Society for Industrial and Organizational Psychology, San Francisco, CA.
This entry was posted in Client Realized Benefits, ROI on Testing, Selecting a Testing Provider, Testing Best Practices and tagged , , , . Bookmark the permalink. Comments are closed, but you can leave a trackback: Trackback URL.

2 Comments

  1. Posted August 24, 2009 at 6:44 pm | Permalink

    Great idea, but will this work over the long run?

  2. Meredith
    Posted August 26, 2009 at 12:59 pm | Permalink

    Wonderful thought. It is important that if you are using a standardized numeric value for scoring that the numeric value will be able to withstand the test of time.

    The setting of the Modified Angoff score is done at least annually. We look at the statistical performance of the assessment and if certain metrics such as disparate impact or unreliability begin to become an issue, the Angoff score will be modified accordingly.

    In using the Angoff & Modified Angoff score, NurseTesting follows the Uniform Guidelines on Employee Selection Procedures (1978) as the guiding principals upon which we are developing all of our newly valdiated exams.

    Please let us know if this did not adequately answer your question!

One Trackback

  • Subscribe to Our Blog


  • Blog Categories

  • Social Links