Scoring/Reporting
CRITERION-REFERENCED EXAMS
Licensure exams, such as the ones administered by the National Board of Examiners in Optometry, are criterion-referenced. Criterion-reference exams are appropriate for licensure because no artificial limits are placed on the number of candidates who can pass or fail. Instead, performance is judged against a pre-determined standard: sufficient knowledge, skills, and abilities for the safe and effective practice of optometry. This standard is held constant across different exam forms and administration years. In summary, a criterion-referenced exam bases a candidate’s pass/fail decision solely on her or his exam performance, without reference to other candidates.
In contrast, norm-referenced exam results are interpreted with respect to the performance of other test-takers. Examples of norm-referenced exams include the SAT and intelligence tests. Results of these exams can include percentile rankings of candidates: scoring at the 85th percentile means that one performed better than 85% of similar candidates (i.e., the “norm group”). Norm-referenced exams sometimes utilize a normal, bell-shaped, curve to achieve a desired distribution of scores (i.e., grading on a curve). This practice is not used with criterion-referenced licensure exams.
In contrast, norm-referenced exam results are interpreted with respect to the performance of other test-takers. Examples of norm-referenced exams include the SAT and intelligence tests. Results of these exams can include percentile rankings of candidates: scoring at the 85th percentile means that one performed better than 85% of similar candidates (i.e., the “norm group”). Norm-referenced exams sometimes utilize a normal, bell-shaped, curve to achieve a desired distribution of scores (i.e., grading on a curve). This practice is not used with criterion-referenced licensure exams.
PASS/FAIL DECISIONS
It is essential to choose defensible passing scores to minimize potential misclassification of candidates. Misclassification can result in significant negative consequences. If the passing score for an exam is set unreasonably high, then it is more likely that a qualified candidate will fail the exam. On the other hand, if the passing score for the exam is insufficiently rigorous, then the risk of an unqualified candidate passing the exam increases. Because these are high stakes examinations, NBEO uses psychometric practices aligned to industry standards and best practices. Establishing the pass-fail cutoff score for an examination is accomplished through a formal process known as standard setting.
A standard setting study involves convening panels of practicing optometrists representing the breadth of the profession in terms of demographics, training, and modes of practice. These subject matter experts are trained in standard setting methodologies where they explicitly consider both the difficulty of the exam and the expectations for safe and effective practice at the point of licensure. In other words, the goal of criterion-referenced standard setting methods is to determine what level of candidate performance aligns with professional expectations for competence, given the content and difficulty of the exams. NBEO selects specific standard setting methods and variations that are appropriate to the distinct formats and scoring methods of the Part I ABS, Part II PAM, Part III PEPS, and other NBEO exams to determine each cut score.
Just as content expectations change over time, necessitating updates to NBEO content matrices, expectations for the level of competence expected of a newly licensed optometrist evolves over time as well. Therefore, NBEO conducts new standard setting studies to revalidate the passing scores of the exams approximately every 5-7 years. In other years, different exam forms are placed on the same scale through a psychometric statistical process known as equating. This process ensures that all candidates are held to the same performance standard regardless of when they take the exam or which exam form they receive.
Although NBEO establishes overall pass-fail standards, it does not make licensure decisions. Licensure is the legal responsibility of the state boards of optometry, some of which require that candidates attain individual section standards. Therefore, it is possible for candidates to meet the NBEO threshold for passing, but not a state boards’ standards. To avoid confusion, all candidates are advised to contact the administrative offices of the state boards to which they seek licensure to fully understand how NBEO scores are utilized, and any additional jurisdictional requirements.
A standard setting study involves convening panels of practicing optometrists representing the breadth of the profession in terms of demographics, training, and modes of practice. These subject matter experts are trained in standard setting methodologies where they explicitly consider both the difficulty of the exam and the expectations for safe and effective practice at the point of licensure. In other words, the goal of criterion-referenced standard setting methods is to determine what level of candidate performance aligns with professional expectations for competence, given the content and difficulty of the exams. NBEO selects specific standard setting methods and variations that are appropriate to the distinct formats and scoring methods of the Part I ABS, Part II PAM, Part III PEPS, and other NBEO exams to determine each cut score.
Just as content expectations change over time, necessitating updates to NBEO content matrices, expectations for the level of competence expected of a newly licensed optometrist evolves over time as well. Therefore, NBEO conducts new standard setting studies to revalidate the passing scores of the exams approximately every 5-7 years. In other years, different exam forms are placed on the same scale through a psychometric statistical process known as equating. This process ensures that all candidates are held to the same performance standard regardless of when they take the exam or which exam form they receive.
Although NBEO establishes overall pass-fail standards, it does not make licensure decisions. Licensure is the legal responsibility of the state boards of optometry, some of which require that candidates attain individual section standards. Therefore, it is possible for candidates to meet the NBEO threshold for passing, but not a state boards’ standards. To avoid confusion, all candidates are advised to contact the administrative offices of the state boards to which they seek licensure to fully understand how NBEO scores are utilized, and any additional jurisdictional requirements.
SCALING AND REPORTING
For NBEO examinations a total raw score is computed and then converted to a scaled score ranging from 100 to 900. A score of 100 represents the lowest scaled score, while 900 represents the highest possible scaled score. A score of 300 represents the overall pass-fail cutoff score. The use of scaled scores allows for direct comparison of scores from one examination administration to another because the passing standard is always the same -- a scaled score of 300.
Sections and special examinations are reported on a 0 to 99 scale. On this scale, zero represents the lowest possible score, 99 represents the highest possible scores, and 75 represents the pass-fail cutoff score. These scores also are transformed using a single scaling equation.
Section scaled scores cannot be added to reproduce the overall scaled scores. This is because sections have a different number of items and overall point value. A simple sum of these scaled scores would skew the outcome. The scaling of the section from 0 to 99 does not have a summative translation to the scaling from 100 to 900. That is, adding up the scaled scores in sections is not equivalent to the overall scale score due to the process of scaling itself.
Sections and special examinations are reported on a 0 to 99 scale. On this scale, zero represents the lowest possible score, 99 represents the highest possible scores, and 75 represents the pass-fail cutoff score. These scores also are transformed using a single scaling equation.
Section scaled scores cannot be added to reproduce the overall scaled scores. This is because sections have a different number of items and overall point value. A simple sum of these scaled scores would skew the outcome. The scaling of the section from 0 to 99 does not have a summative translation to the scaling from 100 to 900. That is, adding up the scaled scores in sections is not equivalent to the overall scale score due to the process of scaling itself.
HOW TO READ A SCORE REPORT
