LETTER
Year : 2021  |  Volume : 67  |  Issue : 1  |  Page : 59-60

Reply to Letter to Editor regarding the article, "The power of subjectivity in competency-based assessment"

Correspondence Address:
T Singh
SGRD Institute of Medical Sciences and Research, Amritsar, Punjab
India

Source of Support: None, Conflict of Interest: None

Check

DOI: 10.4103/jpgm.JPGM_1269_20

1. By quoting an MCQ example, responders have tried to project our statement “Objective assessments generally use 'norm-referenced' approach without any specified criteria, while in subjective assessment the performance of students is generally assessed against a pre-determined criterion and thus follows 'criterion-referenced' approach” as misleading.


Most of this discussion chips in with performance assessments and therefore examples using MCQs are most inappropriate. It is reiterated here that every test item is developed by someone, often a subject expert, and represents a value judgment on what content should be tested and the best possible answer.^[2] While one can say that criterion or norm referencing can be subjective or objective, what we have asserted is that subjective assessments use Criterion referenced approach as the performance of students is generally assessed against a predetermined criterion, and this statement is based on the accepted position in literature.^[3] Moreover, point of contention here is not the use of norm or criterion reference for computation of results, but for applying these concepts at the time of decision-making. Once the assessment is over, the result can be computed in any way depending on the requirement; it's the “reference” at the point of decision-making that matters.


2. The responders have also objected to the statement “in a sense, objective assessment is comparable to 'cross-sectional' study while subjective assessment is 'longitudinal' in nature” and called it misleading. We have clarified in the paper itself that in subjective assessment opinions are built over a longitudinal period after supervision and observation in the light of complex patterns of data collected to reach a coherent collective decision. Subjective assessment is continuous in nature and has elements of formative assessment (of the article), thus making it comparable to a longitudinal study.
3. On our statement, “Standard setting itself is a purely subjective decision,” the responders have cited a reference, to mislead the readers into believing that item response theory (IRT) is a standard setting framework, which it is not.


IRT refers to a class of psychometric measurement models used to estimate examinee ability on the trait being measured and the difficulty of the examination items on the same scale. It eliminates the confounding of test difficulty and student proficiency.^[4] This measurement model is used for accurate test scoring and development and evaluation of test items. Even when using IRT-based standards to accommodate difficulty of items, it is again the panelists who rate the scores to set standards.^[5] Hofstee's, modified Angoff's, Nedelsky's, or borderline group methods, which are commonly used for setting standards for assessment in medical education, are largely based on expert subjective judgment of panelists.^[6] Although standards are subjective, they are not arbitrary and have to be defensible.


4. Our paper does not present any conflict or debate between subjectivity and objectivity. Rather, we are trying to get subjectivity its due, especially for competency-based assessments. This paper offers the possibility of accepting and acknowledging subjectivity as an inherent component for measuring performance which has often been ignored in pursuit of objectivity. Competence is a complex phenomenon and regardless of its definition in terms of traits (knowledge, skills, problem-solving skills, and attitudes) or competencies or competency domains, human judgment is crucial to interpreting assessment results. Expert subjective judgment is also needed to collate information across individual data points and when individual data points are information-rich, and contain qualitative information, simple quantitative aggregation is out of the question and resorting to expert judgment becomes imperative.^[7] Even within conventional education and conventional assessment, the value of expert subjective decisions is being recognized as evidenced by increasing use of mini-Clinical Evaluation Exercise (m-CEX), Direct Observation of Procedural Skills (DOPS), Professionalism Mini Evaluation Exercise (P-MEX), and Multisource Feedback (MSF) amongst others, which are largely subjective tools. Though not used in the MCI model of curriculum, awarding an Entrustable Professional Activity (EPA) to a trainee is again highly subjective decision and is widely used wherever competency-based medical education is in use.



5. Programmatic assessment (PA) is well grounded in theoretical notions around assessment and is based on sound empirical research. Nevertheless, PA needs to be explored further as medical educators first need to identify ways to compile subjective data for comparisons across or against some standard for the purposes of high stakes decision-making. The most important prerequisite for this is the high validity and credibility of these assessments in order to actually support the learners in their learning process and the acquisition of competencies.^[8] PA uses expert subjective judgment as one of its tenets and we as educators need to be clear about the role of subjectivity in performance assessment. Decision regarding promotion of a student, based on multiple data points (including those from highly objective tests), accumulated over a period is made by a committee and again depends heavily on subjective judgments. We feel that this could be one of the reasons for slow acceptance of PA, despite it being theoretically sound. Understanding this process may bring out opportunities to design and shape assessment practices in a meaningfully positive way for utilizing the power of expert subjectivity. Only then will the PA be possibly considered by the medical educationists for general use.

:: References

1.	Virk A, Joshi A, Mahajan R, Singh T. The power of subjectivity in competency-based assessment. J Postgrad Med 2020;66:200-5.   [PUBMED]  [Full text]
2.	ten Cate O, Regehr G. The power of subjectivity in the assessment of medical trainees. Acad Med 2019;94:333-7.
3.	Keynan A, Friedman M, Benbassat J. Reliability of global rating scales in the asse?ssment of clinical competence of medical students. Med Educ 1987;21:477-81.
4.	Downing SM. Item response theory: Applications of modern test theory in medical education. Med Educ 2003;37:739-45.
5.	DeChamplain AF. Standard setting methods in medical education: High-stakes assessment. In: Swanwick T, Forrest K and O'Brian BC, editors. Understanding Medical Education: Evidence, Theory and Practice. 3rd ed. Oxford, Wiley; 2019. Pp 356.
6.	Ben-David MF. AMEE Guide No. 18: Standard setting in student assessment. Med Teach 2000;22:120-30.
7.	van der Vleuten CPM, Schuwirth LWT, Driessen W, Dijkstra J, Tigelaar D, Baartman IKJ, et al. A model for programmatic assessment fit for purpose. Med Teach 2012;34:205-14.
8.	Rotthoff T. Standing up for subjectivity in the assessment of competencies. GMS J Med Educ2018;35;Doc29. doi: 10.3205/zma001175.