Journal of Postgraduate Medicine
 Open access journal indexed with Index Medicus & ISI's SCI  
Users online: 563  
Home | Subscribe | Feedback | Login 
About Latest Articles Back-Issues Article Submission Resources Sections Etcetera Contact
 
  NAVIGATE Here 
  Search
 
 :: Next article
 :: Previous article 
 :: Table of Contents
  
 RESOURCE Links
 ::  Similar in PUBMED
 ::  Search Pubmed for
 ::  Search in Google Scholar for
 ::Related articles
 ::  Article in PDF (288 KB)
 ::  Citation Manager
 ::  Access Statistics
 ::  Reader Comments
 ::  Email Alert *
 ::  Add to My List *
* Registration required (free) 

  IN THIS Article
 ::  Abstract
 ::  Introduction
 ::  Materials and Me...
 ::  Results
 ::  Discussion
 ::  Conclusions
 ::  References
 ::  Article Tables

 Article Access Statistics
    Viewed4327    
    Printed167    
    Emailed12    
    PDF Downloaded24    
    Comments [Add]    
    Cited by others 6    

Recommend this journal


 


 
EDUCATION FORUM
Year : 2010  |  Volume : 56  |  Issue : 4  |  Page : 297-300

Evaluation of clinical skills for first-year surgical residents using orientation programme and objective structured clinical evaluation as a tool of assessment


Department of General Surgery, T N Medical College and B Y L Nair Ch. Hospital, Mumbai, India

Date of Submission26-Apr-2010
Date of Decision22-May-2010
Date of Acceptance01-Jul-2010
Date of Web Publication7-Oct-2010

Correspondence Address:
J S Pandya
Department of General Surgery, T N Medical College and B Y L Nair Ch. Hospital, Mumbai
India
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/0022-3859.70950

Rights and Permissions


 :: Abstract 

Background: Postgraduate specialities require a combination of knowledge and clinical skills. The internship year is less structured. Clinical and practical skills that are picked up during training are not well regulated and the impact is not assessed. In this study, we assessed knowledge and skills using objective structured clinical examination (OSCE). Aim: To evaluate the clinical skills of new first-year surgical residents using orientation programme and OSCE as a tool for assessment. Settings and Design: Observational study. Materials and Methods: Twenty new first-year surgical residents (10 each in 2008 and 2009) participated in a detailed structured orientation programme conducted over a period of 7 days. Clinically important topics and skills expected at this level (e.g., suturing, wound care etc.) were covered. The programme was preceded by an OSCE to test pre-programme knowledge (the "pre-test"). The questions were validated by senior department staff. A post-programme OSCE (the "post-test") helped to evaluate the change in clinical skill level brought about by the orientation programme. Statistical Analysis: Wilcoxson matched-pairs signed-ranks test. Results: Passing performance was achieved by all participants in both pre- and post-tests. Following the orientation programme, significant improvement was seen in tasks testing the psychomotor and cognitive domains. (P = 0.0001 and P = 0.0401, respectively). Overall reliability of the OSCE was found to be 0.7026 (Cronbach's coefficient alpha). Conclusions: This study highlighted the lacunae in current internship training, especially for skill-based tasks. There is a need for universal inclusion of structured orientation programmes in the training of first-year residents. OSCE is a reliable, valid and effective method for the assessment of clinical skills.


Keywords: Clinical skills assessment, objective structured clinical evaluation, residents′ training programme


How to cite this article:
Pandya J S, Bhagwat S M, Kini S L. Evaluation of clinical skills for first-year surgical residents using orientation programme and objective structured clinical evaluation as a tool of assessment. J Postgrad Med 2010;56:297-300

How to cite this URL:
Pandya J S, Bhagwat S M, Kini S L. Evaluation of clinical skills for first-year surgical residents using orientation programme and objective structured clinical evaluation as a tool of assessment. J Postgrad Med [serial online] 2010 [cited 2019 Mar 25];56:297-300. Available from: http://www.jpgmonline.com/text.asp?2010/56/4/297/70950



 :: Introduction Top


All postgraduate specialities in the field of medicine, especially those in surgery, require a combination of knowledge and clinical skills. Existing methods of evaluation like common entrance tests (CETs) at the postgraduate and subspecialty levels test only theoretical knowledge. Formal medical school teaching adheres to the Medical Council of India (MCI) guidelines, but the internship year is less structured. Inconsistencies in quality of training during internship ensure that incoming surgical trainees frequently are underconfident and poorly prepared to assume the different responsibilities that comprise patient care. [1] Clinical and practical skills are supposed to be picked up "on-the-job." While many institutions provide entry-level orientation programmes, these are neither well regulated nor is their impact always objectively assessed. The traditional clinical examination has been shown to have serious limitations in terms of its validity and reliability. [2] In this study, we attempted to assess and compare the practical knowledge and skills of entering first-year surgical residents before and after an orientation programme (the teaching-learning tool), using Objective Structured Clinical Examination (OSCE) as the assessment tool .The OSCE was designed and validated by senior surgical faculty. In a similar study from the UK, face and content validity were assessed by a panel of clinicians and from feedback from OSCE participants. [3]


 :: Materials and Methods Top


Approval from the Hospital Ethics Committee was sought and obtained before carrying out this observational study and written informed consent was obtained from the participants.

The participants were 20 new first-year surgical residents registered for MS degree with Maharashtra University of Health Sciences. Two consecutive batches of 10 residents each entering in May 2008 and May 2009 were included. They received a pre-orientation assessment OSCE (the "pre-test") to judge baseline competence followed by a lecture and demonstration-based orientation programme, which was then followed by a post-orientation assessment OSCE (the "post-test") to assess the impact of the programme. The entire activity spanned 10 days and was held early during the first month of their first post.

The OSCE was designed by senior surgical faculty with valuable input from senior residents as to what exactly they expected of their new juniors. The topics chosen were representative of the type and level of clinical and practical skills required by a first-year resident. These included demonstration of clinical signs, performance of ward procedures, radiological identification and common theory topics. Case-note writing skills, basic administrative duties, consent-taking and breaking bad news were also tested. Before administration of the tests, the stations were demonstrated to senior surgical faculty and validated by them. Both pre-and post-test OSCE consisted of 10 stations each [Table 1] and [Table 2]. All three domains, i.e. cognitive, psychomotor and affective, were tested separately. A combination of actual patients, simple models for simulation, case scenarios, brief viva voce and questions needing written answers was used. Each station was scored out of five marks using preset checklists. Each point on the checklist was scored according to a binary, i.e. "yes/no," scale and marks were given accordingly. Surgical staff and residents acted as observers to score the stations. Candidates were required to score 25 out of a total of 50 marks (50%) to pass.
Table 1: OSCE stations grouped by domain tested


Click here to view
Table 2: Sample OSCE station

Click here to view


After the pre-test, a lecture and demonstration-based orientation programme designed by senior faculty was conducted by them over 7 days. Important clinical, practical and administrative "must-know" topics [Table 3] were taught in an informal small-group setting with plenty of opportunities for hands-on practice by the residents.
Table 3: Orientation programme schedule - Lectures and demonstrations

Click here to view


The post-test was held immediately on completion of the programme. Feedback questionnaires from the participants were completed after the orientation programme and after each assessment.

The results were tabulated and analyzed using standard statistical software (SPSS, v. 17).


 :: Results Top


The results of both batches of new residents were considered together as one batch of 20 to facilitate analysis.

Passing performance (50%) was achieved by all 20 residents in both pre-test and post-test.

Analysis of both pre- and post-test results showed that the new residents fared better overall on questions testing the cognitive domain and less well on those testing the psychomotor and affective domains. This finding was expected and was attributed to both a lack of sufficient practical experience as well as a lack of useful training during the internship year.

Pre- and post-test results were then grouped into three groups [Table 4],[Table 5],[Table 6]: cognitive domain station scores (four stations, 20 marks, "knowledge"), psychomotor domain station scores (five stations, 25 marks, "skills") and affective domain station scores (one station, 5 marks, "emotional content"). Pre- and post-test scores for each group were compared to determine whether the post-test results were better and, if so, whether the difference was of significance. The participants' feedback was also scrutinized to get an idea of the impact of the orientation programme, the response to the use of OSCE as an assessment tool and to gather suggestions for improvement.
Table 4: Scores for stations testing cognitive domain

Click here to view
Table 5: Scores for stations testing psychomotor domain

Click here to view
Table 6: Scores of stations testing affective domain


Click here to view


In the post-test, overall performance by all participants improved. The Wilcoxson matched-pairs signed-ranks test was found to be appropriate for analyzing the data. The difference between the pre-and post-test scores for the stations testing the psychomotor domain was found to be significant (two-tailed, P = 0.0001). The difference between pre-and post-test scores for the stations testing the cognitive domain was also found to be significant (two-tailed, P= 0.0401). However, there was only minimal improvement in the post-test scores for the affective domain station, and the difference between the pre- and post-test scores was not significant (P = 0.8203).

Overall reliability of the OSCE was found to be 0.7026 (Cronbach's coefficient alpha), showing good reliability.

Data gathered from the feedback forms indicated that the residents felt more informed and empowered after the orientation programme. The OSCE format of assessment was appreciated as being comprehensive and fair.


 :: Discussion Top


Orientation programmes for incoming residents are offered in many forms: as a set of lectures in one day, as provision of "resident's manuals", as practical demonstrations, as computer simulations or as simulated clinical "calls." [4],[5] In some countries, formal courses like Basic Life Support, Advanced Cardiac Life Support and Basic Surgical Skills courses are mandatory for new residents. [4] Surgical residents in India complete their course in 3 years. Most of their time is spent on patient care and there is little left for formal training. Postgraduate departments have to constantly come up with innovative ways to impart knowledge-based confidence to the new juniors. In this study, the orientation programme was administered in a lecture-demonstration format suitable to the department's resources.

The results of this small study indicated that provision of basic clinical and practical instruction at the beginning of a surgical residency resulted in an immediate improvement in performance. To that end, the orientation programme was successful as a teaching-learning tool. Maintaining development and improvement of skills would however require such programmes to be held on an ongoing basis in addition to the formal postgraduate curriculum.

The results of the pre-test highlighted the lacunae in internship training. Questions answered consistently incorrectly served to point out common misconceptions about ward work, e.g. the use of undiluted hydrogen peroxide solution on healthy wounds.

Similarly, consistent mistakes in the post-test indicated which procedures needed close supervision and which residents needed extra help.

There were some limitations however. Although a statistically significant difference was shown in the means of nine of the 10 stations tested, it would still be incorrect to attribute the improvement in results entirely to the orientation programme. The sample size was small, the residents might have understood the OSCE format better the second time around and might have worked on their skills during the orientation week as deficiencies had been pointed out in the pre-test. In order to remove these biases, our follow-up study would include larger numbers, multiple batches, longer and more periodic testing and comparison between senior and junior residents. In particular, more stations would be included to test the affective domain (communication skills and professionalism).

The OSCE format is fast becoming the accepted method for clinical assessment all over the world. [2],[6],[7] It is proven to be a reliable, valid and effective assessment tool due to its clear advantages of objectivity, fairness and reproducibility. This was our experience as well. In this study, the residents participated in both pre- and post-test OSCEs with enthusiasm and viewed the stations as learning opportunities. They felt that they had been examined fairly and that many topics had been tested in a short time. The most cited positive feedback was that the entire programme addressed the juniors' common problem areas in a friendly yet objective way; thus, increasing their confidence. The most cited suggestion was that more such programmes held more frequently and covering more topics should be held.

A more proactive stance by postgraduate teachers in India toward the initial and ongoing formal training of surgical residents is suggested. Creative use of methods like problem-based learning and moulages (dynamic case scenarios) can supplement more traditional seminars and case presentations. "Mentoring" of the new residents by senior residents and/or faculty members like the postgraduate guide could be an informal adjunct to training. Regular orientation programmes and yearly evaluation using OSCE are also suggested.


 :: Conclusions Top


From this study, we conclude that structured orientation programmes are desired by and objectively benefit new residents. Evaluation of new residents' clinical skills can be successfully performed using OSCE. The OSCE format of assessment was found to be reliable, valid and effective. It can be easily adapted as a teaching tool and assessment tool. Interesting and creative ways to encourage learning by residents need not be resource-dependent. All contribute toward the ultimate goal of improved patient care.

 
 :: References Top

1.Bansal RK. Need for strengthening of internship training in India. Educ Health (Abingdon) 2004;17:332-8.  Back to cited text no. 1  [PUBMED]  [FULLTEXT]  
2.Newble D. Techniques for measuring clinical competence: objective structured clinical examination. Med Educ 2004;38:199-203.  Back to cited text no. 2  [PUBMED]  [FULLTEXT]  
3.Walters K, Osborn D, Raven P. The development, validity and reliability of a multimodality objective structured clinical examination in psychiatry. Med Educ 2005;39:292-8.  Back to cited text no. 3  [PUBMED]  [FULLTEXT]  
4.Nielsen PE, Holland RH, Foglia LM. Evaluation of a clinical skills orientation program for residents. Am J Obstet Gynecol 2003;189:858-60.  Back to cited text no. 4  [PUBMED]  [FULLTEXT]  
5.Meier AH, Henry J, Marine R, Murray WB. Implementation of a Web- and simulation-based curriculum to ease the transition from medical school to surgical internship. Am J Surg 2005;190:137-40.  Back to cited text no. 5  [PUBMED]  [FULLTEXT]  
6.Bansal PK, Saoji VA, Gruppen LD. From a "generalist" medical graduate to a "speciality" resident: can an entry-level assessment facilitate the transition? assessing the preparedness level of new surgical trainees. Ann Acad Med Singapore 2007;36:719-24.  Back to cited text no. 6  [PUBMED]  [FULLTEXT]  
7.Barman A. Critiques on the objective structured clinical examination. Ann Acad Med Singapore 2005;34:478-82.  Back to cited text no. 7  [PUBMED]  [FULLTEXT]  



 
 
    Tables

  [Table 1], [Table 2], [Table 3], [Table 4], [Table 5], [Table 6]

This article has been cited by
1 Tools to reduce first year nursing studentsę anxiety levels prior to undergoing objective structured clinical assessment (OSCA) and how this impacts on the studentęs experience of their first clinical placement
Annette Stunden,Elizabeth Halcomb,Diana Jefferies
Nurse Education Today. 2015; 35(9): 987
[Pubmed] | [DOI]
2 WITHDRAWN: Tools to reduce first year nursing studentsę anxiety levels prior to undergoing objective structured clinical assessment (OSCA) and how this impacts on the studentsę experience of their first clinical placement
Annette Stunden,Elizabeth Halcomb,Diana Jefferies
Nurse Education Today. 2014;
[Pubmed] | [DOI]
3 Development of a Minimally Invasive, Injectable, Shape Memory Suture and Delivery System
Jeffrey L. Olson,Robin Shandas,Michael Erlanger
Annals of Biomedical Engineering. 2012; 40(7): 1520
[Pubmed] | [DOI]
4 Development of a minimally invasive, injectable, shape memory suture and delivery system
Olson, J.L. and Shandas, R. and Erlanger, M.
Annals of Biomedical Engineering. 2012; 40(7): 1520-1529
[Pubmed]
5 Lessons learnt from evaluation of the orientation program for new surgical residents using objective structured clinical examination-Based assessment
Pandya, J.S. and Bhagwat, S.M. and Kini, S.L.
Journal of Postgraduate Medicine. 2012; 58(1): 85
[Pubmed]
6 Improvement of psychometric properties of the objective structured clinical examination when assessing problem solving skills of surgical clerkship
Al-Naami, M.Y., El-Tinay, O.F., Khairy, G.A., Mofti, S.S., Anjum, M.N.
Saudi Medical Journal. 2011; 32(3): 300-304
[Pubmed]



 

Top
Print this article  Email this article
Previous article Next article
Online since 12th February '04
© 2004 - Journal of Postgraduate Medicine
Official Publication of the Staff Society of the Seth GS Medical College and KEM Hospital, Mumbai, India
Published by Wolters Kluwer - Medknow