CLIPP Assessment Tools Validation Process

The CLIPP Assessment Tool Validation project is a joint project between MedU and Rush Medical College.  Keith Boyd, MD and Sharon Sholiton, MD of Rush created a case analysis tool for use with CLIPP cases that provides an assessment tool to evaluate student learning from CLIPP cases.  This project, funded and managed by MedU and led by CLIPP Editor-in-Chief Sherilyn Smith, MD, will research and further develop the tool to validate that it accurately discriminates between those who understand the CLIPP case content and can apply that knowledge in a structured assessment from those who cannot.

In an effort to create the highest quality tools for use in the pediatric clerkship, we will be conducting a validation study related to this assessment tool. This will be a multi-step process and will focus on establishing the validity of the model of assessment and the specific link of the assessment tool to CLIPP. We are convening a panel of experts to participate. These panelists will participate in a Delphi procedure to establish expert consensus about the method of assessment in the CLIPP Case Analysis Form, the ease of use of the assessment tool and the accuracy of answer keys associated with this new assessment tool. Over the course of the next year, MedU together with our partners at Rush and the panel of expert reviewers seek to make our results and the tool more widely available by producing scholarly work including manuscripts related to the project as well as workshops about the validation project and the specific methods used. The planned outcome is to make the tool widely available to clerkship directors who use CLIPP as part of their pediatrics clerkship program.

The following paragraphs describe the background and rationale behind the case analysis tool.

CLIPP cases provide an opportunity for M3 students to supplement their clinical patient experiences through interaction with a cadre of computerized simulations of patient encounters. These cases present a complement of classic pediatric diseases that may not be experienced individually by each student. As a clerkship director, this computerized bank of core patient encounters helps to provide a more uniform experience for students in diverse institutions throughout all seasons of the year.

The on-line platform allows clerkship directors to monitor time spent on each case by the student, but not to assess the student’s responses or provide direct feedback. Completion of a simple, structured written worksheet (the CLIPP case analysis tool) by the student will enhance his/her learning experience by facilitating direct communication between the clerkship director and the student regarding the case.

As the student works through each CLIPP case, use of the CLIPP tool facilitates the development of clinical reasoning skills by clearly delineating the steps involved in patient evaluation and clinical care. The student identifies and documents pertinent data obtained via history and physical exam, and then succinctly summarizes the key information into a problem statement. This focused patient assessment then drives the development of a prioritized differential diagnosis.  Finally, the student generates an initial diagnostic and treatment plan. This staged process parallels the structure of the cases themselves, as well as the process through which clinicians assess patients.  Using the tool requires the student to develop independent learning skills and allows the clerkship director to assess his/her clinical reasoning process and provide individualized feedback.

This tool was developed explicitly for use in conjunction with the CLIPP cases at Rush University Medical Center in Chicago, IL and has been used in that clerkship since July 2002. The CLIPP case analysis process and assessment tool reinforce the Problem Oriented Medical Record system that is utilized at Rush throughout the preclinical and clinical curriculum.

fmCASES Update August 2011

Maintenance and ongoing development of fmCASES

During the past year, fmCASES completed its first comprehensive evaluation and improvement of the original fmCASES. The Editorial Board and 29 case editors reviewed each case to ensure accuracy and consistency of information, conveyed with a consistent style. The editors also responded to many inquiries and suggestions for improvement from preceptors and students.

We conducted a detailed analysis of the content covered by fmCASES, and compared the coverage to national curricular guidelines. The national objectives used were primarily from the Society of Teachers of Family Medicine’s (STFM’s) Family Medicine Clerkship Curriculum (originally entitled C4) and the STFM’s Family Medicine Curriculum Resources (FMCR) family medicine clerkship curriculum. This detailed analysis guided edits to all the cases this year and also led to the creation of four new cases that are available as of July 2011:

  • Case #30: 27-year-old female - Labor and delivery - Mrs. Gold
    Author: Wetona Suzanne Eidson-Ton, M.D., M.S., University of California, Davis
    Editor: Jason Chao, M.D., Case Western Reserve University
  • Case #31: 66-year-old female with shortness of breath - Mrs. Hernandez
    Author: Ellen Miller, M.D., New York Medical College
    Editor: David Anthony, M.D., M.Sc., Brown University
  • Case #32: 33-year-old female with painful periods - Ms. Tomlin
    Author: Jessica Servey, Lt. Col, USAF, MC, Uniformed Services University
    Editor: Katherine Margo, M.D., University of Pennsylvania


  • Case #33: 28-year-old female with dizziness - Mrs. Saleh
    Author: Theresa Woehrle, M.D., M.P.H., Keck School of Medicine at University of Southern California (USC)
    Editor: Stephen Scott, M.D., M.P.H., Weill Cornell Medical College in Qatar

Beginning July 2011, subscription to fmCASES includes the 29 original cases and the four newly developed fmCASES, in addition to 5 CLIPP and 2 SIMPLE cases that covered core objectives. These cases were assembled to represent the full breadth of care of family medicine, to the depth of knowledge expected by a 3rd year medical student.

Currently, we have 82 institutions subscribing to fmCASES, including 72 allopathic schools, 7 osteopathic schools, 1 PA program, and 2 Canadian schools.

Written exam questions and a final exam created

A team of family medicine educators led by Roger Berkow and Kent Sheets constructed a database of 300 multiple-choice questions during the first half of 2010, so that subscribers to fmCASES will be able to create written examinations that test student mastery of this content. These questions were validated during 2010-11, and revised based upon item analysis. New questions were constructed to cover all 40 cases in the fmCASES’ curriculum.

As of this academic year, fmCASES subscribers have the option to use the 360-item exam bank, free of charge, as part of their fmCASES subscription. A clerkship director can use these questions to create an institution-specific exam that assesses the student's understanding of the cases used at that school. Alternately, a clerkship director may choose to subscribe to a standardized 100-item exam that covers the complete content of all 40 cases in the fmCASES’ curriculum.  iInTIME provides an exam-scoring service for this exam for a fee of $250.  iInTIME will be able to provide comparison data to institutions that use the scoring service.

We are investigating solutions to address the variance in clerkship lengths and curriculum coverage.  With programs that range from 4 to 8 weeks, a single standardized exam, whether a shelf exam or the fmCASES exam, may not meet the specific needs of each individual clerkship program, as the fmCASES exam is intended to cover a student's knowledge of the entire curriculum.  The fmCASES Editors-in-Chief are working with the Society of Teachers of Family Medicine (STFM) to evaluate potential solutions to this challenging problem. 

To learn more about the fmCASES exam, please visit MedU at http://www.med-u.org/virtual_patient_cases/fmcases_final_exam_faq.

Keeping STFM community informed of the fmCASES

At both of the 2011 STFM meetings, the medical student education-focused conference and the annual spring conference, we held an integration workshop and a poster session to inform STFM members about the progress of fmCASES and to assist users to integrate fmCASES effectively into clerkships.  We plan to continue this outreach during 2012. And, also in 2012, we will offer a special invitation workshop at STFM’s Conference on Medical Student Education to help case editors understand their role within the larger project, as well as to provide an opportunity for the Case Editors to meet and work with the Associate Editors and Editors-in-Chief.

Looking to the future

To enhance the value of fmCASES, the editorial board has been developing plans to improve the polling questions and to create precepting tools.

Polling questions are designed to force the learner to make one choice, but there is no one absolutely right answer. These questions can help the student expand the repertoire of behavior appropriate in clinical care, and to build new approaches to managing complexity, uncertainty, and ambiguity. We will use student feedback as well as interviews and focus groups to learn how students are viewing the polling questions, and then convene the case editors and associate editors to generate a plan for improvement.  

In recent years, a significant portion of medical education has shifted to the ambulatory setting. Many of the family medicine clerkships in the U.S. depend upon volunteer, community preceptors. It can be difficult enough to have a student working side-by-side with you in a busy outpatient practice, and so how can this physician have time to pull in content from fmCASES? Using interviews and focus groups, we plan to identify the needs of ambulatory preceptors and develop tools and resources to facilitate their teaching of medical students. We hypothesize that well-designed tools based on fmCASES will improve student learning and enhance the preceptor’s pride in ensuring that the student learns the required content. We hope that the preceptors will also find increased efficiency in teaching if they can integrate fmCASES’ content into discussion about patients' health.

 

CLIPP updates for 2011/12

Annual case updates underway

CLIPP Case editors are busy conducting the annual review of their cases to make sure medical content is up-to-date, accurate and , meets the curricular learning objectives for that case. In addition, case editors are working with the CLIPP Editorial Board to revise the case learning objectives to ensure clear and consistent wording across cases. We hope this will be especially helpful to clerkship directors as you plan your curriculum. Please note that whenever content changes are made to the case, the resources associated with that case (case summary/KTPs, Final Exam Questions, and Questions for Further Consideration) are subsequently reviewed and revised as needed.

We would like to thank CLIPP case editors and associate editors for their hard work to ensure both accuracy of case content and implementation of effective pedagogical practices.

Important changes to take place as of July 1

Please note the following changes that will be implemented on or before July 1.

CLIPP Case 24: The case of Matthew, a 2 year-old boy with altered mental status, has been extensively revised and should be available July 1st. The original case involved severe plumbism, whereas the revised case will focus on a broader view of toxic ingestions (including recognition of toxidromes and management of acute ingestions, curricular objectives that were not covered well previously within CLIPP). Lead screening is now covered in Case 3(3 year-old well child visit)

CLIPP Case 28: During the last week of June, Case 28 will be unavailable for a day as we implement substantial revisions. Although the learning objectives for this case about developmental delay will remain unchanged, the case editor and associate editor have implemented substantial improvements to the organization and flow of case content as well as important and comprehensive updates to developmental screening techniques.

Diagnostic Network Scoring:Cases with a diagnostic network include three important steps to help students hone their clinical reasoning skills. The student is first asked to identify key findings for the patient at the current point in time. Next, the student develops a differential diagnoses. Finally, the student rates the connection between each finding and each diagnosis using a five-point scale (“strongly speaks against” to “strongly speaks for”). In an effort to further hone the diagnostic reasoning process, we are modifying the networks across all CLIPP Cases where students will now be asked to rate the network connections using a three-point scale (does not speak for, neutral, speaks for). We hope this will provide even more clarity while reducing frustration and time spent working through the diagnostic networks.

New CLIPP Teaching Tool

In response to requests for additional teaching materials related to the CLIPP cases, we have created a series of questions that supplement and extend the teaching in CLIPP and we need your feedback!

Each CLIPP case now has additional questions (Questions for Further Consideration, or QFC’s) that target one or more of the following areas: basic science correlation, clinical correlation, clinical reasoning, management (includes diagnostic testing), ethics/professionalism, communication and/or biostatics.

These questions can be used in a variety of ways: topics for group discussion, student self-study, oral examination questions, supplemental clinical reasoning exercises, or for 4th year student curricula (to name a few). 

We encourage you to take a look at the QFC’s which have been published within the password-protected CLIPP Instructors’ Area at med-u.org. Please consider providing feedback about the QFC content and/or organization as we strive to make this a user-friendly and useful resource for the pediatric clerkship (email: qfc-feedback@med-u.org).

SIMPLE Exam Development

The Simulated Internal Medicine Patient Learning Experience (SIMPLE) virtual patient cases are in use in the Internal Medicine Clerkship at 75 medical schools. The 36 virtual patient cases in SIMPLE are designed to encompass the learning objectives of the Clerkship Directors in Internal Medicine (CDIM) – Society for General Internal Medicine Core Medicine Clerkship Curriculum and focus on teaching core internal medicine knowledge and clinical reasoning. Based on extensive feedback from the 75 schools that use SIMPLE, the SIMPLE Editorial Board developed an exam to assess student understanding of the content of the 36 virtual patient cases for Internal Medicine. But it's not just any assessment, the SIMPLE Exam will uses Key Features problems to assess students' ability to apply what they have learned to clinical decision making...and it will be administered online!

Key feature problems consist of a case scenario followed by one or more questions pertaining to the problem’s key features, or critical steps in the resolution of the problem. According to Page and Bordage a key feature “focuses on a step in which examinees are most likely to make errors in the resolution of the problem”, and represents “a difficult aspect of the identification and management of the problem in practice”.  Key features examinations have been shown to effectively assess the ability to make correct clinical decisions and are used in the Medical Council of Canada’s national qualifying examination, the Royal Australian College of General Practitioners certification examination for fellowship, and the American Society of Colon and Rectal Surgeons’ Colon and Rectal Surgery Educational Program Self-assessment Examination. A version of a key features examination has been developed for Canadian medical students, but there are no reports of key features examinations developed for on-line assessment of U.S. medical students.

Most medical schools, utilize the National Board of Medical Examiners Medicine Subject Exam to assess students’ knowledge at the end of the Medicine Clerkship.10  The majority of questions are constructed in a standard multiple-choice question format, and the exam is administered on paper.  The online Key Features Examination differs from the Subject Exam in its focus on clinical decision-making, alignment with the SIMPLE virtual patient cases and the CDIM curriculum, question format, and delivery method.  Therefore, the first step in validating the examination requires a study of the acceptability and feasibility of the examination.  Information from this study may be used to revise the examination delivery method, procedures, and orientation, or content.

References

  1. Page G.  Bordage.  Allen T.  Developing key-feature problems and examinations to assess clinical decision-making skills.  Academic Medicine.  1995;70:194-201.
  2. Page G.  Bordage G.  The Medical Council of Canada’s key features project:  A more valid written examination of clinical decision-making skills.  Academic Medicine.  1995;7(2):104-110.
  3. Tamblyn R. Abrahamowicz M. Dauphinee D.  Wenghofer E. Jacques A. Klass D. Smee S. Blackmore D. Winslade N. Girard N. Du Berger R. Bartman I. Buckeridge DL. Hanley JA.  Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities.  JAMA.  2007;298(9):993-1001.
  4. Trudel JL.  Bordage G.  Downing SM.  Reliability and validity of key feature cases for the self-assessment of conol and rectal surgeons.  Annals of Surgery.  2008;248:252-258.
  5. Wenghofer E. Klass D. Abrahamowicz M.  Dauphinee D. Jacques A. Smee S. Blackmore D. Winslade N. Reidel K. Bartman I. Tamblyn R.  Doctor scores on national qualifying examinations predict quality of care in future practice.  Medical Eduation.  2009;43:1166-1173.
  6. Farmer EA.  Hinchy J.  Assessing general practice clinical decision-making skills:  the key features approach.  Australian Family Physician, 2005.  34(12):1059-1061.

MedU at AAMC


Association of American Medical Colleges 2010 Annual Meeting
November 5-10, 2010
Washington DC

MedU will be represented at the AAMC 2010 Annual Meeting. Please join us at Exhibit Booth 809 to learn more about our virtual patient cases. iInTIME staff, MedU Editorial Board members, and editorial board members from CLIPP, fmCASES, SIMPLE, and WISE-MD will be available to answer any questions you may have. We hope to see you there!