Introduction

Life is a test. It is only a test.  If this was your actual life, you would have been given better instructions.   ---Unknown.

Junior was being chided for his low grades. Little Robert, who lived a few doors away, was held up as an example. "Robert doesn't get C's and D's does he?" asked his father. "No," Junior admitted, "but he's different. He has very bright parents."
---Jacob M. Braude, Unknown
 
Whether an examination is of the ‘objective’ type involving multiple choices or of the essay type, it can be devised so as to emphasize an understanding of the broad principles of a subject.  Indeed, even when one examines on detailed knowledge, it can be done in such a way as to require an understanding by the student of the connectedness between specific facts.      ---Jerome Bruner (The Process of Education, pp. 30-31)

The words assessment and evaluation are often used interchangeably.   In our discussions, the term ‘assessment’ will refer to students and the term ‘evaluation’ will refer to programmes, courses, classes, etc.   

Top

 Specific Outcomes

Student assessment is the heartbeat of the curriculum.  It shapes and guides learning and helps to assure society that graduates are capable and competent.  Similarly, evaluation of curriculum, programmes and courses is essential in order to make informed decisions about the distribution and use of educational resources, to sustain the quality of existing programmes and to introduce changes that enhance the probability of achieving desired outcomes.

At the end of the Course, participants will be more informed assessors and evaluators as demonstrated by their ability to:

1. Describe the criteria for and benefits and goals of effective student assessment
2. Compare and contrast the benefits and limitations of different assessment methods
3. Modify existing assessment practices to incorporate principles of effective assessment
4. Match assessments with expected learning outcomes, objectives, educational approaches and methods
5. Increase the use of formative assessment in the teaching/learning process
6. Review, modify and develop ways to evaluate presentations, courses, units, modules, curricula 

Top

 Outline of Topics:  Informed Assessor/Evaluator

• Principles of Assessment
  General considerations
  Formative/Summative
  Criterion/Norm referenced
  Validity & reliability
  Standard setting
  Progress testing
• Examiner’s Toolkit
  Written assessment
  Oral assessment
  Performance assessment
  Portfolios
  Assessment of non-cognitive factors     
• Evaluation: Class/Course/Module/Unit/ Program
  Basic concepts in evaluation

Top

 Case Studies

Case Study 1: “Back From the Conference”

“Hi Dimitri, how was your trip and the meeting?”, beamed Anita.  “It was very interesting and educational.  I learned a lot and I want to share it with you and the others in our Department.  I think we should try to do some of the things I heard about and saw at the meeting like formative assessment, progress testing, etc.” replied Dimitri.  Anita and Dimitri were faculty members in the Department of Physiology and co-taught in the Human Physiology course during the second year of the curriculum at The Central European Higher Institute for Medical Education.  Dimitri had been teaching for 15 years but this was the first time he had been able to attend an international meeting about medical education.  Anita had been at the Institute for four years.  “I attended several presentations about test construction, using multiple methods to assess outcomes and how to write quality multiple choice questions.  I think we should re-evaluate our tests and try to use more questions that require students to think and understand the material and decrease the number of questions that require pure recall.”  “Sounds interesting, but won’t it be a lot of work,” asked Anita?  “I’m not sure, I don’t think so.  Lets talk about it some more and include the rest of the Department.  What do you think?” “OK, perhaps you could make a short presentation about what you learned at the meeting with suggestions about what we might consider doing,” replied Anita. 

Later that day, after discussing this with the chairman of his department, Dimitri and three other teachers, with some help from the Department of Medical Education, were charged to re-examine the match between expected outcomes of the department’s teaching and its relationship to the curriculum.  They decided that they needed to review their teaching, learning and assessing of the sciences basic medicine in their department over several years of the curriculum.  The Chairman asked them to report back to the committee in 4 months with some recommendations to be considered by the teachers in the Department.  You have been appointed to this committed with Dimitri. 

It is now 5 months later.  Your group is meeting to review its work and prepare its recommendations. 

Working in small groups, review the outcomes and competencies you drafted this morning (Framing medical education; Relevance, context and outcomes; Fink’s significant learning, SMART objectives).  You have two basic tasks:

1. Indicate how you will meet the criteria for effective assessment
2. Review, choose and match assessment methods from the tool box and other sources to your outcomes and competencies.
3. Prepare and present a brief summary on a flip chart.
   

What does Dimitri need to do now?

 Case Study 2: “How do we know they can do it?”

Arnaldo, the chair of the medical school curriculum committee, opens the meeting by saying, “My colleagues, it’s pretty clear what needs to be done.” Holding up a 35 page document he continued, “This is the most recent study of our national health system.  I know you have all seen it.  We need to focus our attention on student assessment during the clinical years.  Especially we need to make sure that we assessing students across the full spectrum of the necessary outcomes in each of the clinical departments and overall.”  Alan, from the Department of Pediatrics spoke up, “We all know our graduates are fine and we are too busy with patient care to devote more time to anything else.”  Elaine, chairwoman of the Department of Pathology and Laboratory Medicine countered, “I agree with Arnaldo, we have to take this seriously and look at what is currently available and what we can do.  Remember, next year we have our annual accreditation and I think they will want to see how we are assessing students.”

After several more comments, it was decided that a working group of clinicians, with some help from the Department of Medical Education would re-examine the match between expected outcomes of the curriculum and the teaching, learning and assessing of clinical skills and clinical performance and report back to the committee in 6 months with recommendations.  You have been appointed to this committed. 

It is now 5 months later.  Your group is meeting to review its work and prepare its recommendations. 

Working in small groups, review the outcomes and competencies you drafted this morning (Framing medical education; Relevance, context and outcomes; Fink’s significant learning, SMART objectives).  You have two basic tasks:

1. Indicate how you will meet the criteria for effective assessment
2. Review, choose and match assessment methods from the tool box and other sources to your outcomes and competencies.
3. Prepare and present a brief summary on a flip chart.
   

Top
 

Study Guide: Informed Assessor/Evaluator

Principles of Assessment

2.0            General Considerations

• What guidelines were used to assure effective student assessment.
• To what extent were different levels of knowledge assessed?
• To what extent were written learning outcomes and objectives available to students?
• To what extent were examination questions representative of specified content and learning outcomes?

2.10          Formative/Summative

• Describe how formative assessment was used as part of the learning process?
• What was the relative balance between formative and summative assessment?
  

2.20          Criterion/Norm Referenced

• How were results of student performance interpreted?
• What were the consequential outcomes of the assessment used?

2.30          Validity & Reliability

• What criteria were used to assure valid and reliable interpretations of assessment results?
• What forms of validity were explored and used?

2.40          Standard Setting

• What was the form of “Gold Standard” (bench mark) against which assessments were compared? Explain why it was considered a gold standard?
• Which method(s) for standard setting was used?
•  How were standards defined, developed, and implemented?

 
Examiner’s Toolkit

2.50          Written Assessment

• What was the relative balance between selection-type (multiple choice, true-false, matching, interpretive exercises) and construction or supply-type questions (short answer, short essay, long essay, modified essay)?
• To what extent were questions related to learning outcomes and objectives?
• Were questions reviewed for discrimination and difficulty?
• How, if at all, was written assessment applied to small-group, self-directed, case-based learning     

2.60          Oral Assessment

• What criteria were used for oral assessment of students?
• How were criteria linked to expected learning outcomes?
• How were patients, simulations, or other situations selected for inclusion in oral assessment?
• How, if at all, was oral assessment applied to small-group, self-directed, case-based learning
•  How was formative oral assessment incorporated into learning?

2.70          Performance Assesment

• How was performance assessment matched to desired outcomes?
• What student performance was deemed acceptable as evidence of learning?
• How were assessment methods and instruments selected?
• To what extent did assessment include direct observation of authentic tasks and work?
• What methods of observing, recording and scoring were used (checklists,rating scales, global, direct observation, anecdotal records)?

2.80          Portfolios

• To what extent did both teacher and students have an understanding of the use of a portfolio for development (formative), and/or assessment (summative)?
• Did learners have a guide to help them begin their portfolio?
• Were criteria for acceptable entries in the portfolio discussed with the students?
• How was progress in learning evaluated?
• Was sufficient time allotted for reviewing portfolios?

2.90          Assessment of Non-Cognitive Factors

• How were communication, professionalism, attitudes, empathy and other non-cognitive factors assessed?
• To what extent were self and peer assessment part of non-cognitive a  assessments?
  What was assessed?
• How were performance criteria and standards for acceptable performance identified and communicated to learners?
• How were results reported?
• Describe strategies for remediation (re-training) if necessary?

Evaluation: Class/Course/Module/Unit/ Program

2.100        Basic Concepts in Evaluation

• To what extent was evaluation formative and summative?
• To what extent and how was evaluation linked to educational outcomes?
• What were the organizing concepts and specific steps used for evaluation?
• How were anonymity and confidentiality of students protected?
• What qualitative & quantitative methods were used for the evaluation?
• How was information reported and shared?
• What were the consequential outcomes of the evaluation?   

Top

 Resource Material

Schuwirth, L.W.T., Van der Vleuten, C, (2004) Changing education, changing
assessment, changing research? Medical Education 38: 805-812.

http://hsc.unm.edu/SOM/TED  Multiple teaching resources for all types of teaching from the Office of Teacher and Educational Development, University of New Mexico School of Medicine, Albuquerque, New Mexico.

Recommended

Boulet, J.R., McKinley, D.W., Whelan, G.P., Hambleton, R.K. (2003) Quality
Assurance Methods for Performance-Based Assessments.  Advances in Health Sciences Education 8:27-47.

Hays, R B H., Davies, A.,  Beard, J.D., Caldon, L.J.M., Farmer, E.A. Finucane, P M., McCrorie, P., Newble, D.I., Schuwirth, L.W.T., Sibbald, G.R. (2002). Selecting performance assessment methods for experienced physicians.  Medical Education 36:910–917

Shepard, L.A. (2000) The role of assessment in learning culture.  Educational Researcher, 29 (7): 4-14.

Schuwirth, L.W.T., Van der Vleuten, (2006). A plea for new psychometric models in
educational assessment.  Medical Education 40: 296–300

Top