Outcomes Assessment and Grades

A couple of recent articles in Inside Higher Ed caught our eye – one on grades and grade inflation, and the other on the creation of the National Institute for Learning Outcomes Assessment.

It seems obvious to us that grading and assessment are largely the same thing. Barring sampling programs, or initiatives designed to assess program outcomes (aggregating student results rather than considering the success of individuals), grading IS assessment.

It’s just that the typical grade (A-, B+ etc.) is an extraordinarily blunt instrument.

Imagine reading a car review (okay, bad example – who is reading car reviews anymore?) or a film review that is simply a letter grade. Many reviews feature letter grades, but they come after a thousand words of measured criticism. And it is subjective criticism, but we largely accept the skill of a Roger Ebert and take their points seriously. They are assessing the film, and they do it through a narrative response built upon well-established criteria.

Education is even messier than film reviewing, because the letter grades awarded are all over the place. To draw the analogy out a little farther, imagine trying to pick a movie to see from the following:

• 20 films, all rated B+ or higher (with no narrative or other information)
• 20 films, each rated four times by separate reviewers, where the individual grades are all over the map but the averages are still B+ or higher

You wouldn’t know which film to see…and likewise our system of letter grades is useless for assessing knowledge.

Annotate for Word 2003!!!

Feedback is central to a teacher/student relationship, and the more we experiment with existing feedback tools the more we feel that e-learning technology company’s have forsaken this crucial process for more glitzy tools – like streaming media drop boxes and VoIP solutions. But does a teacher need a wi-fi Skype phone in her classroom? Or a better way to give constructive feedback to her students?

As a small step towards remedying, I have been involved with a project called 11trees.com. 11trees.com’s first tool – available in free and PRO versions – is a customized add-in for Microsoft Word that makes it easy to add many of the typical comments teachers make to students on their writing (at least in high school and college) to a Microsoft Word document.

So many of us are collecting work electronically, or reviewing drafts that our students email to us, that it just makes sense to have a more legible, efficient tool to leverage Microsoft Word.

We released a version for Word 2007 (isn’t it almost 2009?) and many teachers and administrators have found us through Amazon.com, this blog, and kind mentions in a number of different publications and blogs. But we also heard from a large number of teachers begging us to build a version for the Mac or Word 2003.

Apple is whole other kettle of fish, unfortunately, although we are exploring a Mac Office 2008 version of Annotate. But Word 2003 was a no-brainer, since so many institutions and individuals have resisted – often for good reasons – upgrading.

Annotate for Word 2003 Toolbar

The Ribbon interface that is new to Word 2007 is the first major change to Word in many, many years. Most users find it daunting at first (where did my commands go?), but quickly get used to the new arrangement. Because features are grouped by their larger function, Word 2007 can be considerably easier to use than previous versions. The Annotate Ribbon integrates seamlessly with the Word 2007 user interface, and we recommend considering upgrading if you have the opportunity (many schools offer upgrades to teachers for free).

That said, Word 2003 works just fine, most people already have it. So we worked hard through the first couple of weeks of October to develop a Word 2003 version, which in some ways is superior to the Word 2007 version (hot-keys, for one reason).

So check out Annotate for Word 2003!

Annotate for Word 2003 Drop-down menu

Comment Inserted with ONE Click

Effective Peer Review Practices: Leveraging the LMS

The folks at Waypoint Outcomes just posted a white paper that discusses best practices in designing peer review, particularly in blended (hybrid) or purely online teaching modalities.

The approach described is LMS-agnostic, and only mentions Waypoint via Appendices to highlight how Waypoint improves upon many of the issues/opportunities highlighted.

>> Download the PDF version of the white paper

We are very interested in feedback – what strategies have you found to be effective?

Read an excerpt:

Introduction

Peer review is a widely accepted practice, particularly in writing classes, from high school through college and graduate school. The goal of peer review is typically two-fold:

1)      To help students get valuable feedback at the draft stage of their work.

2)      To help students more deeply understand the goals of the assignment.

Unfortunately, peer review is often used as a busy-work activity, or a process that takes advantage of conscientious students while allowing others to do superficial work. For instance, many teachers will hand out a list of peer review questions in class, and then give students 30 minutes to review two papers written by their colleagues. An open-ended question might be:

• “Did the writer adequately summarize and discuss the topic? Explain.”

Many students will write “Yes” under this question and move on. Without review by the instructor (difficult to do when many instructors have 50 to 150 students), these students can destroy the social contract of a peer review. Other students will spend a lot of time making line edits to the draft – correcting grammar, making minor changes to sentences etc. At the draft stage this is probably inappropriate – the focus should be on ideas and big-picture organization, not embroidery. Plus, some students aren’t qualified to be dictating where the semicolon should go.

“Peer review can be some of the most important writing students ever do – because they have a real audience for their work.”

Dr. Scott Warnock
Drexel University

Students aren’t alone in having these problems; in 1982, Nancy Sommers published her highly influential piece, “Responding to Student Writing,” in which she commented about how little teachers understand the value of their commenting practices, and that, essentially, they don’t know what their comments do. She raised numerous long-standing points in her evaluation of teachers’ first and second draft comments on papers. Two of her major findings:

1)      Teachers provide paradoxical comments that lead students to focus more on “what teachers commanded them to do than on what they are trying to say” (151).

2)      She found “most teachers’ comments are not text-specific and could be interchanged, rubber-stamped, from text to text” (152). One result is that revising, for students, becomes a “guessing game” (153). Sommers concluded by saying, “The challenge we face as teachers is to develop comments which will provide an inherent reason for students to revise” (156).

Most teachers have experienced this last point when a student asks, “what do you want.” The student doesn’t understand the larger goal of the assignment and has learned that achievement comes through figuring out the personal foibles of their current teacher.

These outcomes are unfortunate, because peer review (and written feedback from teachers) can be one of the most powerful learning experiences for students.

From our perspective, peer review should:

1)      Help the students improve their work through the drafting process.

2)      Deepen the understanding of the assignment and its goals for both authors and reviewers (and teachers!).

3)      Allow instructors to assign more authentic work without requiring they read and grade piles of papers – so they do more coaching than grading.

4)      Give students opportunities to create authentic work – that is, peer reviews written (or spoken) to a real audience: the author. Peer review, in the words of Drexel University’s Dr. Scott Warnock, “can be some of the most important writing students ever do – because they have a real audience for their work.”

Read the rest in the PDF

Assessing Critical Thinking

Many of our users know about the Waypoint Public Library – a shared library of both Assignments and Elements created by our clients. Each month we’ll highlight a unique approach to assessment and feedback and make it easy for you to copy and utilize it.

As a first installment we thought we’d start with a double-shot of critical thinking, a crucial skill difficult to assess and of interest to educators from middle school through graduate school. These Assignments don’t formally address “critical thinking” as a skill, but seek to differentiate summarizing facts from making original connections while synthesizing information.

There are two versions of this Waypoint Assignment: one intended for peer review (pdf), and the other for an instructor (pdf) to use. Specific references (to writing handbooks etc.) have been removed. You can easily copy these Assignments from the Public Library and edit them to suit. The two Assignments are:

• Critical Thinking/Writing (SubMet Library) 7/10/08
• Critical Thinking/Writing Peer Review (SubMet Library) 5/30/08

They both make use of Checklists, but you’ll notice that the first few Observations in the instructor versions have traditional ‘rubric’ choices. So the detailed Observations could be easily dropped and the Checklist Element converted to a Performance Element.

>> Read more about copying an Assignment from the Public LIbrary
>> See the detailed version of the instructor Assignment
>> See the detailed version of the peer review Assignment