The Ultimate ‘Outcome’

January 17, 2010

We often use the example of assessing driving skills in faculty development workshops.

We like the example because most of us feel that we know a good driver when we see one (and that we’re good drivers!), but it is an immediately difficult skill to assess. Try developing a rubric for driving skills…

There are also lots of regional peculiarities that help personalize the conversation. Examples:

  • Jug handles in New Jersey
  • Ice/snow driving in Maine
  • Requirement in Connecticut to verify that you are not a child molester when purchasing/operating a van (!)

In one such workshop, a teacher made a terrific assessment insight: they argued that observing how a driver acted at a suburban stop sign would tell you a lot about their driving skill and attitude. Do they:

  • Screech to a halt?
  • Roll through the stop sign?
  • Ignore the “if two cars arrive at the same time to a 4-way stop, the car to your right goes first” rule?
  • Let the car’s momentum end, then accelerate away after looking both ways?

This ‘outcome’ trumps a lot of minute detail that ‘experts’ try to build into such assessments (remember, everyone thinks they know how to drive, so their rubrics are immediately complex).

Last night I saw the documentary film In A Dream, which contains an even higher level form of outcome: if you were trying to measure parenting skill, imagine what would become evident if an adult child of the parent created a documentary. And captured on film the father abandoning the mother for another woman. After 40 years of marriage.

It’s a great documentary, available on DVD. And how more subjective can you get than parenting over a lifetime? The results require thinking and analysis on the ‘assessors’ part – obviously any filmmaker will have an agenda, a viewpoint, and edit footage for certain effect. But the film feels like an authentic, balanced portrait and is intensely moving.

Assessing a presentation, or a collage, or a term paper should seem fairly simple in comparison. So rather than worry about multiple criteria measured nine different ways, think about what leading indicators are appropriate for your students and the task at hand.


Outcomes Assessment and Grades

February 16, 2009

A couple of recent articles in Inside Higher Ed caught our eye – one on grades and grade inflation, and the other on the creation of the National Institute for Learning Outcomes Assessment.

It seems obvious to us that grading and assessment are largely the same thing. Barring sampling programs, or initiatives designed to assess program outcomes (aggregating student results rather than considering the success of individuals), grading IS assessment.

It’s just that the typical grade (A-, B+ etc.) is an extraordinarily blunt instrument.

Imagine reading a car review (okay, bad example – who is reading car reviews anymore?) or a film review that is simply a letter grade. Many reviews feature letter grades, but they come after a thousand words of measured criticism. And it is subjective criticism, but we largely accept the skill of a Roger Ebert and take their points seriously. They are assessing the film, and they do it through a narrative response built upon well-established criteria.

Education is even messier than film reviewing, because the letter grades awarded are all over the place. To draw the analogy out a little farther, imagine trying to pick a movie to see from the following:

  • 20 films, all rated B+ or higher (with no narrative or other information)
  • 20 films, each rated four times by separate reviewers, where the individual grades are all over the map but the averages are still B+ or higher

You wouldn’t know which film to see…and likewise our system of letter grades is useless for assessing knowledge. Read the rest of this entry »


Annotate for MAC Users

January 26, 2009

Thanks to some very excited English teachers and a constant stream of emails from interested Mac users, we now have a version of Annotate for Word that runs on Mac Office 2004.

Yes, that was five years ago, but progress is progress.

With the latest release of Mac Office (2008), Microsoft dropped support for Visual Basic for Applications, which is what we wrote the older version of Annotate with (the Word 2007 version is written using .NET). But just as with Word 2007 on the Windows side of the world, many teachers haven’t bothered upgrading. So there are still a lot of Word 2004 computers out there, and we hope to help a lot of those computers help their owners create better feedback for their students.

The free version of Annotate for Word 2004 for Mac Office 2004 (we haven’t figured out a more elegant way to name the thing that is also specific enough) isn’t quite ready, but we’ve got a number of PRO users going already. So don’t hesitate to be in touch…

http://www.11trees.com/annotate-for-word.html


Annotating Student Work with Microsoft Word

June 13, 2008

UPDATE: Annotate for Word is now a commercial product (as of September 2008). You can learn more about the free and PRO versions of Annotate for Word by visiting www.11trees.com.

We’ve made considerable progress designing Annotate – an add-in for Microsoft Word. Here are some recent screenshots.

Annotate Ribbon for Word 2007: Citation Drop Down Showing
(click image for a larger version)

The above screenshot shows the different categories of comments available in Annotate: Commenting, Argument, Style, Organization, Citations, and Mechanics. Because citation issues can be so detailed, there is a drop down list with multiple choices. To insert a pre-written comment, the instructor simply highlights text, then clicks on a button in the Annotate Ribbon (or selects a more specific item from a drop down).

Highlight Text

After clicking or choosing, a comment like the one below appears:


(click image for a larger image)

In the above example, the “Place Citations” button was clicked, and a simple Post-It note appeared in the text. Notice that the student’s work (the selected text above) has been pulled out, turned purple (a nice neutral color), and the note is automatically inserted. This note takes advantage of a including some additional Advice and a Reference to Purdue’s fantastic OWL site.