As we long suspected:

“At the most celebrated institutions of higher education in the United States, the teaching quality of the adjuncts is many times better than that of those on the tenure tack.”

Inside Higher Ed didn’t pull any punches in their review of Off-Track Profs: Nontenured Teachers in Higher Education from the MIT Press. The book, just out, reports results on a study funded by the Mellon Foundation that looked at adjuncts teaching at 10 leading research institutions. The authors, university administrators themselves, had seemingly total access to data and personnel.

As important as the finding about ‘quality teaching’ (more on this in a moment), is the study’s analysis of the drivers behind the growth of adjuncts. It isn’t just the cynical need to save money. In fact, the decision to place a course with an adjunct results from many factors. This makes sense, since few university administrators have any sort of cost-cutting philosophy to their leadership. If they did, many aspects of university departments would change before an increased hiring in adjuncts.

As usual, the definition of ‘better teacher’ is based entirely on course evaluations completed by students. Which means the results are worthless. Anti-adjunct (or anti-data) partisans will reject the findings out of hand. And they’d be correct to do so, although the conclusion feels right to us. It would have been interesting to see the authors of the study bounce their data against RateMyProfessors.com data…but that’s another story.

The whole illogical mess is another reflection on the emotional and cultural decision making that drove the financial melt-down. Is the goal to educate students? Or is the goal to bring in research $ and publish obscure texts (the article casually mentions that the course load for “many tenured professors has fallen from four to three a year.” THREE COURSES A YEAR?

Is the goal to sell as many mortgages as possible? Or to make sensible loans that will actually be paid back?

The whole dynamic runs very close to a terrific new piece from Atul Gawande in The New Yorker this week. We’ve blogged about Gawande before; he is taking on the 30,000 foot issues in medicine with an eye for detail and counter intuitive conclusions that are obvious once pointed out. We feel similar work should be (and could be) done in education.

Gawande visits the US county with the highest Medicare costs in the country and asks, “what is different about this place”? The answers are obvious and challenging at the same time. The opening shot, devastating to anyone who isn’t in medicine, is the immediate realization that no one in leadership positions at the hospitals and health care providers he visits knows how much money is being consumed in their facilities. That is, lowering cost is not on their radar screen. The equivalent would be UPS not bothering to know how many packages were moving through its system on a daily basis. The health care executives don’t know because they don’t care; total cost – at the individual patient level – isn’t a concern.

There are some bright spots – health care systems like the Mayo Clinic where physicians are paid salaries, rather than “incentivized to provide services.” These systems spend less money on people – and the result is a healthier population. There is a lot of data behind this conclusion, but basically the more you visit hospitals and doctors, overall, the sicker you get. If your doctor orders three follow up tests, a ‘minor’ preventative surgery, and puts you on drug your outcome is probably going to be poorer than finding more creative ways to deal with the situation.

Unlike healthcare, educators (at the world’s leading institutions!) are basing decisions on course evaluations. Is this identical to deciding on the effecitveness of a hospital by asking a the (surviving) patients how they feel? Just about. The information is useful, but the data cannot be used to drive decision making.

And as in healthcare, the stakes are the same: education costs America huge sums of money, much of it to ‘incentivize’ pseudo-academics to invent ever more elaborate systems of language and criticism that keeps their obscure world afloat. They are no different than the doctors in Texas that enjoy a particularly entrepreneurial culture, and charge the shit out of the Medicare system to spin up their own revenues.

Unlike healthcare, America as a whole does not need to come to some grand conclusion and force change. Education (especially higher education) is an increasingly market-driven business, and efficient providers are wading in…within 10 years the luster of a non-profit, ‘prestigous’ degree while have completely worn away (a generation of graduates with huge bills and no jobs will accelerate this process even more) and 18 year olds will become smart consumers. Universities, as we know them now, are the equivalent of the ice harvesting companies of the early 20th century; they are worried about course evaluations like the ice harvesters designed more effective insulation to move ice long distances. When refrigeration became possible in the home, new companies took over the entire ‘ice’ market.

A couple of recent articles in Inside Higher Ed caught our eye – one on grades and grade inflation, and the other on the creation of the National Institute for Learning Outcomes Assessment.

It seems obvious to us that grading and assessment are largely the same thing. Barring sampling programs, or initiatives designed to assess program outcomes (aggregating student results rather than considering the success of individuals), grading IS assessment.

It’s just that the typical grade (A-, B+ etc.) is an extraordinarily blunt instrument.

Imagine reading a car review (okay, bad example – who is reading car reviews anymore?) or a film review that is simply a letter grade. Many reviews feature letter grades, but they come after a thousand words of measured criticism. And it is subjective criticism, but we largely accept the skill of a Roger Ebert and take their points seriously. They are assessing the film, and they do it through a narrative response built upon well-established criteria.

Education is even messier than film reviewing, because the letter grades awarded are all over the place. To draw the analogy out a little farther, imagine trying to pick a movie to see from the following:

• 20 films, all rated B+ or higher (with no narrative or other information)
• 20 films, each rated four times by separate reviewers, where the individual grades are all over the map but the averages are still B+ or higher

You wouldn’t know which film to see…and likewise our system of letter grades is useless for assessing knowledge.

Think back to your own college days. What did grades really say about the person? How well they had learned to play the game? How hard they worked? How much talent and mental horsepower they had? Probably all of these. And for this reason grades are a somewhat useful gauge for companies looking to hire college graduates. But they are no measure of learning, even with the amount of effort that goes into creating them.

One of my earliest engineering jobs was an internship at Unisys. I went to lunch with a group of managers who were discussing the internship program. An HR manager brought up the idea of raising the minimum GPA required to a 3.5/4.0 scale. The managers all looked at each other, and to their great credit laughed, and then (to a person) said that if those standards were applied they wouldn’t have earned their initial job at Unisys.

This may be a commentary on Unisys (which didn’t invent the iPod, eBay, Facebook, Google, or the Blackberry), but I don’t think so. It’s the understanding in industry that there is more to learning than a GPA.

Thanks to some very excited English teachers and a constant stream of emails from interested Mac users, we now have a version of Annotate for Word that runs on Mac Office 2004.

Yes, that was five years ago, but progress is progress.

With the latest release of Mac Office (2008), Microsoft dropped support for Visual Basic for Applications, which is what we wrote the older version of Annotate with (the Word 2007 version is written using .NET). But just as with Word 2007 on the Windows side of the world, many teachers haven’t bothered upgrading. So there are still a lot of Word 2004 computers out there, and we hope to help a lot of those computers help their owners create better feedback for their students.

The free version of Annotate for Word 2004 for Mac Office 2004 (we haven’t figured out a more elegant way to name the thing that is also specific enough) isn’t quite ready, but we’ve got a number of PRO users going already. So don’t hesitate to be in touch…

http://www.11trees.com/annotate-for-word.html

Introduction

Peer review is a widely accepted practice, particularly in writing classes, from high school through college and graduate school. The goal of peer review is typically two-fold:

1. To help students get valuable feedback at the draft stage of their work.
2. To help students more deeply understand the goals of the assignment.

Unfortunately, peer review is often used as a busy-work activity, or a process that takes advantage of conscientious students while allowing others to do superficial work. For instance, many teachers will hand out a list of peer review questions in class, and then give students 30 minutes to review two papers written by their colleagues. An open-ended question might be:

• “Did the writer adequately summarize and discuss the topic? Explain.”

Many students will write “Yes” under this question and move on. Without review by the instructor (difficult to do when many instructors have 50 to 150 students), these students can destroy the social contract of a peer review. Other students will spend a lot of time making line edits to the draft – correcting grammar, making minor changes to sentences etc. At the draft stage this is probably inappropriate – the focus should be on ideas and big-picture organization, not embroidery. Plus, some students aren’t qualified to be dictating where the semicolon should go.

Students aren’t alone in having these problems; in 1982, Nancy Sommers published her highly influential piece, “Responding to Student Writing,” in which she commented about how little teachers understand the value of their commenting practices, and that, essentially, they don’t know what their comments do. She raised numerous long-standing points in her evaluation of teachers’ first and second draft comments on papers.

Two of her major findings:

1. Teachers provide paradoxical comments that lead students to focus more on “what teachers commanded them to do than on what they are trying to say” (151).
2. She found “most teachers’ comments are not text-specific and could be interchanged, rubber-stamped, from text to text” (152). One result is that revising, for students, becomes a “guessing game” (153). Sommers concluded by saying, “The challenge we face as teachers is to develop comments which will provide an inherent reason for students to revise” (156).

Most teachers have experienced this last point when a student asks, “what do you want.” The student doesn’t understand the larger goal of the assignment and has learned that achievement comes through figuring out the personal foibles of their current teacher.

These outcomes are unfortunate, because peer review (and written feedback from teachers) can be one of the most powerful learning experiences for students.

From our perspective, peer review should:

1. Help the students improve their work through the drafting process.
2. Deepen the understanding of the assignment and its goals for both authors and reviewers (and teachers!).
3. Allow instructors to assign more authentic work without requiring they read and grade piles of papers – so they do more coaching than grading.
4. Give students opportunities to create authentic work – that is, peer reviews written (or spoken) to a real audience: the author. Peer review, in the words of Drexel University’s Dr. Scott Warnock, “can be some of the most important writing students ever do – because they have a real audience for their work.”

Peer Review and the LMS

Before considering pedagogy, the first challenge to peer review is largely one of workflow. How to effectively set up teams of reviewers? How to manage student work? How to enforce deadlines and reward students who are conscientious?

Learning Management Systems like Blackboard, Blackboard Vista/CE6, and Moodle can streamline the peer review process.

Through much trial and error and philosophizing, along with combining best practices from many educators, from middle-school through graduate school, we propose a process to minimize the load on the teacher and maximize the benefits of peer review. This process leverages the best of the Learning Management Systems to automate much of the process, leaving the teacher to focus on value-added activities far more useful than busywork.

PEER REVIEW PROCESS

1)    DEVELOP THE ASSIGNMENT

Students need to do something. Without getting into curriculum design debates, we will assume that ‘something’ is fairly clearly defined. Let’s also assume that there are defined deadlines for work. Here’s a suggested timeline for a typical 4 to 5 page paper:
Drafts: Tuesday, 9/22  by 11:59pm
Peer reviews: Friday, 9/26 by 11:59pm
Final Drafts: Tuesday, 9/29 by 11:59pm

One of the immediately valuable aspects to peer review is its ability to combat procrastination. Too often we assign projects, give a deadline four or five weeks out, and expect students to perform. No manager would assign a major project to a new employee and then check back four weeks later (at least no effective manager). They would require status reports, check in informally from time to time etc. So by requiring some sort of draft a week before the final deadline, teachers can help students get started.

To help gain students’ attention, we highly recommend that the quality of peer reviews be worth 20% or more of the final Assignment grade (based largely on effort, not skill, since this can be difficult work for students).

We also recommend that students be required to include a cover letter (in the same document), addressed to their reviewers, that details their goals with the assignment and specifies the top two or three areas in which they would like feedback. This helps students reflect on their work and also give the peer reviewers a ‘heads up’ concerning specific issues.

2)    DESIGN A RUBRIC OR SET OF CRITERIA

This is the stage where things can begin to get challenging for teachers new to the idea. Too often the criteria are in our heads, or we don’t share them with students until well after the students have begun working on their projects. Luckily, the 80/20 rule applies to student work: 80% of the projects we assign, from middle school through graduate school, are covered by 20% of the potential criteria we might think of. So there are lots of models to borrow from.
The criterion listed below helps students decide how original an argument might be (useful from high school through college), breaks the issue down and uses language students can probably understand.

It could be easily modified for a particular audience – made either more sophisticated or simplified to highlight the differences between performance levels. You can find the entire rubric in Appendix A.

This rubric might be exactly the same for students and teachers, or teachers might use a slightly different version (with more direct language). The rubric should be posted along with the Assignment. Ideally, students would help design and refine the rubric.

It is important to use open-ended questions when designing peer reviews; this example gives just the declarative ‘observations’ that help the student distinguish between levels of performance. To see how open ended questions can help structure the review process, see Appendix C: Maximizing Peer Review with Waypoint Outcomes.

1. (Excellent) Wow, this is a highly insightful argument – you go far beyond our classroom discussion and readings.
2. (Good) Okay, you`ve definitely introduced an interesting (not predictable) argument..but it doesn`t seem thought through enough or lacks necessary evidence.
3. (Fair) This is bordering on a summary of information rather than an argument.
4. (Poor) There is hardly any insight here – it`s mostly summary of various points of view or conventional wisdom (that is, predictable).
5. (Unacceptable) ZERO argument. The author never presents fresh ideas or interpretation.

There are a number of pedagogical choices implied in the above sample criterion: whether to include specific performance descriptors (Good etc.), how sophisticated or informal to make the language etc. Each educator should make these decisions to suit their needs.

This rubric or criteria set should be made available to students in advance, to guide the development of their draft and to structure their response to the student.

3a)    COLLECT THE DRAFTS – VIA DISCUSSION

Many instructors will try to assign specific students to specific peer review groups. Certainly, if a teacher knows their students very well and can specify teams, then all the better.

However, creating and managing groups manually can create a pile of work for the instructor. What happens when a student emails to say they are sick, and they can’t post a draft? Does the instructor need to edit the peer review groups and move students around?

The trick is to reward the students who complete their work on time and not reward (notice the term isn’t necessarily punish) those who do not. Whether collected in a face-to-face class, or enforced with deadlines in the LMS, a certain percentage of students will not submit a draft on time, endangering the entire process or requiring instructor intervention to rebalance the work. In our experience, assigning exact ‘peer review partners’ is inefficient and unnecessary to achieve the stated goals of a peer review process.

Whether randomly created or designed, the Groups feature of most CMSs can help structure the who-reviews-whom question.
There are benefits to different strategies, but here’s an approach using Groups and Discussion Boards. Let’s assume that we want students to peer review two of their fellow students’ papers:

1. Create Groups and place six students in each (so a class/section with 24 students would have four Groups). This way, every student is guaranteed to get two reviews, even if one or two of their fellow students neglects to post a draft or neglects their work.
2. Create a Discussion Topic for each Group – so only the students in the Group have access to the discussion. Make sure the Discussion Topic will be unavailable after the deadline.
3. Require students to post an attachment and the plain text of their draft. This way, the instructor can quickly review drafts without opening attachments, while students can open fully formatted versions.
4. Students review the two drafts posted before (chronologically) their own. This way the process isn’t destroyed by students who do not post on time, and again we reward those who adhere to deadlines. Students who post first review the students who post last.
5. Put some teeth into the process (see Appendix B for sample instructions intended for students). We don’t recommend punishment, but rather the requirement that students actively do something – visit the writing center or similar.

Students must post a draft by the deadline to receive peer reviews. If they do not post their own work by the deadline, they still must review two other students’ work.

Obviously the particular solution will need to suit the teacher’s sensibilities, but the idea is to give the students who might miss the deadline more work to do. If they know this in advance, they are more likely to seek the easiest path – following instructions.

Peer reviewers can simply post a response to the author’s original discussion post, using the rubric (criteria) as a guide.

3b)    COLLECT THE DRAFTS – VIA ASSIGNMENT

Discussion Topics can be convenient for grouping students and segmenting the reviews; most LMS platforms will tell the instructor how many unread messages are posted to a. So in our example, there should be 6 posts the morning after the deadline. But often students will post twice (they forget to attach, post a newer version etc.). If an instructor has only one class, it may be possible to click into each discussion topic and see who has posted work. But this quickly becomes busy-work with multiple sections.

An alternative is to use the Assignment drop-box, which in most LMS platforms more clearly shows late submissions (or no submissions). The drawback is losing the small group approach, because students will be able to see ALL submissions. Again depending on instructor sensibility and philosophy, using the Assignment drop-box can be a much more efficient approach. Groups of students need to be documented along with the assignment specifications, so that students can search all submissions for the students they are assigned to review.

If the Assignment feature is used, it may be possible to allow students to type or upload comments, depending on the LMS. Or a separate Discussion topic can be setup for responses. The same guiding principle, of grouping students into sub-groups and requiring they review submissions made chronologically just before theirs can elegantly handle the inevitability of non-compliance with instructions.

4)    MANAGE THE PEER REVIEWS

A similar problem awaits the teacher trying to manage and review the comments students write for one another. One solution is simply to wait for the submission of final drafts, and direct students to include the peer reviews they received. Again, inevitably, some students will forget to attach their peer reviews, defeating the instructor who wants to evaluate the quality of peer reviews.

A much less desirable solution is for the teacher to click through potentially hundreds of pages in the LMS reading posts or comments.
Workarounds can be used: students might need to post their reviews to a separate Discussion Topic, so the instructor can more easily review compliance and quality.

5)    COLLECT FINAL DRAFTS

Increasingly instructors are collecting work electronically – for environmental reasons, convenience (no lugging piles of papers around), and to grade electronically. Whatever the process for collecting work, we suggest students be required to submit:

1. A cover letter, addressed to the instructor that:
   1. details their goals for the assignment
   2. discusses their progress from first draft to final
   3. reflects on feedback they have received earlier in their academic careers
   4. responds to the comments they received from the peer review process
2. The final draft, complete with a works cited page if required
3. First draft
4. Peer review(s)

The cover letter requirement helps students learn different rhetorical strategies (switching from a formal academic voice to a more business-like letter-writing voice), and can be a terrific leading indicator for instructors when it comes time to grade. Hastily written cover letters are often indicative of hastily written drafts. Good cover letters can help students maximize credit for the assignment because the instructor may better understand the subtleties of the work

MAXIMIZING PEER REVIEW

All suggestions to this point have assumed a generic Learning Management System with the typical features of Assignment Drop-boxes and Discussion Topics.
Some LMSs have simple evaluation tools, like ‘grade forms’ in Blackboard Vista/CE6, the Outcomes module in Moodle 1.9+, and the peer review feature in Blackboard 8.0+. These features have varying degrees of utility, and none of them are grounded in the latest composition research – or even the common sense to assume that there is more to evaluating complex work than simple Likert Scales.

Waypoint Outcomes, developed by Subjective Metrics and used by over 40 institutions in the US, Canada, and Europe, has focused on peer review from its earliest days. Whatever the approach taken by an instructor, Waypoint can structure students’ response, help the instructor coach rather than click, and vastly improve the peer review experience for all parties.

Waypoint Outcomes is a tool for building, sharing, and using sophisticated rubrics to create exceptional feedback. Waypoint is tightly integrated with Blackboard, Blackboard Vista/CE6, and Moodle.

Using Waypoint, and instructor develops a set of criteria, creates a peer review Project to grant access to the rubric to students, then manages the peer review process. Students access Waypoint via the LMS, click their way through a detailed rubric, annotating and explaining their choices as they go, then save their work and email the evaluation to the author of the work.

For more on Waypoint, visit http://www.gowaypoint.com

BIBLIOGRAPHY

Sommers, Nancy (May 1982). “Responding to Student Writing.” College Composition and Communication. 33.2: 148-156.

Conan, Neil. “Procrastination Nation.” NPR: Talk of the Nation 12/15/2005 13 Jun 2008 <http://www.npr.org/templates/story/story.php?storyId=5053416>.

Feedback is central to a teacher/student relationship, and the more we experiment with existing feedback tools the more we feel that e-learning technology company’s have forsaken this crucial process for more glitzy tools – like streaming media drop boxes and VoIP solutions. But does a teacher need a wi-fi Skype phone in her classroom? Or a better way to give constructive feedback to her students?

As a small step towards remedying, I have been involved with a project called 11trees.com. 11trees.com’s first tool – available in free and PRO versions – is a customized add-in for Microsoft Word that makes it easy to add many of the typical comments teachers make to students on their writing (at least in high school and college) to a Microsoft Word document.

So many of us are collecting work electronically, or reviewing drafts that our students email to us, that it just makes sense to have a more legible, efficient tool to leverage Microsoft Word.

We released a version for Word 2007 (isn’t it almost 2009?) and many teachers and administrators have found us through Amazon.com, this blog, and kind mentions in a number of different publications and blogs. But we also heard from a large number of teachers begging us to build a version for the Mac or Word 2003.

Apple is whole other kettle of fish, unfortunately, although we are exploring a Mac Office 2008 version of Annotate. But Word 2003 was a no-brainer, since so many institutions and individuals have resisted – often for good reasons – upgrading.

Annotate for Word 2003 Toolbar

The Ribbon interface that is new to Word 2007 is the first major change to Word in many, many years. Most users find it daunting at first (where did my commands go?), but quickly get used to the new arrangement. Because features are grouped by their larger function, Word 2007 can be considerably easier to use than previous versions. The Annotate Ribbon integrates seamlessly with the Word 2007 user interface, and we recommend considering upgrading if you have the opportunity (many schools offer upgrades to teachers for free).

That said, Word 2003 works just fine, most people already have it. So we worked hard through the first couple of weeks of October to develop a Word 2003 version, which in some ways is superior to the Word 2007 version (hot-keys, for one reason).

So check out Annotate for Word 2003!

Annotate for Word 2003 Drop-down menu

Comment Inserted with ONE Click

Like a lot of bloggers and teachers interested in technology and education, I’m a geek. It makes perfect sense to me to do just about everything I can with technology. I’m very fast with a computer, and while my physical desk might be a pile of (seemingly) disorganized papers, my computer is always immaculately organized.

Okay, my inbox gets a bit crazed, but that’s why Xobni invented Xobni.

So I’ve been collecting student work in electronic format for a long time – mostly because it seems so inelegant to walk around with 80 student papers (and a pain when commuting on my bike) when I could have each students’ work stored neatly in a folder on my hard drive. I always have access to my comments and student work, can collect the work in the first place via Blackboard, and don’t have to worry about misplacing student work etc. etc.

I remember the moment that I realized teachers might have another reason for collecting work electronically. I was talking to a technology guru at Carleton College, and asking about workflow and teacher adoption of their elearning platform, Moodle. I have this conversation a lot, and it usually highlights one of the the dirty secrets in higher education: low utilization rate for elearning tools. There are budgets, entire staffs dedicated to technology, but few faculty use tools like Moodle or Blackboard to any great extent. Maybe they post a few files, an announcement or two… I have visited (prestigious) schools where full-time “instructional technologists” will actually scan a professor’s hardcopy syllabus and place the PDF into the appropriate Blackboard course. And that single document is the only resource in the course. For an entire term.

Of course there are the power users – geeks like me, and large introductory classes that make use of online testing (and automatic grading).

But the technology guru at Carleton surprised me. She told me that most faculty collect work electronically and review it on their computers. I was surprised, and asked why. She told me that the faculty had changed their workflow for environmental reasons. They changed their ways to save paper.

I was not surprised, then, to discover that Carleton College was rated in the top ten of over 300 colleges and universities ranked in the 2009 College Sustainability Report Card, published by the nonprofit Sustainable Endowment Institute. The report card covers many areas of an institution’s operations, and the endowment (what the school invests in etc.) and commitment to sustainability (evidenced by an ‘office of sustainability’ and a full-time person directing said office) are major components.

A teacher’s actions, however, are far more visible to students than complex and long-term investments. What struck me the most about the news coverage of the report card (the media loves a ranked list!) had to do with high school students’ attitudes: “Sixty-three percent of 10,300 college applicants recently polled by the Princeton Review said that a college’s commitment to the environment could affect their decision.” Since the report card  didn’t touch on elearning or academic technology, I take the Carleton faculty’s commitment to be a cultural expression of the school’s larger commitment and an indicator that they earned their ‘A-.’

The folks at Waypoint Outcomes just posted a white paper that discusses best practices in designing peer review, particularly in blended (hybrid) or purely online teaching modalities.

The approach described is LMS-agnostic, and only mentions Waypoint via Appendices to highlight how Waypoint improves upon many of the issues/opportunities highlighted.

>> Download the PDF version of the white paper

We are very interested in feedback – what strategies have you found to be effective?

Read an excerpt:

Introduction

Peer review is a widely accepted practice, particularly in writing classes, from high school through college and graduate school. The goal of peer review is typically two-fold:

1)      To help students get valuable feedback at the draft stage of their work.

2)      To help students more deeply understand the goals of the assignment.

Unfortunately, peer review is often used as a busy-work activity, or a process that takes advantage of conscientious students while allowing others to do superficial work. For instance, many teachers will hand out a list of peer review questions in class, and then give students 30 minutes to review two papers written by their colleagues. An open-ended question might be:

• “Did the writer adequately summarize and discuss the topic? Explain.”

Many students will write “Yes” under this question and move on. Without review by the instructor (difficult to do when many instructors have 50 to 150 students), these students can destroy the social contract of a peer review. Other students will spend a lot of time making line edits to the draft – correcting grammar, making minor changes to sentences etc. At the draft stage this is probably inappropriate – the focus should be on ideas and big-picture organization, not embroidery. Plus, some students aren’t qualified to be dictating where the semicolon should go.

“Peer review can be some of the most important writing students ever do – because they have a real audience for their work.”

Dr. Scott Warnock
Drexel University

Students aren’t alone in having these problems; in 1982, Nancy Sommers published her highly influential piece, “Responding to Student Writing,” in which she commented about how little teachers understand the value of their commenting practices, and that, essentially, they don’t know what their comments do. She raised numerous long-standing points in her evaluation of teachers’ first and second draft comments on papers. Two of her major findings:

1)      Teachers provide paradoxical comments that lead students to focus more on “what teachers commanded them to do than on what they are trying to say” (151).

2)      She found “most teachers’ comments are not text-specific and could be interchanged, rubber-stamped, from text to text” (152). One result is that revising, for students, becomes a “guessing game” (153). Sommers concluded by saying, “The challenge we face as teachers is to develop comments which will provide an inherent reason for students to revise” (156).

Most teachers have experienced this last point when a student asks, “what do you want.” The student doesn’t understand the larger goal of the assignment and has learned that achievement comes through figuring out the personal foibles of their current teacher.

These outcomes are unfortunate, because peer review (and written feedback from teachers) can be one of the most powerful learning experiences for students.

From our perspective, peer review should:

1)      Help the students improve their work through the drafting process.

2)      Deepen the understanding of the assignment and its goals for both authors and reviewers (and teachers!).

3)      Allow instructors to assign more authentic work without requiring they read and grade piles of papers – so they do more coaching than grading.

4)      Give students opportunities to create authentic work – that is, peer reviews written (or spoken) to a real audience: the author. Peer review, in the words of Drexel University’s Dr. Scott Warnock, “can be some of the most important writing students ever do – because they have a real audience for their work.”

Read the rest in the PDF

]]> http://authenticassessment.wordpress.com/2008/07/14/effective-peer-review-practices-leveraging-the-lms/feed/ 2 11trees http://authenticassessment.wordpress.com/2008/07/10/assessing-critical-thinking/ http://authenticassessment.wordpress.com/2008/07/10/assessing-critical-thinking/#comments Fri, 11 Jul 2008 02:02:19 +0000 11trees http://authenticassessment.wordpress.com/?p=95 ]]>

Many of our users know about the Waypoint Public Library – a shared library of both Assignments and Elements created by our clients. Each month we’ll highlight a unique approach to assessment and feedback and make it easy for you to copy and utilize it.

As a first installment we thought we’d start with a double-shot of critical thinking, a crucial skill difficult to assess and of interest to educators from middle school through graduate school. These Assignments don’t formally address “critical thinking” as a skill, but seek to differentiate summarizing facts from making original connections while synthesizing information.

There are two versions of this Waypoint Assignment: one intended for peer review (pdf), and the other for an instructor (pdf) to use. Specific references (to writing handbooks etc.) have been removed. You can easily copy these Assignments from the Public Library and edit them to suit. The two Assignments are:

• Critical Thinking/Writing (SubMet Library) 7/10/08
• Critical Thinking/Writing Peer Review (SubMet Library) 5/30/08

They both make use of Checklists, but you’ll notice that the first few Observations in the instructor versions have traditional ‘rubric’ choices. So the detailed Observations could be easily dropped and the Checklist Element converted to a Performance Element.

>> Read more about copying an Assignment from the Public LIbrary
>> See the detailed version of the instructor Assignment
>> See the detailed version of the peer review Assignment

We recently presented on a simple change to collecting work from students: ask them to include a cover letter, addressed to the instructor, with their submission of work. This cover letter should reflect upon the previous feedback they have received (from instructors and, most recently, their peer reviewers if applicable). It could also give the reader an overview of the goals of their work and specify areas in which they (the student) are most interested in receiving feedback.

We thought it useful to illustrate this process with examples.

The following example is from a college-level writing class where students were studying the effect of war on culture (and vice-versa). They were given the following assignment:

We will read some articles about Iraq, it’s effects, a history of horror movies, and a detailed account of the US involvement in Somalia.

Project 1 is an academic paper: formal diction, MLA citation formatting, credible research – the works. In four or five pages (1,000 to 1,250 words), you will make an original argument concerning the impact of art on war, or conversely war on art. By ‘art’ we mean the visual arts, music, film, novels – almost any creative undertaking. You can be quite liberal in your selection…just be prepared to defend the choice.

The key here is originality. Did Woodstock influence the Vietnam War? It’s probably easy to argue that it did (and also that it didn’t, since we were in Vietnam for another 6 years). Did the poetry of Wilfred Owen horrify the English so much that they avoided a second war with the Germans? No. And neither of these approaches would make a worthy paper.

Your four to five page paper must have at least three credible sources (not including the assigned work for the class).

It is worth noting that the students read several lengthy articles reviewing pop-culture icons (like the Saw series of films) that argued deep connections to more serious issues than the students might at first see. So they were set up to engage intellectually with material of their own choosing, and connect it to a war. There were several Harry Potter essays, but the results were satisfyingly diverse.

Here is an example cover letter from a student:

Example cover letter: Click for a larger image

Here is the rubric used to assess and respond to the student, as it appears in Waypoint (interactive rubric software – the rubric could be translated to a paper-based approach):

Rubric used to assess and respond: Click for a larger image

This student received the following feedback, along with an annotated document (created in Microsoft Word, then appended to the feedback in Waypoint), from the instructor:

Feedback to student: Click for a larger image

Needless to say, this kind of feedback is unusual in any educational setting – but the above was created in about 8 minutes, and the cover letter process (along with other tips and tricks) helps make sure the process is constructive and useful.

It is worth noting that this assignment was given in the middle of the academic term, so the students could be expected to learn from the feedback, then apply it in a final project that did not receive this kind of detailed feedback.

I had the good fortune to be invited to attend Dr. Ken Bain’s Best Teachers Summer Institute this week.

The conference, now in its twelth (or so) year, brings teachers from around the world together to share best practices based on Dr. Bain’s work. He published an award winning book in 2004, called What the Best College Teachers Do that draws universal lessons from the work of master teachers.

Since first meeting Dr. Bain at a conference last February, I have adjusted my own teaching with impressive results. Much of what you will find in his book feels like common sense, but you feel like you are saving ten or twenty years of trial and error on your own part, and standing on the shoulders of the great teachers working around you (who may not be readily available as mentors in your own institution!).

A brief example I took from this week’s conference: assign readings based on questions that come up in class, rather than assigning readings to generate class discussion.

I increasingly use readings available through my university’s online databases rather than anthologies, and I frequently use very contemporary readings. For instance, I used five articles from the New Yorker magazine as part of my recent persuasive writing course in-class essay final. We had read Zadie Smith’s fantastic novel White Teeth, and I asked students to pick one of the following articles and then make a connection between the article and the novel. Since we had been working on these kinds of abstract connections all term, that was the only direction I gave. The articles:

o Our Own Devices: Does technology drive history?

· From the May 12 New Yorker by Jill Lepore

· Explores the impact of individual inventors (Edison etc.)

o Pixel Perfect: Pascal Dangin’s virtual reality?

· From the May 12 New Yorker by Lauren Collins

· Discusses the guy who retouches fashion photographs

o The Last Bite: Is the world’s food system collapsing?

· From the May 19 New Yorker by Bee Wilson

· Discusses market forces changing the role of food in our society

o After Empire: Chinua Achebe and the great African novel

· From the May 26 New Yorker by Ruth Franklin

· Discusses Achebe’s career against the backdrop of colonialism and post-colonialism in Africa. (You don’t need to know his writing to really enjoy this).

o Hungry Minds: Tales from a Chelsea Soup Kitchen (not available online)

· From the May 26 New Yorker by Ian Frazier

· Uplifting history of a soup kitchen in NYC

So students were able to enjoy some choice in what they read and I was guaranteed not to have to read the same essay 85 times, especially since the novel is so multifaceted. Between picking various scenes/characters from the novel, an article from the above list, and a particular issue from those long articles, I receive a wonderfully diverse set of essays.

Some of the connections were truly impressive. Here’s a long introductory paragraph linking the immigrant experience of Samad Iqbal, one of the novel’s central characters, with (of all things) the worldwide food crisis:

My parents came to America as immigrants. They were surprised at first to see the vast difference in cultures between American and India, but they were able to realize that change is imminent. They have to get used to things around here. In White Teeth, Samad and Alsana are immigrants to England. The difference between them and my parents is that they are reluctant to change. They do not let people help them or change their ideals. They also end up breaking their family because of their stubbornness. Similarly, in a New Yorker article, the same stubbornness can result in a major food crisis in the coming decade. Unless people decide to change their ways we may end up in a famine.

Another, linking Irie’s body-image issues in the novel to the “Pixel Perfect” article about photo-retouching.

What is the social beauty norm? Since 1857, artists have been retouching “those portraits for which the camera may be said to have laid out the foundation.” This allows us to assume that vanity over one’s appearance – to the point of altering a picture – is not a recent occurrence.

Irie Jones, a character in White Teeth by Zadie Smith, exemplifies a typical self-conscious, gullible teenager. She dislikes the shape of her curvy body and her wild, curly hair. Chapter 11 begins with her looking ata sign that says, “Lose Weight to Earn Money.” Knowing that losing her curves would take a long time, she decides to instead take matters into her own hands and change another facet of her appearance – her hair.

(then later in the essay – this one takes a non-5 paragraph approach)

The reason Millat doesn’t sleep with Irie until close to the end of the novel is because he is searching for flaws in the pretty girls. Altered pictures, such as the ones by Dangin, motivate us to find fault in perfect forms. When we see regular people on the streets, we see their “blips,” for instance, the boy with the slightly crooked nose, or the girl with frizzy hair – and we move on. Yet when we see a perfectly modified picture, our human curiousity drives us to search for her flaws, which may take a while to find, and we keep looking.

Millat has this natural curiousity; he spends time with pretty girls to find their “blips.” Irie, to him, is a girl he knows from childhood, and he already knows all of her imperfections, inside and out. He has a drive to see what else is out there, even if truly deep down he knows he likes Irie and thinks she is beautiful with all her curves and curls.”

These students (both essays developed the ideas much further – but to me they had a clearly original idea from the beginning) connected deeply with both the issues/themes in the novel and an idea from the article they chose. So this was a successful use of assigned readings; but I still predicted what they might want to read rather than letting our class discussion take us (in an investigative manner) to the readings.

There was a moment at the conference when the idea of outcomes assessment, and the gulf between what we actually DO as teachers and our real goals for our students, became very clear.

A journalism professor commented on the idea of giving feedback, and allowing students to experiment, before any formal grading. He did a lot of drafts with students, and while many students put considerable effort into these early versions, a number of students only became motivated when there was a real deadeline. Another set of students had trouble with the idea of a deadline – they seemed content to hand work in late, which as the professor said, could cost hundreds of thousands of dollars in the real world of journalism. “Meeting deadlines is an outcome,” he said.

Sometimes it is only when we try to explain a complex idea to others that we truly understand it ourselves.

Deadlines are important in many more disciplines than journalism. I taught in an interdisciplinary engineering program for nine years. Students wrote proposals, and we talked to them about industry, and how even the slightest deviation from the expected format/content (never mind being late) could result in a proposal being discarded unread.

But did we assess this crucial skill (or value)? Only in the sense of deducting points from late submissions. And even then the system was incoherent; some professors deducted points for late submissions, others didn’t.

I only had a brief chance to talk to the journalism professor. I wasn’t able to find out if his program articulated this crucial outcome in their academic goals, or if they stayed with higher-level abstractions. He had experimented with many approaches, none of them satisfactory. He felt the issue of deadlines and the quality of revision defied the rubrics approach, and I know what he means. If “timeliness” is a criteria on a rubric, and you get a Pulitizer-winning article two days late, how can the document even qualify for reading and grading?

This issue leads to the more general one of procrastination in students of all levels, backgrounds, and disciplines. We do very little in higher education to stack the deck in favor of our students, preferring to assign term papers in week 2 of the term, then collect 20 page documents in week 15 (many of them written in the 24 hours preceeding the deadline). That isn’t how industry works – what boss gives a new employee 13 weeks of unsupervised labor on a major project?

As for the challenge of designing an assessment for the outcome of “meets deadlines,” it seemed to me there were several opportunities.

One would be to get students comfortable with the challenge of completing work in a rush. Perhaps, at the beginning of class (the first class of the term?) the students must interview their neighbor and write a 250 word profile due at the end of class. Because it is a journalism class, the expectation would be “press ready.” With a few of these under their belts, enforced by a strict deadline, students would grow more confident in their abilities to write under stress.

As for revising in a journalism class, maybe some real-world tension could invigorate things. If 25 students are enrolled, perhaps the professor (the editor) tells the students up front that 10 of the 25 drafts will actually have to run as-is in the next edition of the magazine/newspaper. So students would be afraid that their rather shaky draft would be published…potentially motivating them to write better drafts.

This might be overly dramatic for some, but by the time students are in an advanced journalism course they should have editing skills, writing skills, and time-management skills. The expectation is fair, and would probably invigorate students rather than the standard “writing for the teacher” response to draft review.