How To Crowdsource Grading

I loved returning to teaching last year after several years in administration . . . except for the grading.  I can't think of a more meaningless, superficial, cynical way to evaluate learning in a class on new modes of digital thinking (including rethinking evaluation) than by assigning a grade.  Top-down grading by the prof  turns learning (which should be a deep pleasure, setting up for a lifetime of curiosity) into a crass competition:  how do I snag the highest grade for the least amount of work? how do I give the prof what she wants so I can get the A that I need for med school?  That's the opposite of learning and curiosity, the opposite of everything I believe as a teacher, and is, quite frankly, a waste of my time and the students' time. There has to be a better way . . .

 

So, this year, when I teach "This Is Your Brain on the Internet," I'm trying out a new point system supplemented, first, by peer review and by my own constant commentary (written and oral) on student progress, goals, ambitions, and contributions.   Grading itself will be by contract:   Do all the work (and there is a lot of work), and you get an A.   Don't need an A?  Don't have time to do all the work?  No problem.  You can aim for and earn a B. There will be a chart.  You do the assignment satisfactorily, you get the points.  Add up the points, there's your grade.  Clearcut.  No guesswork.  No second-guessing 'what the prof wants.' No gaming the system.  Clearcut.  Student is responsible. 

 

But what determines meeting the standard required in this point system?  What does it mean to do work "satisfactorily"?  And how to judge quality, you ask?  Crowdsourcing.  Since I already have structured my seminar (it worked brilliantly last year)  so that two students lead us in every class, they can now also read all the class blogs (as they used to) and pass judgment on whether the blogs posted by their fellow students  are satisfactory. Thumbs up, thumbs down.   If not, any student who wishes can revise. If you revise, you get the credit.  End of story.  Or, if you are too busy and want to skip it, no problem.  It just means you'll have fewer ticks on the chart and will probably get the lower grade.  No whining.  It's clearcut and everyone knows the system from day one.  (btw, every study of peer review among students shows that students perform at a higher level, and with more care, when they know they are being evaluated by their peers than when they know only the teacher and the TA will be grading). 

 

What this teaches my students is responsibility, credibility, judgment, honesty, and how to offer good criticism to one's peers--and, in turn, how to receive it.  The beauty comes in the fact that those who judge one week are among those who are judged the next.  Throughout the course, "judgment" will be a main subject of the course, as it should be in a course entitled "This Is Your Brain on the Internet."   Contributing to the whole of a group is another skill this course will emphasize.  How is cognition different in a customizing, process-oriented, collaborative online public environment?  How do we learn to contribute and collaborate well in such an environment.  Little in our formal education prepares us to be responsible participants of the Internet.  This course proposes an evaluation system that matches the purpose of the course, where students learn how to be responsible judges of quality and helps them learn to be responsive to feedback as well.    I can't imagine better skills to learn within the safe confines of a class, with a prof on hand to offer constructive feedback (including to those giving feedback).  

 

Here's the syllabus for the course.  Skip to the bottom for the section on grading.  I'm happy for comments.  And we (the students and I) will let you know how it works.

----

Subsequent addition (Aug 15):   This post has garnered so much attention that I wrote a longer follow-up which explains the history and theory of evaluation motivating this experiment:  http://www.hastac.org/blogs/cathy-davidson/crowdsourcing-grading-follow 

-----------
ISIS 120S-01, English 173S-05: This is Your Brain on the Internet
Spring 2010    IMPS Space,  Rm. 230, John Hope Franklin Center

DESCRIPTION:
This is Your Brain on the Internet is open to any student fascinated by how we come to know the world and how we may or may not know the world differently in the Information Age. Our quest in this course will be to explore many different, quirky, eccentric, and exceptional models of mind in order to force ourselves to think, together, about what models best suit our digital, interactive, collaborative age.  Although we are in a great era of neuroscience and are learning more and more about our mental processing, what we do not know about how our brain works is infinitely more vast than what we know.  Thus we make models to try to explain ourselves to ourselves.  Every era (and the present is no exception) and every culture imagines its own models of mind.  In the scientific method, this hypothesis then both shapes experiments and data collection and uses experimental findings and the data collected to test, refine, or (sometimes) refute the hypothesis.  

This class advances an argument:  We are living in one of the most momentous times of change in human history.  We have changed.  Now we need to name the paradigm that has already shifted.   This class will be testing that argument in myriad ways.   We will be thinking together about how we know the world, how we think, and how we think about thinking as individuals, as groups, as a culture, as subcultures, in a historical moment, as mediated by and through technology.  The readings are intended as provocations.  Some are evocative, some controversial, all have strong points of view, all are polemical in the sense that they advocate for models of mind, collaboration, interaction, and mediation.  All are also situated in the sense that they do not look at cognition in an abstract sense, as divorced from social concerns, but as deeply rooted in cultural arrangements, so another focus of the course will be on new ways that humans interact with one another as friends, business partners, and members of a global information community.  How are collaborations different when they are face-to-face than when virtual, mediated by technology?  Our own classroom will move back and forth between actual and virtual experiences, including observation of highly complex collaborative environments (including choreography, improvisation, and other ways of interacting with and without words), some of which involve technology and some of which do not.
 
The course is conceived as a trans-disciplinary exploration in which we will consider the deep structure of cognition and community in a digital age. Well learn from theoretical and expressive books and articles ranging from neuroscience to films and literature, from various experimental and mainstream films as well as from a range of non-traditional sources (websites, interactive games and virtual environments, new media art exhibits, a backstage tour, conversations with social networking activists and community organizers, demonstrations by performance artists and illusionists, Virtual Reality tours, etc.) We will also learn from engaged collaboration (collaboration by difference) with others who have complementary skills, strengths, attitudes, and assumptions.  Every student will have at least three opportunities to work with partner(s) in the class and learning to be conscious and conscientious about that interactive, collaborative process is one of the learning methods of this course.

Readings:   Readings will be chosen by our student leaders from among the following but some other texts, films, experiences, websites, interactive projects, and so forth will be added or dropped by student leaders as the course unfolds. 
Jean-Dominique Bauby, The Diving Bell and the Butterfly
*Norman Doidge, The Brain That Changes Itself
Cathy N. Davidson and David Theo Goldberg, The Future of Learning Institutions in a
Digital Age [online; free download from MIT Press]
*Anna Everett,  ed., Learning Race and Ethnicity: Youth and Digital Media (MacArthur
Foundation Digital Media and Learning Series) [selections online]
*Temple Grandin,  Animals in Translation
*Christopher Kelty, Two Bits:  The Cultural Significance of Free Software [selections
online]
*Daniel Levitin, This Is Your Brain on Music
Mark Haddon, The Curious Incident of the Dog in the Night-Time
*Jeff Hawkins, On Intelligence
*Tara McPherson, ed., Innovative Uses and Unexpected Outcomes (MacArthur
Foundation Digital Media and Learning Series)  [selections online]
*Howard Rheingold, Smart Mobs
*Clay Shirky,  Here Comes Everybody:  The Power of Organizing Without Organizations

This course is student-driven:  All classes will be lead by pairs of students who will also give us reading assignments (books, articles, websites, films) and writing/creating assignments (setting us ways to interact with the material prior to or in class as well as after it).  The student leaders for each session will also evaluate every other students contributions, a process that will continue throughout the class, on our class Wordpress site.  Discussing the role and purpose of evaluation to the learning process will be a key feature of "This Is Your Brain on the Internet" and is part of making all of us more responsible citizens of the interactive, customizing, crowdsourcing Information Age.  One purpose of this course is for all of us to become used to peer evaluation, peer response, peer collaboration and to use these collective processes as aids toward our mutual learning goals.   Students will be encouraged to respond back to the student leaders making the comments and to discuss these responses in class.

There will be no exams and no formal, final research papers required in this class.  Any student who would like to write a final research paper can pitch an idea to the class.  If accepted, the student will be invited to write the paper.   In all other cases, students will work together on a final, collaborative multimedia online project that will be made available on a public website, probably the HASTAC (www.hastac.org) or the ISIS site.

Final Project:  Students may come into the class with a final project in mind and find collaborative partners, or we can begin discussing ideas from the beginning.  A class wiki will be set up for this purpose. 

Student Participation in the Set-Up of the Class:  The burden of customizing our WordPress site for the purposes of individual projects, of proposing other software possibilities or other technologies, will be shared across the students in the class.  We will have laptop-required and laptop-free days.  We will be blogging and twittering and facebooking to a group site. (Policy warning:  I do not friend current students on Facebook.)  Students will receive some grading points (see below) for contributions in this area.   Students without tech skills  will be urged to contribute in other ways--creatively, critically, in performance, offering feedback, and so forth.  Collaboration by difference is a theme we will take seriously in the course.

Grading and Evaluation.  After returning to teaching after several years as an administrator, I found grading to be the most outmoded, inconsequential, and irrelevant feature of teaching.  Thus for ISIS 120, S 2010, all students will receive the grade of A if they do all the work and their peers certify that they have done so in a satisfactory fashion.  If you choose not to do some of the assignments and receive a lower grade, thats permissible.  You will be given a chart at the beginning of the course with every assignment adding up to 100 points.  A conventional system will be assigned (95-100 points = A-, etc).  We total the scores at the end and you get the points youve achieved.  If, on any one assignment, peers rank the work unsatisfactory, you will either not be assigned any points for that assignment or you can submit a revised assignment in response to the class critique.  Revision and resubmission results in full points.  In other words, everyone who chooses to do the work to the satisfaction of his or her collaborative peers in the course will receive an A, but no one is required to do all of the work or to earn an A. 

In lieu of a final exam, students will write an evaluation of the class (in addition to the university-required student evaluations).  This will emphasize what you learned in the class, what you feel you accomplished (with "accomplished" self-defined).  I will offer feedback on your self-assessment, amounting to an "evaluation" of your contribution to the experiences of, in Toffler's phrase "learning, unlearning, and relearning" that are central to "Your Brain on the Internet." 

 

--------------------------------------------

 

Cathy N. Davidson is co-founder of HASTAC, a 11,000+ network committed to new modes of collaboration, research, learning, and institutional change.  Davidson teaches at Duke University where she co-directs the Ph.D. Lab in Digital Knowledge.   She is author of The Future of Thinking:  Learning Institutions for a Digital Age (with HASTAC co-founder David Theo Goldberg), and  Now You See It:  How the Brain Science of Attention Will Transform the Way We Live, Work, and Learn (Viking Press). She is co-PI on the HASTAC/MacArthur Foundation Digital Media and Learning Competitions.  In October 2012, she and David Theo Goldberg, HASTAC's cofounders, were named Educator of the Year by the World Technology Network.   NOTE:  The views expressed in Cat in the Stack blogs and in NOW YOU SEE IT are solely those of the author and not of HASTAC, nor of any institution or organization. Davidson also writes on her own author blog,  www.nowyouseeit.net .  The paperback of Now You See It was published July 31, 2012 :  http://tinyurl.com/bqquoaz

halavais

Never underestimate grade orientation

I can see this working with a small course. I tried something similar several years ago at Buffalo. My mistake was to make it a "curved" class (though only a positive curve). Two "gangs" (one a group of fraternity brothers, the other just people who met and formed up) reached an agreement that they would vote up each others' work no matter what, and non-members work down, no matter what, in order to increase their own grade in the class favorably, and hurt others' grades. I wrote it up a little here.

When I intervened, I got complaints: I had set up the rules, several said, if I didn't like the outcome, how was it their fault.

That's not to discourage the idea or the experimentation. Will be glad if yours has more success, and I've had some success with similar approaches, but be forewarned :).

 

Cathy Davidson

Lisa Dugan's Experiment in Crowdsourcing Grading

My Facebook friend Lisa Duggan has given me permission to cut and paste her comment about this blog, reposted to my Facebook page: 

"I've done something like this with my big undergrad class, Intersections: Race, Gender & Sexuality in US History, for years now. They do all the work, at a "good faith" level of quality (earning a check from their TA), show up on time to all classes and participate in discussion sections--they get an A. Grades scale down from there. The greatest thing about it is that many students without previous educational privilege *love* it and often do extremely well when not being judged in the usual way--reading a book a week, writing response papers every week, and ultimately participating at grad student level. Entitled students who try to skate by on a good prose style do not like it at all... :>). ---from Prof Lisa Duggan, Prof of Social and Cultural Analysis, NYU

Cathy Davidson

Katie King's Experiment in Crowdsourcing Grading

My Facebook Friend Katie King has also given me permission to cut and paste the comment she made about this blog, reposted to Facebook, on this HASTAC site:

 

"I come from the "written evaluations" school of education, but I know it is very labor intensive and not easy to implement for large scale mass education.... I cannot tell you how much I loathe grades and grading as well as sincerely believe they are pernicious. But I don't see how in today's assessment climate anyone pays attention to this -- the common sense ideologies are so strong here that countering them is harder and harder." --- Prof of Women's Studies, U of Maryland

ibogost

In defense of grades

I'll go against the grain. I like grades. Or more precisely, I like that grades represent a normalized scale of relative accomplishment. 

In my classes, I have a simple grading policy. For each of my assignments, I describe the basic requirements in careful detail. If they choose, students have the option of simply meeting those requirements. Doing so constitutes "doing fine" and earns the grade of C. Failing to meet the requirements earns a grade below C. Meeting the requirements with additional care, creativity, and effectiveness earns a grade of B. And doing excellent or superb work earns the grade of A. 

In the first meeting of my classes, I am very careful to explain the virtues of "doing fine." I devote the entire first day to a discussion of fineness. Most careers require little more than doing fine. Indeed, most people chose to "do fine" in their choice of work, whether it is in public relations, dermatology, baggage handling, lawyering, pet grooming, sous-chefing, postal delivery, investment banking, or higher education. There's nothing wrong with doing fine, really. Doing fine means that you did everything that was necessary. Nobody got hurt, but nobody got excited either. Doing fine is a great way to insure that you can spend the evenings with your family or clear your mind to work on your boat on the weekend. Doing fine is a valid lifestyle choice.

But as educators, we are doing our society a disservice by equating "doing fine" with doing well, or with being excellent. When I hear about starting students out at "A" and letting things fall off from there, I wince. A is arbitrary, sure, but if we have to use this arbitrary scale of A - F then we might as well balance it against some non-arbitrary concept. And one non-arbitrary concept is the difference between sucking, doing fine, and being excellent. When showing up and getting by is enough to count as excellence, something's wrong.

I think part of my attitude comes from having spent a lot of time in industry. Do you know how hard it is to find people who want to do more than "fine" in industry of any kind? It's very rare. And like I said, nobody gets hurt, but nobody gets excited either. I believe that the university is a good place and a good time for students to learn about the difference between getting by (note that I didn't say "mediocrity") and striving for excellence. I also think they should be allowed to make the choice, which is not to say that I shouldn't be required to encourage them. For example, we've seen cases in fields like finance over the past year where "doing fine" turned out not to be doing fine, at all. That's an interesting lesson to consider.

I empathize with the idea, no doubt true, that students perform better when they are evaluated by their peers. But is it any surprise that they do when they know they'll get a B or an A from a professor or TA? At least when their peers are involved, there's the chance of social praise or mockery.

Notice that nothing I have said here precludes the idea of collaborative grading, taking account of improvement, doing written evaluations, or any other principles advanced above. Nothing I'm saying defends the American grading system as an absolute good or as a natural order. Rather, I'm suggesting that we ought to take the idea of excellence very, very seriously. And for all its ills, the letter-grading system gives us a quite helpful abstraction to facilitate that process, if we choose to use it in the right way.

Cathy Davidson

I like this

I like your thinking on this subject . . . I'm just betting that when each class is "run" by a team of students and part of their job is responding to all the blogs and being a grader, that the standard of excellence goes way, way up, and my interest is in helping students to internalize excellence, curiosity, and evaluative skills rather than enforcing them from outside.  I have no problem with anyone else giving grades---especially when it is carefully thought out, Ian, as you have thought it out.  For me, personally, they are a sham.  I hate the "is this an A- or a B+" thinking at semester's end and then the "I deserved an A- not a B+ pleading."  So I want to try something new.  I've never done it before and it may be a one-time-only deal.  At present, it's untested by moi and an experiment.  I promise to report. 

 

 The interesting evaluation experiments that have been conducted in several fields  suggest that, if they think they can get an A, students will perform to the A standard.  If their peers are judging them, they tend to exceed that by a considerable shot.  I have definitely found that to be the case when students do their own projects and have to present them to one another as opposed to just writing a paper and handing it in to me.   Whether this new way of doing things that I'm constructing has similar outcomes remains to be seen.

 

I really like your thinking and really like your sharing it, Ian.  Many thanks!    Remember, I'm a newbie at teaching, after a long hiatus, so all of this is still new, experimental, and exciting for me.   I'm seeing what I can do and also pushing at the very idea of what "evaluation" is.   I'm really hoping that others who have tried various methods or who have strong opinions on this matter will join this impromptu forum.

ibogost

Minus and Plus

You know, I didn't even think about this, but one of the things that might influence my relative taste for letter grades is that Georgia Tech has no minus and plus grades. I could certainly use them on my own assignments, if I chose, but when it comes time to submit my grades at the end of the term, it's just A B C D F, and that's what goes on the student's transcript. So I've pretty much adopted that approach with all my grading. I'm with you on the pleading and pedantry. Maybe try giving up on +/-, even in the peer evaluation, and see if that clarifies things. 

Cathy Davidson

Interesting

yeah, those plus/minus . . . . the narcissism of small differences tyrannizes us again!  

Joe_McCarthy

A related story in HBR on motivations, minimums and maximums

I'm coming to this discussion rather late, having discovered it via Cathy's latest update.

I just read "A Story About Motivation" in Harvard Business Review - that was tweeted by @DanielPink, author of "Drive: The Surprising Truth About What Motivates Us" - about a group of consequential strangers who spontaneously joined forces to help an elderly man with a walker descend the slippery steps of his apartment building and get into a Access-A-Ride bus ... while the driver watched and waited, but did not rise to the occasion to do more than the minimum. As with this post, the comments considerably enrich the story, and one in particular, by Todd Cherches, seemed closely related to Ian's comment and I thought others might find the reference to ABCD - going Above & Beyond the Call of Duty - inspiring ... so I'm copying it below:

The driver, like anyone in any job, has a choice -- (and it IS an intentional CHOICE!) -- to approach his work in one of two ways: settling for doing the bare minimum...or seeking to do the "bare maximum."

The "bare minimum" of anything is defined as "the smallest possible quantity, or the least fulfilling -- but still adequate -- condition that is required, acceptable, or suitable for some purpose." This is how way too many people approach their jobs.

The "bare maximum" (my definition) is the exact opposite of that. It is about always looking to do the MOST you possibly can, to add extra value in everything we do, in any ways possible.

Those who approach their work, and their customers, and their life, with an "ABCD" attitude and mindset -- as you exhibited by helping the old man -- go through their day on constant look-out for ways in which they can deliver "ABCD" -- by going "Above & Beyond the Call of Duty." Those who do, find that that in and of itself (as you described) is its own reward.

wfarren

there has to be a better way

Wow.  This sounds like a great class. Very interesting.

I'm currently trying to do something similar (in some ways), getting learners to become involved and responsible for their own assessment and that of their peers. (Your writing btw, has been incredibly useful toward this effort and has informed me in many ways, so thanks.) I'll also be using Wordpress, along with a forum, wiki, the Mahara efolio system, blog comments (ratable), etc. as a way for students to receive genuine feedback. I'm trying to set up the site (class) (http://www.plearn.net/class) in such a way that each student's work will be easily viewable using categories, tag searches, etc. Students will learn how to manage an online portfolio, using Mahara, will decide who gets to see it, and when.

In keeping with the goals of the International Coalition for Electronic Portfolio Research, assessments used on the site will look at:

  • transforming accountability driven by testing into richer conversations around inquiry into learning; and
  • opening a detached, hierarchical academy to engagement across the multiple knowledge spaces of the digital world. (source)
  • If you get a chance, I would love for you to visit (http://www.plearn.net/class) and send suggestions/comments my way. If I can make this thing work, I'd also be very happy to have anyone who wishes, tag along, and study the effort from a social media/learning standpoint. Having your class look in and participate, would be great.

    Thanks for all your efforts in furthering the cause of education.

    Kind regards,

    Bill Farren

    Cathy Davidson

    PLearners

    Hi, Bill Farren,

     

    I just spent some time on your site and I'm fascinated.  As I understand it, students vote on where to send you, you go there, and then, from that actual place, you orchestrate learning communities and participatory learning for fifty students who might themselves be based anywhere in the world.  Do I have it right?   So you are both site-based in the country and teaching virtually, using the specific site as the nexus of a range of learning experiences.   As your class gets going, I hope you or your students will blog on the HASTAC site, or reblog here, to tell us all about your pedagogical experiences (by which I mean everything that happens).  I think you'll find an interested community here.

     

    On technical matters:  I've not used the electronic portfolio method but am learning more about it and will look into  Mahara. I also recommend Howard Rheingold's Social Media Classroom.  http://socialmediaclassroom.com 

     

    Good luck with this wonderful project, and kind regards to you, too,

     

    Cathy

     

     

    wfarren

    that's it

    "As I understand it, students vote on where to send you, you go there, and then, from that actual place, you orchestrate learning communities and participatory learning for fifty students who might themselves be based anywhere in the world.  Do I have it right?"

    In essence, that's it, Cathy. My big challenge right now will be to see if I can get any paying students in order to make it happen. There's not a whole lot of info about this type of thing out there. (As far as I know, it hasn't been done before--at least by an individual educator not formally affiliated with an educational institution.) It's pretty much sailing without any charts (and wind, at times).

    I'd be very happy to have studens involved with the HASTAC community as I know it's chock full of people involved in cutting-edge projects willing and able to provide great feedback.

    I bumped into Howard Rheingold's Social Media Classroom after I had set up most of the structure for my class using Wordpress. It looks like a great tool and one that I think the less techy educator would be more inclined to use. It's something that I might migrate to eventually. I think there's value in having a more unified out-of-the-box solution, the likes of Social Media Classroom. For those that are familiar with Wordpress and it's modular architecture, they'll be able to see an example of it being used as a class platform. Maybe some good people will develop educationally-tailored modules for WP, like integrated efolios. So far, I've been able to find a lot of really nice tools to integrate into Wordpress, although it has taken some time and effort.

    Kind regards,

    Bill

    Evan Donahue

    Grades vs accomplishment

    One thing I think worth noting given the discussion here is that a lot of the time the debate about what exactly to do with grades seems to collapse student performance into a reductive, one-dimensional scale of quality. It seems to me that imagining of "an assignment" as possessing certain criteria for successful accomplishment and then grading based on meeting those criteria and on surpasing them, even in creative ways, does two things: first, it a priori excludes from the discussion the possibility of approaching the assignment as a whole in new or creative ways, and second, it focuses in the measurement of worth on the individual product when the relevant object that should be considered is the development throughout and beyond the course.

    I personally view each assignment as an opportunity to try out new approaches, methodologies, or directions that I have never tried before and have no idea how well they will work. The result isn't always pretty, because not everything I try works as well as I thought it might, or perhaps it needs a new approach or some fine tuning. Pedagogically I find the benefit for attempting something like that to be huge, however often times I do this at a great risk to my own grade, which I think is unfortunate.

    Moreover, thinking in terms of individual assignments is necessarily reductive. If I produce a poor assignment at first because I attempted something that didn't work very well, but continue working with those ideas and eventually yield something really nice, how should that be graded? One could argue that the grading will balance out if you do well on the later assignment to make up for the first, but what that glosses over is the fact that if I do poorly on something I was initially working with, I may very well be discouraged from continuing with that. Also, sometimes the delay on the return is longer than the scope of the class. Sometimes I work the whole semester trying to work something out that I never quite get down like I want to, but definitely make a ton of progress in working out.

    These sorts of situations don't really work well with the traditional model of grading, but I think are worth considering in the context of questioning that model. The model of evaluation, I think, does more than encourage or discourage work, it shapes the very way work is imagined. It is interesting to me that an earlier poster cited thinking derivative from industry, because I guess my philosophical stance would be to question whether the same mentality belongs in the academy. "Producing" products is the goal of industry, whereas "producing" products in the form of assignments here... is made the goal by our methods of evaluation but I would suggest certainly doesn't have to be and I would actually say should not be.

     

    Cathy Davidson

    Process

    Thanks for this post, Evan.  In my ideal world, grades in general would be replaced by an evaluation.  That's how it works at Reed, Hampshire, and a few other colleges and my sense is that the learning is ferocious, partly because there's no rigid "grade" at the end.   What I'm hoping with my method is that students learn to think about the whole process of what constitutes an evaluation.   Once one is out of school, evaluation is a far more subtle process (typically).   I really dislike the whole "assessment basis" of learning in America, a trend that is getting worse not better---some states want to have standardized tests even for university teaching.  Given the era we live in, where process-based contributions, customizing, and interaction in collaborative modes are all part of the hallmark of new forms of communication, I think we need far more attention not to "assessment" and "outcomes" but to how we can work together to yield something more exciting and interesting than, seeing only with our own eyes, we might be capable of delivering.   Thanks for sharing your thoughts from a students' pov. 

     

    And here's something encouraging to think about:  in less than twenty-four hours, and in the lazy summer time, and on a Sunday and a Monday (the slowest internet days traditionally), over 1300 people have read this post.   That means there are a lot of people out there thinking about grading.   I sense we're all concerned about the mismatch between staid and disciplinary and arbitrary ways of teaching, thinking, and also grading--and looking for a better way.  Thanks for contributing to that conversation.

    Evan Donahue

    A "student's" perspective

    I am not sure how much my views reflect a "student's" perspective. I have actually met with surprising opposition in the few cases where I've discussed my views on grades and learning with other students. This is highly anecdotal, at least one or two I've discussed it with were very much in favor of grades as a way to build a resume and indicate to employers that you performed well at xy or z, which to me, as someone who resolutely intends never to get a job or be a productive member of society, was a very foreign concept.

    Cathy Davidson

    One who is being graded . . .

    I don't mean you "represent" all students any more than I represent all profs, but all the other comments come from people doing the grading and your comment is from a student still in the process of going through a system where you are graded.

    Evan Donahue

    "A student's perspective"

    Yeah, I originally titled my post "A student's perspective" but then I decided that I wasn't really coming entirely from my experience as a student since I do get a certain amount of exposure to the professorial side of thinking about grades and students from esteemed professors donahue and wald.

    Cathy Davidson

    Donahue and Wald? I'm not

    Donahue and Wald?  I'm not sure I know them.  lol.

    wfarren

    a lot of good thoughts

    There are a lot of good thoughts here Evan and I'd have to agree with what you're saying. Right now I'm leaning more toward open ended, fluid, varied assessment, more in the realm of efolios, which simply allow for student reflection, peer, and "visitor" feedback. Like you mention in your comment, it does't seem like today's reductive grading systems allow for much experimentation on the student's part. They generally don't encourage experimental approaches, or open-ended time frames, when dealing with a certain idea or problem. The process is too often truncated in favor of the content.

    Cathy Davidson

    URL for CHE Piece on "How to Crowdsource Grading"

    Here's the url for the excellent Chronicle of Higher Education piece by Erica Hendry:  http://chronicle.com/blogPost/Duke-Professor-Uses/7538/

    lhitch

    Just in time

    For the past two years a contingent from Northeastern has taught an incredibly multinational, multiethnic, and primarily recently graduated from undergrad students in a masters program in Australia.  It has been high on impossible to figure out how to entice the majority of the students to read the material.  The problem is not language. It is the global 'non-reading' phenomenon and my colleagues and I have been desperately searching for creative ways to encourage reading and discussion when we return in a week to Australia to teach again this year.

    As the students are for the most part tech savvy and can easily sit in class and send a text message with one hand below the desk thinking we do not see this maneuver; are amazed to learn that we are on Facebook; and all carry mobile devices far more sophisticated than the conventional Blackberry, crowdsourcing grading is what we have been looking for.

    The concept of crowdsourcing will not, pardon the pun, be foreign to them and as many are from cultures where pride is important this concept just might work.  We will begin teaching August 10 and will connect back with this blog to update everyone on "international" crowdsourcing grading.

     

    Cathy Davidson

    global thanks

    Thanks so much for this interesting comment.  I must say that I first began experimenting with different forms of grading when I taught in Japan and found that students did incomparably better when they worked as groups than when they competed alone or against one another.   And they worked so hard so they could feel pride within their group.  Good luck trying this and let us know how it goes.  It's all an experiment---but that is, after all, what learning is supposed to be. 

    chrismurphy

    grades are a dirty, rotten, homework assignment

    I have enjoyed reading this blog entry, as well as the follow-up comments from others. Here follows a few thoughts from a working professional who teaches a technical class at an art school for a masters program. I begin my third go at it this fall, and will be teaching both an in-person course and for the first time and online version of it (to a separate class). My previous teaching experience comes from another era as a flight instructor.

    The singular annoyance I have in teaching is grading. I really despise it from beginning to end, and so I find amusing the three tiered granularity for grading: sucks, fine, and excellent. So long as we are stuck with grades, I would relish in the possibility I could instead give out S, F, and E instead of A, B, D, E or F. I don't see why we need more granularity than this in academia, but I haven't done any research on the history of grades and why we have this particular system.

    I'm on-board with the need for evaluation, and feedback. But that should be occurring throughout a semester, to the point that the final grade isn't the big surprise it seems to be, along with the end of semester pleading. There is constant evaluation in flight training, for example, but no grades. And the final evaluation by an examiner, if you meet the requirements for certification, your are made a pilot on the spot with a temporary certificate. If you don't meet the requirements, you get "pink slipped", thank you, please try again.

    However, in a school where students are graded on a finer granularity, you not only can't merely pass/fail, but you really can't even (fairly) drop the +/- designation. This is the mathematical equivalent of rounding, which invariably introduces an error. A student who qualifies for a "grade point" for that of a B+ instead receives either points for a B (grade deflation), or points for the A (grade inflation). It confuses better than better-than-average, with excellence. Perhaps ridiculous, but this is, after all, about discrimination.  I think we're obligated to use the school's system even if we prefer a different level of granularity.

    I think culprits are school admissions, and corporate human resource department, who drive grade importance and inflation. It's a cheap brainless method of evaluation to thin a large herd of applicants. Better evaluators take more into account than grades or GPA, but such evaluation takes more time and, if challenged, resource consuming to defend.

    I've come to equate grades with salary, or the fee the student would earn on a contract. In the real world, you are contractually obligated to provide goods or services for a certain amount of money, and if you don't the consequences may be: a.) You get fired. That's an F. b.) You get laid off. That's a D. c.) You don't get a raise. That's a C. d.) You get a raise, or perhaps an industry aware or some form of recognition. That's a B. e.) You get a raise, a promotion, and/or a particularly prestigious industry award or form of recognition.

    For freelance contract work it's a little different. That tends to be multi-leveled. At first glance it's pass/fail. If you really screw up, you might get paid nothing, or even find yourself in the middle of a lawsuit. If you did OK or excellent, you will get paid 100% of the contract. But if you did just OK, you may not get another contract with that same party.

    Point being, I try to relate grades, which never happen in the real world, to what will happen "out there". Grades are metaphor. And for me this makes it a little easier to evaluate their work, to think of myself as a fictional employer. Could my emphasis and evaluation be flawed? It almost certainly does contain flaws.

    Anyway, this is quit a long commentary. My class, for an art school, as a strong component of hard science to it, so I can't exactly leave up a majority of the grading to the class at large. Suffice to say, I like the idea of incorporating this concept of crowdsource as a percentage of the final grade, and am now considering that for this next semester.

    ProfHanley

    the wisdom of crowds

    Cathy - - your course looks great, and I'm going to steal a bit here and there for a course in New Media in Scholarship and Teaching that I'll be teaching this fall.

    Peer evaluation is certainly not new.  It's a standard practice in composition classes and has been for at least a couple of decades.  The fact that it's treated as new in a literature classroom says more probably about the pedagogical inertia of the literary institution.  I won't go into my own non-grade centered methods of evaluation.  ('Tho I would mention John McClymer and Lucia Knoles' article, "Erstaz Learning, Inauthentic Testing" and Randy Bass's work as great starting points for rethinking student evaluation in the American Studies classroom.)

    "Crowdsourcing" is a hot metaphor from the e-biz world.  Crowds are usually thought of in terms of large numbers, and collective intelligence or emergent behavior usually arises in conditions of dense, multifarious interaction - - think hives, flocks, traffic jams, and Wikipedia.  James Surowiecki, in one of the founding texts of "crowdsourcing" (The Wisdom of Crowds) says there are basically three conditions for catalyzing collective intelligence: cognitive diversity (especially disagreement and contest), independence, and decentralization.  I'm not sure how a "crowd" of two students could actualize these dynamics - - one of the main theses of emergent behavior is that under the right conditions quantity morphs into quality.  The question of the critical boiling point where these changes occur is a focus of much debate and work among those who study emergence (e.g. Mitch Resnick's computer modeling work with StarLogo.)

    My main point: Peer evaluation (or grading, only one form of peer evaluation) is a powerful tool.  However, I wonder if applying the "crowdsourcing" metaphor to a couple of students giving thumbs up or thumbs down doesn't sell the "wisdom of the crowd" idea a bit short.  I.e. imagine if we really could foment authentic collective intelligence in the classroom and create a true "wikinomics" of knowledge.  (To do this, what would we have to change about the way we think about teaching, about learning, about student performance, and about institutional roles and relations?)  As with other instances of the "crowd," this might really challenge our reigning notions of expertise, hierarchy, collaboration, and knowledge.  For me, that's where the pedagogical rubber hits the institutional road - - and where things become both really interesting and seriously disruptive.

    Evan Donahue

    Interesting you mention that.

    Interesting you mention that. I was recently meditating for unrelated reasons about the plausibility of exposing student work to a real 'crowd' by your standards. Many of the classes I take have class blogs that require us to post every so often, but it really amounts to very little since few people read more than they have to to post. I wonder if it might be an interesting pedagogical tack to have students take their topics and engage forums, bloggers, maybe blog themselves with sufficient quality and visibility to be read, etc and bring that back to the classroom, or keep track of it in some fashion. I'd imagine it depends on what is being taught -- the paradigm I had in mind was politics, as there are so many bloggers. Depending on what is being taught though I could see something like that. Just a thought.

    julietklein

    Parallels to Performance Assessment

    Cathy's comment on the problem of how assessment in the US educational system is handled is right on the mark, though we can look positively to outstanding work being done across K-16 on performance-based, also known as "authentic," assessment. The literature is rich classroom models, guidelines, and heuristics that provide flexible approaches to interdisciplinary and collaborative learning as well. To date, though, few have drawn a link between that literature and the dynamics of digital learning. The time is ripe for doing so.

    Cathy Davidson

    It is a course ABOUT Evaluation in a Digital Age

    Yes, this is exactly the point.  My frustration with grading is in the context of a course that is about how to rethink our learning and teaching practices in a digital age where collaboration and customizing are literacies to be practiced and improved.  I want a form of evaluation suitable to the form and mission of the course.  So, quite precisely, I want to put together the rich literature on assessment with a concept of digital instutitions and online participation.  THAT is what I'm seeking in this course.  Lesson #1:  Read my blog, read the CHE and Inside Higher Education and all the various other reports, read all the comments, and then blog about assessment and assessment culture--including the snarky comments on the web, made with utter cynicism and not even the attempt to read and think about my original post, as well as scores of productive and helpful comments such as yours.  

     

    I cannot imagine a better way to start "This Is Your Brain on the Internet" than by thinking about the different ways we respond and what it means when someone either pitches in and supplies more ideas as part of a process versus the way someone cynically says (as some have!) "Ah, yet another lazy prof trying to get out of work instead of doing things the good old traditional way we've 'always' done it in the past."   I am guessing students will be astonished to see the disrespect and lack of engagement of some of the comments, including those made by other professors.  It will be a fantastic way to begin my course on neuroscience and digitality, on new models of mind for an information age.

     

     

     

     

    DukeEgr93

    Quick Q. re: Laptops

    Just got here from The Chronicle article - good stuff.  Had a quick question (in addition to longer ones I am still formulating :) ) - is there a program in place to loan students in the course laptops for the semester if they do not have one?  Just wondering about the logistics of that.  I know some of our courses in Pratt have been using tablet PCs that are carted in so was just curious how that issue is handled here.  -Michael

    Cathy Davidson

    Will check into this

    It is my understanding that the library and CIT at Duke have loaners.  I'll check into this!

    nelsonlopez

    Larger classes

    How do you control the use of laptops in larger groups? Last semester I asked my research class of 35 students to stop using their laptops because it was hard to tell who was facebooking or who was doing research. I have used a similar grading system at Binghamton and it produced mixed results. I asked the students to comment on their peers' projects on YouTube and to rate them with YT stars system. It worked well until I realized people were deleting critical comments, and keeping the "I loved your presentation" ones. 

    DukeEgr93

    Indelible Wiki

    I have classes of around 120-140 -- not all of whom have or bring laptops.  For those that do, I've found that one or two shots across the bow for doing inappropriate tasks works...  Also, the room I teach in has three cameras in the back of the room and two passageways that allow for stealthy retreats to the rear of the room.  One time, I left a message on a students' facebook telling them to stop looking at facebook  :)

    Regarding the YT system - the ability to control/delete comments is a problem.  Have you thought about putting such a thing on some kind of wiki system that has a persistent history?  That, plus an admonition to retain all commentary, might work better than a system where the student has complete control over the content and is the only one who can access its history.  There may even be plugins for different wikis to allow for a rating system.

    ReyCarr

    Student Self-Grading

    Many universities have policies against involving students in self-grading schemes. We examined this in detail several years ago, and conducted what is probably the only research study on the factors that impact self-grading. See  Carr, R. (1977). The effects of specific guidelines on the accuracy of student self-evaluation. Canadian Journal of Education, 2(4), 65-79.

    We found that the more explicit the criteria for self-grading, the more likely there will be a match between the student's own assessment and what grade the faculty members would have assigned (if grading was completed in the traditional way).

    The problem with student self-grading, and what often creates the perception of "Mickey Mouse course" or "easy grade" is that faculty members have selected student self-evaluation as a way to abandon their assessment responsibility. This is not the same thing as using self-grading as a developmental tool to empower students. Both intentions reduce a faculty member's involvement in dealing with grades, but the empowerment model takes considerable reflection and work to create truly explicit criteria for students to be able to honestly make assessments based on their learning rather than their need.

    gharp

    Gaming Grading

    I am having students self-grade as an attempt to reinforce course themes.  I've gone so far as to make it a game since the main themes of the class are games, the role of design in decision making, and cooperation/defection. 

    The ability to have self-grading and crowdsourced assessment probably varies somewhat with course content, size, structure etc.  This happens to be a design class, perhaps most closely aligned with the field of graphic design.

    In the syllabus I explain that they must evaluate each other, and they are free to assign points in any way they choose according to the criteria I set out.  However, if I suspect they are gaming the system, I retain the option of adjusting the relative weight of those scores to my own inputs.

    For the first assignment, all grading was compiled anymously and students will get the scores back as feedback on their game design assignments.  So the idea is that the class form, content and method start to work together to build up elements of games.

    Next week, in anticipation of or third assignment on corruption, we are going to keep names attached to the asessments to see the effect of transparency on the grading itself.  Instead of showing it in 'score' mode, the gardes will be shown in 'network' mode to see if any stronger or weaker biases exist in the class system.

    So far, it seems to work even better than I expected.  It's a small class of ten students, and it seems to be allowing for more continuous feedback and more careful work.  What is interesting is that their grading is probably much more stringent and targeted than I would have given. It will be interesting to see if they recognize this and decide to crash the system in favor of my assessment :)

    Cathy Davidson

    Gaming Grades

    Gaming grades is really a fascinating way of thinking and rethinking how evaluation works.  Games, of course, are all about different forms of evaluation, competition, and interaction--so I can see how thinking through games is also thinking through evaluation.  Will you report back on how it all works out?

    gharp

    I'll try to report back, yes.

    I'll try to report back, yes. It's only a three week 'beta' class so the results should be in soon!

    gharp

    Visualizing Peer Grading

    A discussion this week reminded me that I owe a long-overdue follow-up to this thread.  Last fall I conducted a short, 3-week course on game design, information biases, and design for corruption.  One of the themes for the class was the value of information as feedback and as a means for transparent decision making.

    In order to reinforce these themes in a meaninful way, we conducted a series of peer grading sessions with three iterations:

    1. blind peer evaluation by secret ballot,
    2. peer grading where it was assumed the grader/recipient relationship would be transparent but wasn't, and finally
    3. peer grading where the recipient had knowledge of their assessor before and after grading--i.e. where peer grading was fully transparent.

    In general the process was fully open, but I did add one caveat:

    "While the class will collectively determine the average value for each person’s grade, the instructor determines the weight of that value. Thus, gaming the system may result in manipulation of the weighting, leading to great uncertainty on your part of the final relative contributions to your grade."

    Essentially I reserved the right to also get into the game by adding an addditional strategy if the students didn't play fair.

    Since the goals of the course focused on the value of visual design, information, and feedback as knowledge networking, I created a series of information graphics that allowed students to see their assessments of their peers as well as the ones they received from each other ad their instructor--me.

    Here are a few screen grabs from the data-driven graphics (all created with processing):

     

    You can see the range of scores given by each of the students in theis freeze-frame of the visualization as well as how the variation compared among students.  I didn't do the stats to confirm, but there is the  visual suggestion that my scores of the students contained a bot more variation then the rest.  The teachable moment with this was being able to show the presence or absence of any systematic bias in their own assessments of others--e.g. consistently higher, lower, arbitrary, and so on.  For this demo, names were kept anonymous, but students did know their own numbered id.  Note also the visual impact of the non-participating student.

     

    In the second image you can see the scores starting to cluster around the recipients of the scores.  Here again, variation is easy to distinguish.  Consequently, the students were able to assess their assessors and the consistency of grading--as if providing a sort of verification of the iinterpretations leading to their grades.

     

    There are lots of ways of iterating the exercise (including least of all making the visualization better).  One clear way is to run it during a longer course so that students can really understand what the system is and how to manipulate it in response to the feedback they are getting.  And if they can understand how to hack the grading system, they will be better off when they set of to design socio-semeiotic systems which can handle being hacked in their own right.

    All four visualizations are linked here in this post on grades as information: feedback, patterns, and transparency.

     

    Wade

    Gaming the system

    To address the issue of students teaming up against other students you could simply assign students randomly to small teams and then have the teams grade one another. This would force friends to compete against each other for grades and not allow them to collaborate with each other to game the system.

    I always love seeing professors think outside the box when it comes to improving education. I work for a professor at the University of Missouri who also found the issue of grading to be tedious and found multiple choice tests to be lacking when it came to evaluation and writing assignments nearly impossible in large classes. His solution - create a computer automated writing assignment evaluator called SAGrader that can assess student essays and give personal feedback for students writing instantly. His solution has saved him days of grading time over the course of a semester. However, I really like the more personal idea of crowdsourcing grades.

    I am not certain what the exact solution is but it's great to see professors addressing the issue of student evaluation in higher education. As I student, I am especially frustrated by the limited evaluation methods that do little to assess the true value of a student's work and effort in a course. I wish you luck in this experiment.

     

    Cathy Davidson

    Interesting

    Thanks for this.  I would love to see how a computer grades writing assignments--that would be a pretty remarkable program.  Or are you talking about the level of language and diction that is common on Word and other software programs?  

     

    I actually don't find grading tedious.  And in many kinds of courses grades work perfectly well.  But in a course on digital thinking, on preparing students for the challenges of civil and civic exchanges on line, a grade is almost meaningless as a pedagogical method.  I would like "evaluation" to be as thoughtful and informed as the other elements of this particular course.

     

    You write "I am especially frustrated by the limited evaluation methods . . ."    The limitations are what I, too, find frustrating.  We'll see if my method, in the context of this particular course, proves to be more flexible and expansive. 

    Wade

    Since he teaches Sociology he

    Since he teaches Sociology he wanted a program that could grade content. While style, spelling and grammar are important they aren't what he needed for his course. SAGrader uses a semantic web of knowledge (much like a concept map) that can analyze thousands of different ways students might say a particle phrase or concept. While he still grades final assignments himself, students use the software to get valuable feedback for revising their papers. It definitely

    helps out in a class of over 300 students.

    I'd rather not steer the conversation here elsewhere, but if anyone else is interested they can contact me at wade (at) ideaworks.com.

    As for grading not being tedious I do see your point. Grading can be a valuable way to gain insight into your students lives.

    I have taken several classes where the teacher, despite the class not being pass/fail, simply gave students either an A or an F because the only thing that could be evaluated was whether or not we showed up and made valuable contribution to classroom discussion.

    Evan Donahue

    grading on world of warcraft

    I was struck just now by the confluence of your mentioning that grading works perfectly well in some cases, and your mention somewhere else about cooperation in the form of warcraft raids. Whether grading works perfectly well or not I won't take a particular stance on, but one thing that grading does undeniably do is push some of the graded's energy towards whatever aligns with the grade scale. There is an interesting instance in warcraft raiding where often times many people will bring along a downloaded add-on to the game called "recount." The purpose of the program is to log the total damage or healing that individual players produce over the course of a raid, and publically display it for bragging rights or to figure out which players are good. The damage or healing score is sort of seen as a short hand for how effective the player has been. What it invariably serves to do, however, is compell many players to try to maximize their damage or healing score as much as possible so that others will respect them and invite them on more raids. If only damage or healing were the relevant scores for a successful raid, that would be fine, but it massively discourages players from stopping their individual maximizations to focus on useful roles that can be the difference in a successful or failed raid overall. Many healers will ignore dispelling curses and poisons on their teammates because that is time wasted from casting pure healing. Often times these decisions will come at critical times and doom the attempt. I thought that was something you might be interested in in context.

    gharp

    The Warcraft example is a

    The Warcraft example is a great one of how dialogue among players and developers can change the dynamics of the game.  Lucy Bradshaw (Executive Producer, Spore) spoke about this at length during the Games for Chaneg conference in June.  She discussed the dynamics they discovered in The SIMS between players and developers as the developers would create objects and the players would find new uses or game the sytem in other ways.  She stressed how this became an advantage for development and not a problem as they had initially thought.  By having continuous dialogue, the players became more interested in the game and the developers were always having to adapt and create new ideas.  So when students and players try to game the system, it's up to the developers/instructors to change the rules, signals, scoring, or other to make the game/class more fluid, challenging and informative.

    Cathy Davidson

    Fascinating conversation

    I didn't know about this aspect of WoW, either the Recount example or its etiology.   Fascinating.  I can't wait to discuss your discussion with my students next semester.   I love this concept:  "So when students and players try to game the system, it's up to the developers/instructors to change the rules, signals, scoring, or other to make the game/class more fluid, challenging and informative."    The various close-minded cynics who are raking this idea of "crowdsourced grading" over the coals tend to be convinced that either I, as a prof, or the students are doing this because it's "easy" and we're all "lazy" and trying to game the system.  It takes true gamers to understand that it isn't easy at all to game the system when the system is a sound one.  Thanks so much for this great contribution.

    nilspete

    Primary trait analysis as simple grading method

    While not espousing it for your application, I experimented with an online writing/grading system in the mid-90s. Students posted a short piece of writing into a database. For the assignment I identified (by reading sample student work) keywords that would likely be in a "strong" answer. In addition to nouns, I found "because" to be a useful word. The software counted the presence of the words in the student essay.

    While this was a weak grading, it did put the papers into "piles" and then I would skim the pieces to see if I agreed with the pile.

    What was wrong with that work, I now think, is that it graded without feedback, and it used one authority in the process where more perspectives could be more helpful to the learner. As for automating the task of sorting a bunch of student work into piles, it was a great time saver.

     

    nilspete

    Using crowdsourcing to improve assessment at multiple levels

    Cathy, Exciting work in this Chronicle story. I tried to post this comment there, maybe it worked, or maybe they don't like outbound links in comments.

    We have been thinking along similar lines. David Eubanks suggested that Amazon's Mechanical Turk might be used to crowd-source feedback more widely than just the students in your class. We took his analysis a bit deeper, comparing several different crowd sourcing mechanisms.

    That conversation is part of our thinking (see a working demo) about how to harvest feedback about both the student work and the assignment that prompted it.  We are trying to figure out how to scale these feedback ideas to the level of university accreditation and invite comments on that work.

    I disagree with your critics who think you are abdicating your role as teacher, rather, I think bringing more perspectives to your assessment has the potential to increase the authenticity of the learning.  To test that hypothesis, we have developed a demonstration , using the criteria we see in your syllabus. We invite your readers to try it. We will post the results here early next week.

    Readers: Please read "Duke Professor Uses 'Crowdsourcing' to Grade"
    By Erica Hendry then follow the link to a short survey that will let you assess Erica's work using Cathy's class criteria.

    DukeEgr93

    Cool!

    Great addition to the conversation - an actual survey!  As I commented in it, the survey itself may be a tad "binary" - possibly more targeted questions (satisfied with...what parts?) as well as a Likert scale for level of satisfaction.

    Cathy Davidson

    Your survey in my class

    We'll use your survey in my class and see what we think and contribute to it.  This is exactly the kind of experimental suggestion I was hoping for.   Evaluating methods of evaluation will be a key part of this excercise, and having all of these interesting tools to try will be key.  I love the idea of using mechanical turk for this purpose and then making comparisons across multiple ways of assessing writing to see what you learn from one, what from another.  I wish I were teaching in the fall instead of having to wait until next winter.  On the other hand, we are going to have a fascinating body of work to draw from as we think together about new ways of thinking on line. 

    By the way, did you see my longer, follow up posting? http://www.hastac.org/blogs/cathy-davidson/crowdsourcing-grading-follow.

     

    Thank you so much and good luck with your own work!

    jonippolito

    Crowdsourced feedback + trust metric

    Nils, I enjoyed your table comparing different crowdsourcing paradigms, and agree that the Slashdot model offers the most sustainable long-term approach. My colleagues and I at the University of Maine have pursued a similar course with The Pool, an online environment for sharing art and code that invites students to evaluate each other at various stages of their projects, from intent to approach to release. (Related links via the Chronicle and Harvard.)

    Like Slashdot's karma system, The Pool entrusts students who have contributed good work in the past with greater power to rate other students. In general students at U-Me have responded responsibly to this ethic; it may help that students are sometimes asked to evaluate students in other classes, which makes it harder to gang up in groups.

    That said, I once had two students in different classes game the system by submitting a joint project as though it were two separate ones, and then rate each other as high as possible. This example of "shill bidding" was troubling but didn't have much of an effect. Projects with more collaborators and stages float more prominently in The Pool, and so by separating their collective work into two pieces the culprits diminished their visibility in the process.

    We've only begun to explore cross-university use of The Pool, a few years back with an art class at UC Berkeley taught by Rick Rinehart and this coming fall with Laila Sakr's class on collaboration at UC Santa Cruz. If anyone else wants in on the fun, please let me know!

    EllaA

    Crowdsourcing Web sites for teachers

    Hi Cathy,

    I thought you might be interested in this article, which discusses your "This is Your Brain on the Internet" class and provides some opinion/analysis on the use of crowdsourcing in the classroom:

    http://www.findingdulcinea.com/news/education/2009/august/Is-Crowdsourci...

    I'm also curious as to your thoughts on crowdsourcing sites like SOS Classroom and Scitable.com. Will you set up something similar for Duke students?

     

    Thanks!

     

    Ella

    Cathy Davidson

    Matchy matchy

    Thanks for the url. That was a smart posting that understood that the method of my course and my content match.   I look forward to checkin gout SOS Classroom and Scitable to see if they might work for my class. 

    Cathy Davidson

    From the MacArthur Foundation Spotlight Blog

    This is reblogged from the MacArthur Foundation's Spotlight Blog, a rich resource for anyone interested in digital media and learning:  http://spotlight.macfound.org/main/entry/crowdsource_grading/

     

    Monday 24th August 2009 10:17 am

    How to Crowdsource Grading

    Professor and grantee Cathy Davidson experiments with unconventional grading systems to prepare students to be digital citizens. 

    image

    By Cathy Davidson

    A little over three weeks ago, as I was about to leave the country for a weeks vacation sans Internet, I posted a blog on the new HASTAC site that I assumed would be an uncontroversial placeholder in my absence.  I posted an update of a seminar I taught last spring, This Is Your Brain on the Internet, a course on new theories of cognition for a digital age.  I made some modest revisions in the course that focuses on a new model of mind for the participatory learning, interaction, and collaboration that is part of digital thinking, including an experiment I plan to conduct that revises conventional models of grading.  I came up with an experiment in grading--a point system combined with peer evaluationthat would highlight, in its methods, the mission of the course which is to prepare students for the interactive online forms of participation, feedback, evaluation, and peer review that are part of digital citizenship. The was blog was titled How To Crowdsource Grading.

    I had no idea what kinds of response this quiet blog would engender while I was in a WiFi-free zone.  I was suddenly hit by an avalanche of opinion. The story was picked up by the Chronicle of Higher Education, Inside Higher Education, the Raleigh News and Observer, and the Duke University Chronicle and would have gone much further had I been in the country to field interviews. The responses have been varied, some extremely helpful, some radically, and even rudely, dismissive. Surely, some insisted, if I am doing such an experiment, I am trying to get out of work (a nave idea of pedagogical experimentation if ever there was one).  If the prof isnt the flake, others proclaimed, then surely students will take the course to get an easy A (will they ever be surprised!). 

    But no matter. The rich body of internet chatter on this issue is now Assignment #1 in This Is Your Brain on the Internet. I can think of no better way to teach students about the power of online evaluation and contribution than by having them readand respond tothe range of digital opinions about their course. I promise to report back on the results when this experiment is over. 

     

     

    ormebk

    Crowdsourcing Student's Feedback

    Great post!  If you're looking for an alternative way to use crowdsourcing in the classroom, I encourage you to use CreateDebate in your classroom.  Teacher's can create a debate about any topic they'd like, students add in their opinions, and then vote on each other's arguments.  The best arguments (as voted on by the class)rise to the top of the debate, thus allowing students to grade each other's work.  Check it out here: http://www.createdebate.com/teachers

    Cathy Davidson

    Thank you.  I look forward to

    Thank you.  I look forward to checking this out!

    jonippolito

    CreateDebate or CreateConversation?

    I was glad to learn of this software--but the only options appear to be "2-sided debate" or "popularity contest." Are these really the sorts of interactions we want to encourage among students in a multicultural society?

    Cathy Davidson

    Like "Like" on Facebook: More options needed!

    Ah, you beat me to it.  I really dislike two-sided debates AND popularity contests.  We're on the same page.  It's like "like" on FB.  We need many more options.

    nilspete

    More perspectives and better criteria, less walled garden

    I'm in agreement, real debates have more than two sides, and might have more purposes than pitting sides against one another (such as reaching a compromise).

    Then one could also ask what the criteria for the judgement are. I have tried to train myself to not say "I like it" in response to my daughter's work. I try to talk to her in terms of dimensions that are important to me -- "you seem to have considered several perspectives" "I see you have revised this" etc.

    But my biggest concern would be to get the debate open to a wider community. For example, we worked with a course dealing with race and sports. Students were joining this conversation by creating/ updating entries in Wikipedia. It was in that forum that they engaged the "debate."

    Cathy Davidson

    HASTAC reader VM Kern was

    HASTAC reader VM Kern was prevented from posting this to the site when we were in repair one day so wrote me a side message that others will, I think, find useful.  Here it is.  Thahnks, VM Kern!

    "I just wanted to say that, as another reader has said, I will steal ideas from your experience for my practice of peer review in class (since 1997 - http://kern.ispeople.org/par_en.html, about 1000 student participations)

    What grabbed my attention about your report, especially, was the clear message that the current usual grading system (if not the whole educational system) has a bad business model and crowdsourcing is a sure thing in the future of education and grading. Of course, we need new educational technology to take advantage of crowdsourcing without losing control over its vulnerabilities.

    An information I think you may find interesting about the way Ive found your report: Google Alert (I receive updates on "Peer review" with "learning" or "education") directed me to "Finding Dulcinea", where I saw the link to your post in Hastac. Its a brave new crowdsourced world, but also a socio-technological world, in which software agents collaborate with humans performing important tasks such as alerts and recommendations.

    I have saved the posts for future work with my students (mostly from engineering and computing) at the grad. program in knowledge engineering and management at the Federal U. of Santa Catarina, Brazil, where we've been using peer review as a strong driver for interdisciplinarity (since our research object is knowledge as a production factor and we need to communicate well across different backgrounds and traditions).

    Congratulations!"

    Cathy Davidson

    HASTAC reader VM Kern was

    HASTAC reader VM Kern was prevented from posting this to the site when we were in repair one day so wrote me a side message that others will, I think, find useful.  Here it is.  Thahnks, VM Kern!

    "I just wanted to say that, as another reader has said, I will steal ideas from your experience for my practice of peer review in class (since 1997 - http://kern.ispeople.org/par_en.html, about 1000 student participations)

    What grabbed my attention about your report, especially, was the clear message that the current usual grading system (if not the whole educational system) has a bad business model and crowdsourcing is a sure thing in the future of education and grading. Of course, we need new educational technology to take advantage of crowdsourcing without losing control over its vulnerabilities.

    An information I think you may find interesting about the way Ive found your report: Google Alert (I receive updates on "Peer review" with "learning" or "education") directed me to "Finding Dulcinea", where I saw the link to your post in Hastac. Its a brave new crowdsourced world, but also a socio-technological world, in which software agents collaborate with humans performing important tasks such as alerts and recommendations.

    I have saved the posts for future work with my students (mostly from engineering and computing) at the grad. program in knowledge engineering and management at the Federal U. of Santa Catarina, Brazil, where we've been using peer review as a strong driver for interdisciplinarity (since our research object is knowledge as a production factor and we need to communicate well across different backgrounds and traditions).

    Congratulations!"

    maremel

    My experiences in two years with collaborative feedback: Mixed

    I have been teaching two graduate level digital media business classes at UCLA Anderson for a couple of years now, and added collaborative feedback and grading two years ago.  I give the final grades on assignments, but the feedback to the students comes in part from peers in an online shared space.

    My experiences so far: mixed with insights.

    I agree that grading is the least attractive part of the job.  Often, students who get B's and lower don't understand why their work isn't spectacular, but don't really want the feedback that goes with the real answer to the question.  The real hard part of grading -- qualitative work surrounding unstructured problems -- is very difficult so many teachers don't emphasize that in the graded curriculum...so the qualitative skills don't build, etc.  And then they end up in my class in graduate school having not had to work with messy questions.

    Collaborative grading and feedback surfaces lots of interesting challenges:

    -- Grading is a social contract.  Students come into my class thinking that their work will stay private, as classes have done in the past.  They are surprised when on day 1 I remind them that the syllabus (that they signed up for) states that their work will be shared online and reviewed by classmates. 

    -- Looking into each others' work ups the ante: I have found for two years now that full view of each others' work with feedback brings the quality of the floor up QUICKLY.  It stays fairly high...but the top students also see that they don't have to bust their rear ends as much and over time bring the top down too.

    -- Grading normally gives poor feedback.  An A, B, or C doesn't show what a student needs to do to improve with clarity.  At the same time, I find some students don't know what to do with feedback...or how to give it well.  I've shifted to asking the students to give positive feedback with suggestions, which works a bit better in helping students focus their efforts for the next piece.

    -- What is qualitatively good is a learned skill and needs some distance from the material.  If I even take the feedback from my graduate student TAs, often I get as many different opinions as I have TAs. I think it has taken me years to learn to filter the good with depth from the nice job...and to see the gold in the straw at times.

    -- Real insights have come from the wisdom of crowds in non-graded delivery.  We have marvelous midpoint group presentations and instead of collaborative grading I've created a mini-Academy Awards where students vote on qualities of the presentations...and the awards they get become the feedback on the quality of their work.  It also shows WIDE spectrums of opinions across the class about the positive attributes of a variety of delivery and research approaches.  The class then discusses their results and reactions to each others' work -- nice metacognition bumping along.

    -- Students have rebelled -- I have quite a few students who won't say anything until the end of class, then get very hostile that they had to be put in the position to grade and review other students.  It's hard and they are surprised that it is hard and takes work.  Yup, it sure does.

    Hope that helps and best of luck in the class!

    nilspete

    Rubrics help, as does the right frame for the task

    Maremel,

    A couple of thoughts. A rubric helps structure the "grading" and the feedback. We use the WSU Critical Thinking rubric, and slight derivatives in many classes with success. We coach students to copy language from the expanded form of the rubric into the comments they are making. This helps give educative language to the commenting.

    We have also seen the angry students you experience. On a course evaluation, we asked "What is your role and the teacher's role in your learning?" and got insight into the anger. The students fell into two groups on this question -- a group who viewed themselves as independent agents responsbile for their learning and a group who saw themselves as vessels that the instructor was to fill. The latter group did not appreciate having to do the "teachers job" of grading (aka peer assessing).

    Our take aways are that both use of the rubric to give good feedback and our view of the locus of control for learning need to be clearly communicated to students in order for them to be successful with these methods. But, I'd argue, giving good feedback, and seeking feedback while working are essential skills in professional workers of all stripes.

    Cathy Davidson

    Academy Awards and Rebellion

    Why would graduate students at one of the most distinguished business schools in the nation be rebelling--or getting academy awards--for being required, within the safety net of graduate eduation--to do what they'll be doing in the business world, where the penalty is loss of income, investment, or a job?  What does that tell us about education?  I think I know the answer and I think you do too .. . we are teaching our students, from Kindergarten on, that school is the place you get grades but the actual consequences are deferred.  Those happen in life.   Think about that!   I'm not writing more because I am finding the response to this so interesting it may, well, just be the starting point for my new book.  Your comments are exactly what I'm talking about.   Thanks so much for writing, and good luck.  What you are doing is right---even if your students don't think so!

    maremel

    Plug & Play learning

    With all that is available now, often college classes stay in the "I talk from the front" and you parrot back modality that we complain so much against in K12 reawakening.  Even at technology & education conferences, most of the efforts are on teaching teachers cool tools to bring more interesting things to the front of the classroom and not how to deal with the squishy, real world.  This applies to business university-level work as well.

    Feedback and failure are strong learning tools instead of just getting the answer right or being graded for doing the work assigned. You get what you reward and measure, so we get the type of classes that we reward and incentivize...

    Yikes!  How did we get here!?

    Cathy Davidson

    "You Get What You Reward and Measure"

    This is brilliant and exactly why I want to change the measure:  "You get what you reward and measure, so we get the type of classes that we reward and incentivize..."   You say it exactly, perfectly, right .  . . . and, as you imply, that is so, so wrong.

    Cathy Davidson

    Reprinted from McClatch-Tribune Information Services

    Professor opts for ‘crowdsourcing’ over traditional grading

    Posted on 07. Oct, 2009 by editor in Beyond GP

    By Eric Ferreri
    McClatchy Newspapers (MCT)

    DURHAM, N.C. — In one English class this fall, Duke students will grade themselves.

    That’s the idea behind Cathy Davidson’s “This is Your Brain on the Internet” course, an exploration of thought in the rapidly changing age of digital technology.

    “Do all the work, you get an A,” she writes on a blog explaining her course. “Don’t need an A? Don’t have time to do all the work? No problem. You can aim for and earn a B. There will be a chart. You do the assignment satisfactorily, you get the points. Add up the points, there’s your grade.”

    That practice, called contract grading, has been employed for decades. Here’s the twist: in Davidson’s class, students will decide whether the assignments are completed satisfactorily.

    Two students will lead each class, selecting readings and writing assignments and evaluating student work. There will be no exams or research papers, unless a student wants to write one, according to Davidson’s blog. Students will work together on a final multimedia project.

    On her blog, Davidson explains why she’s eschewing traditional grading for this new method, known as “crowdsourcing,” in which a task usually done by one person is instead done by a group.

    “After returning to teaching after several years as an administrator, I found grading to be the most outmoded, inconsequential, and irrelevant feature of teaching. Thus for (this class), all students will receive the grade of A if they do all the work and their peers certify that they have done so in a satisfactory fashion,” she writes. “Everyone who chooses to do the work to the satisfaction of his or her collaborative peers in the course will receive an A, but no one is required to do all of the work or to earn an A.”

    Davidson’s approach to this course is unusual, said Todd Zakrajsek executive director of the University of North Carolina-Chapel Hill’s Center for Faculty Excellence. Plenty of faculty members take issue with grading, and some surrender a portion of a class to student evaluation, Zakrajsek said. But Davidson is essentially giving students total control of grading.

    Doing so, he says, poses a complicated question.

    “If the real essence of a college education is to become a learned individual, grades really are inconsequential,” he said. “But we’re also using an education to gauge who really is learning.”

    And it also brings the professor’s role into question. Is Davidson’s job to simply give students information, or is she a manager, guiding students as they figure out things for themselves?

    Zakrajsek doesn’t know Davidson, but read the course description on her blog and came away impressed with how thoroughly the course has been thought out. But he does wonder whether students are qualified to decide whether class work is satisfactory.

    “She’s not way out on the fringe,” Zakrajsek said. “She’s just adamant about the fact that she doesn’t like grades. (But) I think students need real feedback to know how they’re doing in the class. Are students the best to make that determination?”

    Zach Perret, a Duke senior studying biology and chemistry, said the new class format may lead students to collaborate more and compete less.

    “It’s a competitive place,” Perret said. “Unfortunately, that culture leads to a little too much competition. It may make it a little more about learning.”

    Lee D. Baker, dean of academic affairs for Duke’s Trinity College of Arts & Sciences, said faculty members are encouraged to try new ways of teaching.

    “Cathy Davidson is a seasoned instructor and an innovative scholar,” Baker said. “And research suggests that the more students are engaged in each aspect of the class, the more learning takes place.”

    The Chronicle of Higher Education and Inside Higher Ed, two trade publications read heavily in academia, have done stories on Davidson’s new course.

    Davidson, who has tenure, was on vacation and difficult to reach. She did leave voice mail for The News & Observer saying in part that she is pleasantly surprised by the attention her blog post about her new course has received. The comment section prompted some teachers to encourage her use of crowdsourcing. Others saw value in professors grading.

    In 2003, Alex Halavais used a similar approach for a communications course he taught at the State University of New York’s Buffalo campus. He likes the idea of having students evaluate each other’s work and believes they learn more from each other than from a professor.

    But in his class of more than 100 students, students created teams and alliances, forging agreements with each other to drive up the grades of each member of a team while driving down the grades of students outside the clique.

    “I had failed to anticipate how collaborative they would be,” said Halavais, now a communications professor at Quinnipiac University in Connecticut. “Not to do good work, but to get good grades.”

    ———

    ON THE WEB

    Davidson’s blog: http://www.hastac.org/blogs/cathy-davidson/how-crowdsource-grading

    ———

    (c) 2009, The News & Observer (Raleigh, N.C.).

    Visit The News & Observer online at http://www.newsobserver.com/

    Distributed by McClatchy-Tribune Information Services.

    Tags: college, technology

    nilspete

    Students can do this and outsiders can help

    Cathy

    In the article: 'Zakrajsek said "... I think students need real feedback to know how theyre doing in the class. Are students the best to make that determination?" '

    We have evidence with lower division classes that students are able to produce the approximately the same assessment of a term paper using WSU's Critical Thinking Rubric as the instructor in the course. When you look at the rubric, and expand the scales, you will see that there is an educative quality to it -- educative to the reviewer and to the reviewed.

    When we tried this with an online version of the rubric and a design problem in a 300-level course, we asked reviewers to write feedback comments for each rubric dimension, in addition to providing a score. We found that students and industry professionals gave more textual feedback than faculty. A tag cloud analysis showed that there were also differences in the types of feedback among the groups. 

    What is important about these findings is that it allows scaling up of the amount of formative feedback students can get. The instructor is not the sole source, and the students need not be the sole source either. The community of practice can be elisted.

    The other conversation we are learning to facilitate is among the reviewers on the utility of the rubric itself. This offers the potential to refine the rubric in the context of its application (both relative to the work and to the community of practice).

    Cathy Davidson

    Real Feedback

    It's  curious to me that people assumed that, if I was experimenting with grading, I was abdicating my role giving feedback.  It's actually the opposite. Feedback is now a main point of the course.   We will be giving feedback to the feedback, and I last year responded to every blog by every student, but on line so everyone could also take the feedback and apply it to their own work.  The whole point is to think about feedback, not just slap a B+ on the paper as if that is feedback.   I'm getting more and more adamant every time I think about this more, every time I do more research.  And, of course, as the students give feedback, they are learning one of the most important skills ever, learning to criticize constructively and not for the end of ego-enhancement or power but to make sure we all arrive at something better in the end.   That's the purpose of this course, in every way.

    nilspete

    Beyond Grades to Rich Feedback

    Cathy, its interesting to return to this conversation short on the heels of the recent report by Kuh et al on institutional assessment. Here is an update on WSU's work on rich feedback in the Harvesting Gradebook

    Cathy Davidson

    Your fantastic site!

    Nils, I'm going to have a blog up shortly on "Crowdsourcing Authority" on the DMLCentral site and have offered the editor a number of links, including to your fantastic site and the Harvesting Gradebook discussion and studies.  If we would want to also quote/reference this image of crowdsourcing, to whom should he write for permission?   http://communitylearning.files.wordpress.com/2009/10/figure-2-peer-ratin...

    nilspete

    License Info

    Cathy

    I added a Creative Commons license to the site, thanks for pointing out that omission. If you mean DMLCentral.net, I just checked out the site (new to me). Post away. We may be able to help you clean up the image (extra blank lines of check boxes) or _even cooler_ help you get the interactive Google Gadget into the site so people can hide/show the layers of data. 

    While I'd prefer to keep the substance of this conversation in this blog comment thread (for your students to read how we collaborate), if you need to contact me on license or technical issues related to your request, please use nils_peterson@wsu.edu

    Sra. Bivens

    implications for foreign language learning

    Hi Cathy,

    I'm very intrigued to read about your experience in a modified grading scenario!  I an a world language teacher (Spanish/ESL to be exact) and am interested in implementing the same or similar concepts in my classes.  My ultimate goal is proficiency and mastery of the language with a growing awareness, tolerance, and understanding of the peoples and cultures.  I would like clarification and/or conversation on this to help with my construct.

    1. From what you've written, I assume that you make comments on the students' work so that their work should be evolving, just as the learning process should be? 
    2. You allow students to correct and improve their work through the comments of their classmates and you? 
    3. As far as individual assignments go, on the smaller scale, how do you take subjectivity out of the grading process? 
    4. What proves satisfactory vs. unsatisfactory? 
    5. How do you assign points to individual assignments? 
    6. Are some assignments worth more points than others? 
    7. What are your ideas for implementing this into a face-to-face and online language course (as I teach both)? 
    8. All of my f2f students have blogs that act as their e-portfolios in my class.  However, is all of your/their work online or is some of it paper-based as well? 
    9. Did you modify the courseload as a result of this new grading concept so that you give fewer assignments since they may have to redo/modify completed work? 

    I, as most educators, want to focus on quality, not quantity, but I do want the students to have enough practice in the target language to master the concepts.  Any advice, suggestions, comments, and/or ideas are greatly appreciated!

    Thanks!
    Teryn

    web: www.freewebs.com/srabivens

    twitter: srabivens