How To Crowdsource Grading
I loved returning to teaching last year after several years in administration . . . except for the grading. I can't think of a more meaningless, superficial, cynical way to evaluate learning in a class on new modes of digital thinking (including rethinking evaluation) than by assigning a grade. Top-down grading by the prof turns learning (which should be a deep pleasure, setting up for a lifetime of curiosity) into a crass competition: how do I snag the highest grade for the least amount of work? how do I give the prof what she wants so I can get the A that I need for med school? That's the opposite of learning and curiosity, the opposite of everything I believe as a teacher, and is, quite frankly, a waste of my time and the students' time. There has to be a better way . . .
So, this year, when I teach "This Is Your Brain on the Internet," I'm trying out a new point system supplemented, first, by peer review and by my own constant commentary (written and oral) on student progress, goals, ambitions, and contributions. Grading itself will be by contract: Do all the work (and there is a lot of work), and you get an A. Don't need an A? Don't have time to do all the work? No problem. You can aim for and earn a B. There will be a chart. You do the assignment satisfactorily, you get the points. Add up the points, there's your grade. Clearcut. No guesswork. No second-guessing 'what the prof wants.' No gaming the system. Clearcut. Student is responsible.
But what determines meeting the standard required in this point system? What does it mean to do work "satisfactorily"? And how to judge quality, you ask? Crowdsourcing. Since I already have structured my seminar (it worked brilliantly last year) so that two students lead us in every class, they can now also read all the class blogs (as they used to) and pass judgment on whether the blogs posted by their fellow students are satisfactory. Thumbs up, thumbs down. If not, any student who wishes can revise. If you revise, you get the credit. End of story. Or, if you are too busy and want to skip it, no problem. It just means you'll have fewer ticks on the chart and will probably get the lower grade. No whining. It's clearcut and everyone knows the system from day one. (btw, every study of peer review among students shows that students perform at a higher level, and with more care, when they know they are being evaluated by their peers than when they know only the teacher and the TA will be grading).
What this teaches my students is responsibility, credibility, judgment, honesty, and how to offer good criticism to one's peers--and, in turn, how to receive it. The beauty comes in the fact that those who judge one week are among those who are judged the next. Throughout the course, "judgment" will be a main subject of the course, as it should be in a course entitled "This Is Your Brain on the Internet." Contributing to the whole of a group is another skill this course will emphasize. How is cognition different in a customizing, process-oriented, collaborative online public environment? How do we learn to contribute and collaborate well in such an environment. Little in our formal education prepares us to be responsible participants of the Internet. This course proposes an evaluation system that matches the purpose of the course, where students learn how to be responsible judges of quality and helps them learn to be responsive to feedback as well. I can't imagine better skills to learn within the safe confines of a class, with a prof on hand to offer constructive feedback (including to those giving feedback).
Here's the syllabus for the course. Skip to the bottom for the section on grading. I'm happy for comments. And we (the students and I) will let you know how it works.
Subsequent addition (Aug 15): This post has garnered so much attention that I wrote a longer follow-up which explains the history and theory of evaluation motivating this experiment: http://www.hastac.org/blogs/cathy-davidson/crowdsourcing-grading-follow
ISIS 120S-01, English 173S-05: This is Your Brain on the Internet
Spring 2010 IMPS Space, Rm. 230, John Hope Franklin Center
This is Your Brain on the Internet is open to any student fascinated by how we come to know the world and how we may or may not know the world differently in the Information Age. Our quest in this course will be to explore many different, quirky, eccentric, and exceptional models of mind in order to force ourselves to think, together, about what models best suit our digital, interactive, collaborative age. Although we are in a great era of neuroscience and are learning more and more about our mental processing, what we do not know about how our brain works is infinitely more vast than what we know. Thus we make models to try to explain ourselves to ourselves. Every era (and the present is no exception) and every culture imagines its own models of mind. In the scientific method, this hypothesis then both shapes experiments and data collection and uses experimental findings and the data collected to test, refine, or (sometimes) refute the hypothesis.
This class advances an argument: We are living in one of the most momentous times of change in human history. We have changed. Now we need to name the paradigm that has already shifted. This class will be testing that argument in myriad ways. We will be thinking together about how we know the world, how we think, and how we think about thinking as individuals, as groups, as a culture, as subcultures, in a historical moment, as mediated by and through technology. The readings are intended as provocations. Some are evocative, some controversial, all have strong points of view, all are polemical in the sense that they advocate for models of mind, collaboration, interaction, and mediation. All are also situated in the sense that they do not look at cognition in an abstract sense, as divorced from social concerns, but as deeply rooted in cultural arrangements, so another focus of the course will be on new ways that humans interact with one another as friends, business partners, and members of a global information community. How are collaborations different when they are face-to-face than when virtual, mediated by technology? Our own classroom will move back and forth between actual and virtual experiences, including observation of highly complex collaborative environments (including choreography, improvisation, and other ways of interacting with and without words), some of which involve technology and some of which do not.
The course is conceived as a trans-disciplinary exploration in which we will consider the deep structure of cognition and community in a digital age. Well learn from theoretical and expressive books and articles ranging from neuroscience to films and literature, from various experimental and mainstream films as well as from a range of non-traditional sources (websites, interactive games and virtual environments, new media art exhibits, a backstage tour, conversations with social networking activists and community organizers, demonstrations by performance artists and illusionists, Virtual Reality tours, etc.) We will also learn from engaged collaboration (collaboration by difference) with others who have complementary skills, strengths, attitudes, and assumptions. Every student will have at least three opportunities to work with partner(s) in the class and learning to be conscious and conscientious about that interactive, collaborative process is one of the learning methods of this course.
Readings: Readings will be chosen by our student leaders from among the following but some other texts, films, experiences, websites, interactive projects, and so forth will be added or dropped by student leaders as the course unfolds.
Jean-Dominique Bauby, The Diving Bell and the Butterfly
*Norman Doidge, The Brain That Changes Itself
Cathy N. Davidson and David Theo Goldberg, The Future of Learning Institutions in a
Digital Age [online; free download from MIT Press]
*Anna Everett, ed., Learning Race and Ethnicity: Youth and Digital Media (MacArthur
Foundation Digital Media and Learning Series) [selections online]
*Temple Grandin, Animals in Translation
*Christopher Kelty, Two Bits: The Cultural Significance of Free Software [selections
*Daniel Levitin, This Is Your Brain on Music
Mark Haddon, The Curious Incident of the Dog in the Night-Time
*Jeff Hawkins, On Intelligence
*Tara McPherson, ed., Innovative Uses and Unexpected Outcomes (MacArthur
Foundation Digital Media and Learning Series) [selections online]
*Howard Rheingold, Smart Mobs
*Clay Shirky, Here Comes Everybody: The Power of Organizing Without Organizations
This course is student-driven: All classes will be lead by pairs of students who will also give us reading assignments (books, articles, websites, films) and writing/creating assignments (setting us ways to interact with the material prior to or in class as well as after it). The student leaders for each session will also evaluate every other students contributions, a process that will continue throughout the class, on our class Wordpress site. Discussing the role and purpose of evaluation to the learning process will be a key feature of "This Is Your Brain on the Internet" and is part of making all of us more responsible citizens of the interactive, customizing, crowdsourcing Information Age. One purpose of this course is for all of us to become used to peer evaluation, peer response, peer collaboration and to use these collective processes as aids toward our mutual learning goals. Students will be encouraged to respond back to the student leaders making the comments and to discuss these responses in class.
There will be no exams and no formal, final research papers required in this class. Any student who would like to write a final research paper can pitch an idea to the class. If accepted, the student will be invited to write the paper. In all other cases, students will work together on a final, collaborative multimedia online project that will be made available on a public website, probably the HASTAC (www.hastac.org) or the ISIS site.
Final Project: Students may come into the class with a final project in mind and find collaborative partners, or we can begin discussing ideas from the beginning. A class wiki will be set up for this purpose.
Student Participation in the Set-Up of the Class: The burden of customizing our WordPress site for the purposes of individual projects, of proposing other software possibilities or other technologies, will be shared across the students in the class. We will have laptop-required and laptop-free days. We will be blogging and twittering and facebooking to a group site. (Policy warning: I do not friend current students on Facebook.) Students will receive some grading points (see below) for contributions in this area. Students without tech skills will be urged to contribute in other ways--creatively, critically, in performance, offering feedback, and so forth. Collaboration by difference is a theme we will take seriously in the course.
Grading and Evaluation. After returning to teaching after several years as an administrator, I found grading to be the most outmoded, inconsequential, and irrelevant feature of teaching. Thus for ISIS 120, S 2010, all students will receive the grade of A if they do all the work and their peers certify that they have done so in a satisfactory fashion. If you choose not to do some of the assignments and receive a lower grade, thats permissible. You will be given a chart at the beginning of the course with every assignment adding up to 100 points. A conventional system will be assigned (95-100 points = A-, etc). We total the scores at the end and you get the points youve achieved. If, on any one assignment, peers rank the work unsatisfactory, you will either not be assigned any points for that assignment or you can submit a revised assignment in response to the class critique. Revision and resubmission results in full points. In other words, everyone who chooses to do the work to the satisfaction of his or her collaborative peers in the course will receive an A, but no one is required to do all of the work or to earn an A.
In lieu of a final exam, students will write an evaluation of the class (in addition to the university-required student evaluations). This will emphasize what you learned in the class, what you feel you accomplished (with "accomplished" self-defined). I will offer feedback on your self-assessment, amounting to an "evaluation" of your contribution to the experiences of, in Toffler's phrase "learning, unlearning, and relearning" that are central to "Your Brain on the Internet."
Cathy N. Davidson is co-founder of HASTAC, a 11,000+ network committed to new modes of collaboration, research, learning, and institutional change. Davidson teaches at Duke University where she co-directs the Ph.D. Lab in Digital Knowledge. She is author of The Future of Thinking: Learning Institutions for a Digital Age (with HASTAC co-founder David Theo Goldberg), and Now You See It: How the Brain Science of Attention Will Transform the Way We Live, Work, and Learn (Viking Press). She is co-PI on the HASTAC/MacArthur Foundation Digital Media and Learning Competitions. In October 2012, she and David Theo Goldberg, HASTAC's cofounders, were named Educator of the Year by the World Technology Network. NOTE: The views expressed in Cat in the Stack blogs and in NOW YOU SEE IT are solely those of the author and not of HASTAC, nor of any institution or organization. Davidson also writes on her own author blog, www.nowyouseeit.net . The paperback of Now You See It was published July 31, 2012 : http://tinyurl.com/bqquoaz