When I teach my undergraduate classes in "This Is Your Brain on the Internet" and "21st Century Literacies," we spend a lot of time on the multiple ways we contribute to collective knowledge by evaluating one another's work. Peer-review for digital communication, in other words. Here's a list of the main ways:
--informal commenting on one another's weekly blog posts
--formal commenting on the assigned posts by the week's two student peer leaders
--informal assessment/evaluation by students of the weekly peer leaders' presentations
--formal written assessment (with a checklist as well as a list of targeted questions plus an open essay section) during the juries of rough cuts of multimedia work
--formal written assessments of my teaching by the students
--formal, final written assessment that I give to each student of his or her own work for the entire course--an open letter of recommendation that the student can present to any future employer or graduate school (I make the terms of openness and the principles clear in the first paragraph).
In next iterations of the class, I hope we have co-developed peer-evaluation software to the extent that we can add leaderboards (crowdsourced assessment of contribution) and a badge system (that would account for all of the above plus add merit systems for less material contributions such as being an effective collaborator, an idea "fire starter," or a "closer").
Given that my teaching (like HASTAC itself) adapts the methods of open web collaborative development (the way the World Wide Web was formed and continues to be reformed---where possible [that's another blog post!]), how we work and collaborate and respond together, how we evaluate our work and give and take feedback so that work can be as good as it can be, is a necessity. Peer review is multifarious, conscientious, and constant because we want our peer-driven world to be better--more inspiring, exciting, and innovative--than the hierarchical world where an expert claims to know all, be in charge of all, and bestow that "all" upon passive learners. Peer review is necessary in a true, collaborative world.
So why, then, in so many discussions of digital scholarship and digital humanities is "peer review" implicitly or explicitly posed as the "opposite" of digital scholarship? I have many ideas about this and that's a good thing because soon I'll be part of a Mellon-funded project to explore this topic. Here are a few ornery insights and I hope I will get a lot of pushback:
--When "peer review" is equated with "tenure review," then there are good and bad reasons for making a distinction, for insisting digital scholarship doesn't play by old rules. Some tenure review is bogus and political, we all know that. Although, in the end, sometimes there is less of that on the peer-level than there would be in a normal corporation or a university if hierarchical "old boy" judgment, unsubstantiated by a peer community were activated. I have given a number of talks about how shocking it was to inherit the files of American Literature, to read decades' worth of reader's reports on articles, and see how often, before around 1970, articles were accepted or rejected on quite personal terms, based on who the author was, who the author studied with, who the author knew, and, even more shockingly, overtly what race, gender, or sexuality the author was said to possess. "XX, isn't that a Hebraic-sounding name?" one rejection report began. "He's a fairy," another said. Really. There was no sense of accountability to some anonymous, institutionally-approved standard. Those sound like the kind of anonymous, hateful comments people leave on blogs or YouTube, but these were delivered by the top people, the arbiters of tenure and promotion, in the whole profession. They wrote with a tacit idea that "we" knew who was or wasn't worth publishing. The "we" was the in-group then running AL; there were some horrible people calling the shots but, at the same time, there were also wonderful people, some of the finest and most generous and I read their comments too. But no one else did. Because it was a closed system. Abuses occur in open systems; abuses happen in closed systems. That's the point. The contemporary system of anonymous double-blind or single-blind and multiply-read peer review can stink . . . but even worse was the old-fashioned prejudice or the too-chummy Old Boy idea of "it's who you know/are/studied under that counts."
Sometimes, when I hear digital humanists talk as if digital publishing will "free" them from the conventions and prejudices of "peer review" I worry that we are going back to an Old Boy system, with far too much clubbiness for my comfort level, with everyone knowing everyone and having an idea of what really does or does not "count" as REAL digital humanities (but does he really write his own code?), where one-form of lock-box closed system replaces not a deficient system of peer review (and I'll be the first to say it has deficiencies) but returns us to an Old Boy system of who supports who, who thinks who is good, and who, well, doesn't really do the right kind of work, the kind we approve of. (Dear Reader: If this does not apply to you, please don't think it does. If it does apply to you, well, then shame on you!)
--When "a peer-reviewed scholarly journal" or "peer-reviewed university press monograph" is set up as "the problem" and online digital publishing is "the solution," we are settling on a totalizing definition of both that is good neither for peer-reviewed journals and monographs nor for the full, wide, open array of forms of open digital publishing that we should be promoting in all their diverse forms.
As someone who has spent a whole career going back and forth between scholarly publishing (with its systems and values and methods and practices) and trade publishing (ditto), and who blogs on the HASTAC site or the DML site or other sites virtually every day, well, let's just say I love to write and I love to write in all its forms. What is clear to me from this life of writing is its infinite variety. The path to a trade book in 2011 is very different from the one I took in 2001. As a historian of the book, I know that material conditions, historical circumstances, institutional priorities, fiduciary constraints, demographies of a profession, copyright, economics, and technological development all change how articles, books, or blogs, or tweets are produced, read, distributed, and paid for. The "profession of authorship" (a smaller and smaller category if by "profession" we mean one supports oneself by this trade) is multifarious and so are the forms of publication--digital or otherwise--that authorship takes. Why over-generalize? That's specious and simply not useful. As with any other over-generalization of The Other, it reduces not just the "problem" but limits the "solution."
If we want digital scholarship to "count" for tenure within currently existing institutions of tenure, then the fact that it is peer reviewed--that it has to pass through a certain process of feedback, evaluation, and revision (even if it holds to the internet dictum of "publish first, revise later") is important; specifying the means of evaluation and assessment should be part of the portfolio that leads to tenure. It may not be traditional peer review (whatever "traditional" means there--each press and journal has its own forms of "traditional") but that makes it even more urgent to spell out its process, the means by which a given community ensures the quality and calibre of contribution, decides on its norms of openness and expertise.
Those levels and forms are exactly what I want my undergraduates to master as they continue to be participatory actors in the world of online publishing in any form. My undergraduates learn the responsibility of peer-to-peer contribution. They learn quality can be enhanced by the multiple ways that they review and comment on and collaborate on one another's work. They do not see the product of our collaborative labor as without peer review but, on the contrary, hyper-peer reviewed in multiple and complex forms. So is much of digital humanities, from original grant applications that lead to the creation of the hardware, software, archive, data bases, and so forth to the decisions about what does or does not get published, how, why, in what form.
Many digital humanities projects have peer-reviewed and authorized sections and then open forums, crowdsouced contributions and so forth. This HASTAC blog, for example, is monitored in the sense that it would be taken down if I were writing irrelevant spam or violated the terms of our community--but it is not referreed for content. I write what I want (so long as I hold to community standards of being a member, a "peer" in this community) and so can anyone who registers to the HASTAC site and who thereby signs on to our community standards. That is what I would call "loose" peer review! Wikipedia, which has no formal institutionalized traditional standards for peer review, is what I would call "intensively" peer reviewed, by the larger community of readers as well as by those 1200 volunteer editors who are always telling us about our "stubs" or our "undocumented" work and so forth.
My point is that the terms are clunky. Does digital publishing need peer review? Yes, no, maybe: depends on what you mean by "digital" and "publishing" and "peer" and "review." Community forms of scholarship are not equivalent to openness. Independent publishing is by no means equivalent of "open" or not "peer reviewed."
Here's another example, from the "self-publishing" forms of an older, pre-digital era. Many years ago, when I was publishing short stories, I used to joke that they were all published at My Friend's Basement Press, tiny presses that some dedicated soul spent hours and hours a day on, for no profit, no gain, and the esteem only of those of us who read and wrote for these tiny literary magazines. They (some were even mimeographed) were forerunners of today's digital publishing websites. Were they free and open? No way! The editors of these magazines had more prejudices than you could shake a stick at. At the time, I was writing Raymond Carver-esque somnambulistic, anesthetized prose and some editors ate that up and others despised it. I knew which was which and increased my chances of publication considerably. As is the case with all publishing, including digital. There are values and norms embedded in all our practices, whether we are talking about the institutionalized practices of peer review at research universities and university presses--or the implicit but still institutional practices at My Friend's Basement Press.
I teach these practices to my students. We practice these practices because we all want to be better citizens of the open web. But we also need to maintain distinctions. Peers review in many ways for many purposes. Any system, including the most open (as the "Comments" on YouTube or anywhere else remind us), is susceptible to abuse. Or to creativity, quality, innovation, and sometimes, when we do it right, inspiration. Blurring distinctions, and going immediately to "good" and "bad" doesn't do anyone any good. Demonizing or celebrating without looking carefully of the history and purpose of our practice (new or old) does nothing but lead to the unwitting replication of the very forms of Old Boyism that we thought, mercifully, hopefully, optimistically, and sometimes oh so naively, that the open web might leave behind.
* * *
AUTHOR'S NOTE: THE 2011 HASTAC ANNUAL CONFERENCE, DEC 2-3 AT THE UNIVERSITY OF MICHIGAN (ANN ARBOR), WILL ADDRESS MANY OF THESE ISSUES AS ITS FOCUS IS "DIGITAL SCHOLARLY COMMUNICATION"--WITH EACH ONE OF THOSE TERMS BEING THE OBJECT OF VIGOROUS DISCUSSION, DEBATE, AND CREATIVITY. FOR MORE INFORMATION, SEE: http://www.hastac.org/blogs/admin/call-proposals-2011-hastac-conference-details
Cathy N. Davidson is co-founder of HASTAC, and author of The Future of Thinking: Learning Institutions for a Digital Age (with HASTAC co-founder David Theo Goldberg), and the forthcoming Now You See It: How the Brain Science of Attention Will Transform the Way We Live, Work, and Learn (publication date, Viking Press, August 18, 2011). below. For an early, prepublication review of Now You See It in Bloomberg BusinessWeek, click here.
A starred review in the May 30 Publisher's Weekly notes: "Davidson has produced an exceptional and critically important book, one that is all-but-impossible to put down and likely to shape discussions for years to come." PW named it one of the "top 10 science books" of the Fall 2011 season.
Image credit: http://www.flickr.com/photos/wakingtiger/3157622458/