Blog Post

Does Digital Publishing Need Peer Review?

Does Digital Publishing Need Peer Review?

When I teach my undergraduate classes in "This Is Your Brain on the Internet" and "21st Century Literacies," we spend a lot of time on the multiple ways we contribute to collective knowledge by evaluating one another's work. Peer-review for digital communication, in other words.   Here's a list of the main ways:

--informal commenting on one another's weekly blog posts

--formal commenting on the assigned posts by the week's two student peer leaders

--informal assessment/evaluation by students of the weekly peer leaders' presentations

--formal written assessment (with a checklist as well as a list of targeted questions plus an open essay section) during the juries of rough cuts of multimedia work

--formal written assessments of my teaching by the students

--formal, final written assessment that I give to each student of his or her own work for the entire course--an open letter of recommendation that the student can present to any future employer or graduate school (I make the terms of openness and the principles clear in the first paragraph).  

 

In next iterations of the class, I hope we have co-developed peer-evaluation software to the extent that we can add leaderboards (crowdsourced assessment of contribution) and a badge system (that would account for all of the above plus add merit systems for less material contributions such as being an effective collaborator, an idea "fire starter," or a "closer").  

 

Given that my teaching (like HASTAC itself) adapts the methods of open web collaborative development (the way the World Wide Web was formed and continues to be reformed---where possible [that's another blog post!]), how we work and collaborate and respond together, how we evaluate our work and give and take feedback so that work can be as good as it can be, is a necessity.   Peer review is multifarious, conscientious, and constant because we want our peer-driven world to be better--more inspiring, exciting, and innovative--than the hierarchical world where an expert claims to know all, be in charge of all, and bestow that "all" upon passive learners.   Peer review is necessary in a true, collaborative world.  

 

So why, then, in so many discussions of digital scholarship and digital humanities is "peer review" implicitly or explicitly posed as the "opposite" of digital scholarship?    I have many ideas about this and that's a good thing because soon I'll be part of a Mellon-funded project to explore this topic.    Here are a few ornery insights and I hope I will get a lot of pushback:

 

--When "peer review" is equated with "tenure review," then there are good and bad reasons for making a distinction, for insisting digital scholarship doesn't play by old rules.   Some tenure review is bogus and political, we all know that.   Although, in the end, sometimes there is less of that on the peer-level than there would be in a normal corporation or a university if hierarchical "old boy" judgment, unsubstantiated by a peer community were activated.  I have given a number of talks about how shocking it was to inherit the files of American Literature, to read decades' worth of reader's reports on articles, and see how often, before around 1970, articles were accepted or rejected on quite personal terms, based on who the author was, who the author studied with, who the author knew, and, even more shockingly, overtly what race, gender, or sexuality the author was said to possess.   "XX, isn't that a Hebraic-sounding name?" one rejection report began.  "He's a fairy," another said.   Really.  There was no sense of accountability to some anonymous, institutionally-approved standard.  Those sound like the kind of anonymous, hateful comments people leave on blogs or YouTube, but these were delivered by the top people, the arbiters of tenure and promotion, in the whole profession.  They wrote with a tacit idea that "we" knew who was or wasn't worth publishing.   The "we" was the in-group then running AL; there were some horrible people calling the shots but, at the same time, there were also wonderful people, some of the finest and most generous and I read their comments too.   But no one else did.  Because it was a closed system.  Abuses occur in open systems; abuses happen in closed systems.  That's the point.  The contemporary system of anonymous double-blind or single-blind and multiply-read peer review can stink . . .  but even worse was the old-fashioned prejudice or the too-chummy Old Boy idea of "it's who you know/are/studied under that counts."  

       Sometimes, when I hear digital humanists talk as if digital publishing will "free" them from the conventions and prejudices of "peer review" I worry that we are going back to an Old Boy system, with far too much clubbiness for my comfort level, with everyone knowing everyone and having an idea of what really does or does not "count" as REAL digital humanities (but does he really write his own code?), where one-form of lock-box closed system replaces not a deficient system of peer review (and I'll be the first to say it has deficiencies) but returns us to an Old Boy system of who supports who, who thinks who is good, and who, well, doesn't really do the right kind of work, the kind we approve of.     (Dear Reader:  If this does not apply to you, please don't think it does.   If it does apply to you, well, then shame on you!)

 

--When "a peer-reviewed scholarly journal" or "peer-reviewed university press monograph" is set up as "the problem" and online digital publishing is "the solution," we are settling on a totalizing definition of both that is good neither for peer-reviewed journals and monographs nor for the full, wide, open array of forms of open digital publishing that we should be promoting in all their diverse forms.  

As someone who has spent a whole career going back and forth between scholarly publishing (with its systems and values and methods and practices) and trade publishing (ditto), and who blogs on the HASTAC site or the DML site or other sites virtually every day, well, let's just say I love to write and I love to write in all its forms.  What is clear to me from this life of writing is its infinite variety.  The path to a trade book in 2011 is very different from the one I took in 2001.  As a historian of the book, I know that material conditions, historical circumstances, institutional priorities, fiduciary constraints, demographies of a profession, copyright, economics, and technological development all change how articles, books, or blogs, or tweets are produced, read, distributed, and paid for.  The "profession of authorship" (a smaller and smaller category if by "profession" we mean one supports oneself by this trade) is multifarious and so are the forms of publication--digital or otherwise--that authorship takes.   Why over-generalize?   That's specious and simply not useful.  As with any other over-generalization of The Other, it reduces not just the "problem" but limits the "solution."  

 

If we want digital scholarship to "count" for tenure within currently existing institutions of tenure, then the fact that it is peer reviewed--that it has to pass through a certain process of feedback, evaluation, and revision (even if it holds to the internet dictum of "publish first, revise later") is important; specifying the means of evaluation and assessment should be part of the portfolio that leads to tenure.  It may not be traditional peer review (whatever "traditional" means there--each press and journal has its own forms of "traditional") but that makes it even more urgent to spell out its process, the means by which a given community ensures the quality and calibre of contribution, decides on its norms of openness and expertise.  

 

Those levels and forms are exactly what I want my undergraduates to master as they continue to be participatory actors in the world of online publishing in any form.  My undergraduates learn the responsibility of peer-to-peer contribution.   They learn quality can be enhanced by the multiple ways that they review and comment on and collaborate on one another's work.   They do not see the product of our collaborative labor as without peer review but, on the contrary, hyper-peer reviewed in multiple and complex forms.   So is much of digital humanities, from original grant applications that lead to the creation of the hardware, software, archive, data bases, and so forth to the decisions about what does or does not get published, how, why, in what form. 

 

Many digital humanities projects have peer-reviewed and authorized sections and then open forums, crowdsouced contributions and so forth.  This HASTAC blog, for example, is monitored in the sense that it would be taken down if I were writing irrelevant spam or violated the terms of our community--but it is not referreed for content.  I write what I want (so long as I hold to community standards of being a member, a "peer" in this community) and so can anyone who registers to the HASTAC site and who thereby signs on to our community standards.   That is what I would call "loose" peer review!   Wikipedia, which has no formal institutionalized traditional standards for peer review, is what I would call "intensively" peer reviewed, by the larger community of readers as well as by those 1200 volunteer editors who are always telling us about our "stubs" or our "undocumented" work and so forth.  

 

My point is that the terms are clunky.   Does digital publishing need peer review?  Yes, no, maybe:  depends on what you mean by "digital" and "publishing" and "peer" and "review."  Community forms of scholarship are not equivalent to openness.  Independent publishing is by no means equivalent of "open" or not "peer reviewed."

 

Here's another example, from the "self-publishing" forms of an older, pre-digital era.  Many years ago, when I was publishing short stories, I used to joke that they were all published at My Friend's Basement Press, tiny presses that some dedicated soul spent hours and hours a day on, for no profit, no gain, and the esteem only of those of us who read and wrote for these tiny literary magazines.   They (some were even mimeographed) were forerunners of today's digital publishing websites.   Were they free and open?  No way!   The editors of these magazines had more prejudices than you could shake a stick at.   At the time, I was writing Raymond Carver-esque somnambulistic, anesthetized prose and some editors ate that up and others despised it.   I knew which was which and increased my chances of publication considerably.   As is the case with all publishing, including digital.  There are values and norms embedded in all our practices, whether we are talking about the institutionalized practices of peer review at research universities and university presses--or the implicit but still institutional practices at My Friend's Basement Press.  

 

I teach these practices to my students.  We practice these practices because we all want to be better citizens of the open web.   But we also need to maintain distinctions.   Peers review in many ways for many purposes.  Any system, including the most open (as the "Comments" on YouTube or anywhere else remind us), is susceptible to abuse.  Or to creativity, quality, innovation, and sometimes, when we do it right, inspiration.   Blurring distinctions, and going immediately to "good" and "bad" doesn't do anyone any good.  Demonizing or celebrating without looking carefully of the history and purpose of our practice (new or old) does nothing but lead to the unwitting replication of the very forms of Old Boyism that we thought, mercifully, hopefully, optimistically, and sometimes oh so naively, that the open web might leave behind.  




*   *   *



AUTHOR'S NOTE:   THE 2011 HASTAC ANNUAL CONFERENCE, DEC 2-3 AT THE UNIVERSITY OF MICHIGAN (ANN ARBOR), WILL ADDRESS MANY OF THESE ISSUES AS ITS FOCUS IS "DIGITAL SCHOLARLY COMMUNICATION"--WITH EACH ONE OF THOSE TERMS BEING THE OBJECT OF VIGOROUS DISCUSSION, DEBATE, AND CREATIVITY.   FOR MORE INFORMATION, SEE:  http://www.hastac.org/blogs/admin/call-proposals-2011-hastac-conference-details
------------

Cathy N. Davidson is co-founder of HASTAC, and author of The Future of Thinking:  Learning Institutions for a Digital Age (with HASTAC co-founder David Theo Goldberg), and the forthcoming Now You See It:  How the Brain Science of Attention Will Transform the Way We Live, Work, and Learn (publication date, Viking Press, August 18, 2011).  below. For an early, prepublication review of Now You See It in Bloomberg BusinessWeek, click here.

A starred review in the May 30 Publisher's Weekly notes:  "Davidson has produced an exceptional and critically important book, one that is all-but-impossible to put down and likely to shape discussions for years to come." PW named it one of the "top 10 science books" of the Fall 2011 season.

For more information, visit www.nowyouseeit.net or order on Amazon.com by clicking on the book below.

 

  [NYSI cover]

Image credit: http://www.flickr.com/photos/wakingtiger/3157622458/

79

10 comments

My Department and even our President turned me down for promotion this year, citing that my digital work needed rigorous peer review (despite have 5 of the top Digital Humanists review my work 2 years ago and give it the stamp of peer review approval). Of course, I'm fighting this with my union. But let me just say that the "good ol boys" network is alive and well in most departments. What they don't understand, they turn down without thoughts towards servicing the needs of our students. I don't want to publish an article in a print journal. That would take 3 years. My work needs to be out there now.

Also, let me say that peer review doesn't work. I just attended a conference where I listened to a paper that was full of errors and mis-statements. At the panel's conclusion, I spoke briefly with the presenter, who informed me that despite my reservations about her work, she had already published this paper as a longer article. When I checked with the handful of experts in this particular field, no one could say that they had been the expert peer reviewer. So, the editor sent the article to general readers in the period instead of experts. That's odd considering all of us experts were cited in the article; why not ask us to read it? But, now, that article will be out there for graduate students to continue the erroneous line of misquotations and shoddy research performed by this tenured professor.

I wax poetic about these issues but it's fairly serious that peer review is broken, but yet tenure and promotion committees rely so heavily on the structure. I'd rather post my articles and book chapters to my blog and get feedback from a myriad of sources than run it through double blind peer review. After last year, I have no respect for the supposed anonymity of peer review. It allows senior people to behave abhorently.  And, I'm determined *not* to replicate that.

In scholarly editing circles, we are contending with opening up digital archives to the masses to crowdsource information. There has been much kerfuffle about controlling those masses and their mis-information. It'll be interesting to see how this debate develops. Meanwhile, I'm going ahead with my crowdsourced digital archive/edition regardless what my department values.

63

I sympathize with you, empathize, and hear you.  Of course I don't know your cv or your university or any of the details but, as I said, abuse happens in all systems and closed systems of any kind can be particularly abusive.   My gut is that, as universities cut back, the new, the unusual, the unexpected, and the provocative will pay a higher price even than before.
 

Also, there are pioneers in every field--and they often do not do well. That is one of the other key reasons we created HASTAC, to raise awareness about the validity of new forms of scholarship.   David and I and the many very distinguished people (by any standards) on the first founding steering committee of HASTAC essentially said, "We are putting our reputations on this line."  It certainly doesn't help every case but it helps a field to grow, just as people like Annette Kolodny helped in their day. Annette is  a first-wave feminist with a CV as long as anyone's I know, a brilliant woman turned down for tenure at the University of New Hampshire which, as I recall, had not then tenured a Jew or a woman . . .    She sued, spent years in litigation, and years (as did I) in adjunct and temporary positions.  She went on to be Dean at the University of Arizona.   If you looked at the end of her career, you would not know that her lawsuit was a game changer for feminist scholars.   There are many other examples.   Digital publishing may be one version now.  Again, I know nothing about the particulars of your case so I am talking generally about the way pioneers often suffer and yet, also, often open up new ways.
 

The MLA, and HASTAC contributed to this, worked on a wiki for tenure and the different things that should count for tenure that do not happen to be peer-reviewed monographs or scholarly articles.  If you are still protesting the decision, and if you have not already, you should consult that document as it was worked on by many major players in the profession.  
 

What I think hurts those who publish digitally, though, is making too many generalizations about the forms of digital publishing and about the forms of peer-reviewed scholarship.  The more one over-generalizes, the more one flattens the diversity of our field(s) and leaves an opening for others to find fault rather than to see the care we give to what we do and why.
 

"Those Masses and Their Mis-Information":   I tend to trust those masses who contribute to digital scholarship on line a lot.  In my experience, their voluntary, under-recognized, and un-recognized contributions spring from a place of generosity, not cruel and self-satisfying judgment.   I hear you.   I very much hope your administrators do.   Thank you for your thoughtful and passionate response and good luck with your case.

66

p.s   Since I wrote the comment, we had a bit of drama here at HASTAC--I inadvertently deleted my original post, your comment, my reply.   Whew.  The amazing Ruby reconstructed it all and I think it is all here.  However, in between that trauma and its recovery, I see you posted your full name and your bio.   Really impressive.   And, yes, YOU are an undisputed expert in this field.   I'm so sorry your colleagues did not see that.   Feel free to refer anyone from your university to this post.   Decisions are always local and situational but, looking at your bio, I see an estimable person who has made a significant contribution to the field.   At most research one universities, that is the most important grounds for tenure.   I am happy to write this publicly and feel free to use it if it serves your purposes.    Good luck to you, Katherine. 

62

...for the support. I received tenure after a year-long battle and am thankful that I now have that type of protection.

I will continue my work. I will continue advocating new teaching and learning strategies for my university's type of students. I will continue to advocate for altering the standards of publishing in my department. I won't give up because I believe in everything that I'm doing and am grateful to HASTAC for its voice. 

I look forward to what's to come in the profession!

61

This is an incredible resource, and one that I've consulted several times. I was even at the MLA seminar on evaluating digital scholarship as one of the test cases when the MLA first took it up. Let me just say,

59

Cathy -- thanks for this post. I'm in the very early stages of an academic career, but in my minimal experience with traditional peer review, I've rarely received helpful comments on a draft. In fact, in his experience publishing in reputable philosophy journals, my partner Phil has had to fight anonymous reviewers on *simple facts*. (As Kathy points out in her comment, this is likely because editors are sending articles to non-experts -- indeed, both Phil and I have been sent articles to review on topics of which we're laughably ignorant.) As if this isn't frustrating enough, the anonymous reviewers are often dogmatic and rude in their (factually erroneous) assertions. NO! NO! NO!, they write in all caps, HOW COULD YOU THINK THIS?! Talk about YouTube-style commenting.

By contrast, I currently have an article up for what's being called a "crowd review" of the forthcoming issue of postmedieval. In the one (very thoughtful, very thorough) comment it's received so far, the reviewer -- someone I follow in various digital networks -- seems to have taken his task to be helping me strengthen my work so that it makes a better contribution to the academic community (how novel!).

Although the site has only been live a few days, I'd be shocked if we see YouTube-ish rudeness during the postmedieval crowd review, partly because of the publicness, but more importantly because that publicness is plugged into established networks. Eileen Joy, editor of postmedieval, smartly rolled out the crowd review process on In the Middle, a well-established blog for all things medieval, thereby leveraging the collective intelligence of an existing community. Personally, the open review is only the end of a long process of peer review, since I've blogged parts of the article; I've kept all my notes, including my annotations on the article's entire bibliography, on my public wiki; I've tweeted about it. As I've pointed out elsewhere, every part of my writing process has been under some form of review, done in conversation with many incredibly generous (and more knowledgeable) peers (including Kathy-with-a-K, whom I've been lucky to have in my digital network for a few years now). Although the end result will (hopefully) be a printed article, I was, I think, practicing digital scholarship through and through. 

Which is to say: when the kinds of policies you've tested in your classroom are implemented more broadly, and at all levels of the scholarly process (both individual and collective), digital scholarship simply *is* peer review. There are still lots of glitches, and integration between the different sites where review happens is far from perfect; but it's a start. 

As always, thanks for the food for thought.

60

Great comment, Whitney.  Thanks.  And no argument with the positives of online peer review which, of course, is what HASTAC is and how I teach, taking in these examples deeply.  I dedicate just about my entire life to the basic HASTAC principle of adopting the principles of open web development to education, K-20.

 

 At the same time, I have no way of knowing how representative Phil's experience is.   Of course there are idiots.  I didn't write Revolution and the Word as a dissertation because I knew no one else in America was doing anything like it and knew I needed a "literary" dissertation to get a job.  That was a long time ago, and I don't think I was wrong.  On the other hand, I edited AL for a decade and some reader's reports were trivial or wrong or silly but many were pretty amazing and I read some of the most selfless, amazing, giving, useful, smart comments, offered by senior people to graduate students and assistant profs, and the same was true when I was on the Duke U Press board and many others. I know senior scholars who are selfless and junior who are incredibly self-centered---and vice versa.   Digital scholars who are rigid and traditional scholars who prize diversity--and vice versa.  In my experience, being digital does not necessarily make you pure and right and open to new ideas.   Sadly.   I just don't believe these things are binaries.  If they were, revolution would be easier and there would be never be formerly-young-radicals-who-now-enjoy-their-own-superiority.  Sigh.   I've known far, far too many of those.  

 

I love what you write and I know we agree on most of it.   A point I want to emphasize, though, is that different systems are susceptible to different kinds of abuse, different hierarchies can emerge from our smugness about what is different from what we are, and I would absolutely love it if the future of the profession were as open as the open web---and not another closed system that exists largely to replicate itself.  If every digital humanist were perfect, replication would be lovely . . . but I'm not sure we all are wondrous, any more than our analog counterparts were and that is why we need all those "eyeballs" and all that "diversity," so far as I'm concerned.  

 

Thanks so much for writing.  Together, I believe (this is my utopianism) we can work towards better ways of working together and a forum like this is as good a place as any to participate in this never-ending process.  

66

"In my experience, being digital does not necessarily make you pure and right and open to new ideas."

>> So true! It's a community effort. 

Phil's experience has a lot to do with the standards of his discipline, or at least of the subfield he's involved in. Which just underscores the fact that there's no one formula (or medium) for making discourse more open. 

52

The problem with Peer Review is Clarke's First Law:

When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.

The very nature of academic institutions requires them to be bureaucratic, and the first Law of Bureaucracy is that Bureaucracies exists for their own survival.

What the Internet has done is broaden the sources of evaluation so that the Peer Review Bureaucracy now is accountable to additional sources and something approaching reality.  Holding that Bureaucracy accountable to outside sources for their biases does create change without either blowing up the system or your becoming a martyr.

For your reference:

Rules of Bureaucracy

Rule 1 – Bureaucracies exists for their own survival.

Rule 2 – Bureaucracies are stronger than the individuals who are in them and stronger than your ability to change them, which is why it is beneficial to be part of a bureaucracy.

Rule 3 – The best way to bring about change is to use the weight and mechanisms of the bureaucracy against itself, or the weight and mechanisms of one bureaucracy against another. You have to be creative and smarter than the bureaucracy to bring about change, but it is easier and more effective than becoming a martyr.  (In the movie "Wargames" the Whopper Computer was defeated by asking it to play itself in game of Tic-Tac-Toe.)

Rule 4 – To fight a "bureaucracy" on issues, you have to overwhelm them with details.

Rule 5 – A True Bureaucrat creates new rules, procedures, and organizational changes with limited feedback for the primary purpose of making sure that the rest of the Bureaucracy knows that the True Bureaucrat is in charge.

Rule 6 – The pen may be mightier than the sword (as long as you keep your head attached), but the electron (Internet) is faster than the pen.

 


60

I love these provocations.  Thank you, both Whitney and Allan.  

64