What reputation systems can teach us about badge system design
Dan Hickey's recent post Research Design Principles for Studying Learning with Badges prompted me to reflect on the distinction between reputation systems and credentialing / badge systems. Why does this distinction matter? In part because the research on recommender and reputation systems that underpin the "anywhere, anytime" learning of the Web have much to teach us about how people participate online, how they establish trust, and how they find, define, and measure quality -- things we care about in badge system design.
Likewise, the information science literature has much to learn from education and learning science disciplines. We need both bodies of research (and many others) if we are to design effective badge systems that genuinely make learning better for the maximum number of learners.
The most important reason for this distinction, though, is because there are asymmetric power-law distributions in online reputation systems and by paying attention to what is already known about technology-mediated social participation, we can consider how we might inadvertently replicate inequity through badge system design. Maybe that's a topic for another post. But there is an implied argument among advocates of reputation-based or peer learning that crowdsourcing credentials is better than the traditional system of credentialing.
Crowdsourcing is certainly different, and many of us in the Badges for Lifelong Learning initiative are advocates of peer-based learning and assessment. But if we care about the democratization of learning anywhere, anytime, we should not assume that crowdsourcing eliminates problems of equity, or that it is a superior way of identifying what is good and censuring what is bad. There is more to it than that, and putting the topic on the table will make badge system design better. Even better, there is plenty of research in the reputation systems literature to guide this conversation.
Anyway, I digress! The point of this post is to respond to Michael Olneck's question "Can someone more fully explain to me the difference between a “reputation system” and a “credentialing” system?"
Michael goes on to say that "Credentials have power because they have reputations of validly indexing real qualities and capabilities. They are awarded by education institutions, which are “chartered” to produce “graduates” who are presumed to have particular qualities and capabilities, and are treated as if they actually have these. If crowdsourced reputational testimonials come to be trusted, and used to make selections, then they will have become “credentials.” [See John W. Meyer, “The Charter: Conditions of Diffuse Socialization in Schools,” In W.R. Scott, Social Processes and Social Structures: An Introduction to Sociology HRW, 1970.]"
I'm not convinced I see the natural progression between crowdsourced reputational testimonials and credentials. If that were true, then a seller's reputation in eBay's Feedback Forum would be the same as a credential. Likewise, the top 10 search results in Google would be credentials. I'm going to try and tease out the distinction using Michael's comment that credentials have "reputations for validly indexing real qualities and capabilities." I think there's more to credentials than that, and thinking about this will help identify (clarify?) what badge systems do that make them unique.
But first, here are a few ways to think about reputation systems:
Paul Resnick's ACM article, Reputation Systems, has a simple technical definition: "A reputation system collects, distributes, and aggregates feedback about participants’ past behavior." There are different kinds of reputation systems, including eBay, which is based on buyers and sellers rating each other. Some reputation systems are based on sophisticated algorithms, like Google's Page Rank. Stack Overflow's reputation system is based on an algorithm that uses points, tags, and badges. I think of these as socio-technical systems designed to aid us in making sense of the (often overwhelming) democratizing effect of the Internet.
Manifesto for a Reputation Society is also a useful primer for thinking about reputation systems. In their paper, Hassan Masum and Yi–Cheng Zhang write that "Reputation is a judgment of quality." In the traditional definition of a credential, it is awarded based on more than just judgment. A credential has been vetted by a so-called higher standard of de facto authority that defines some measure of competence or quality. Badge systems that function with many of the affordances of reputation systems have the curious task of (or making an argument for) formalizing judgment. I do wonder how the validity of that claim -- that peer judgement is a legitimate indicator of competence or skill -- can stand up as a credential outside whatever community bestowed it.
Masum and Zhang write, "Being human, each of us has many limitations: time, access, ability, and experience. The main goal of developing enhanced reputation filters is to do as much as possible despite our individual limitations — to cooperatively pierce the veils of deception, mediocrity, and banality."
How would you write those sentences to apply to a reputation-based or crowdsourced credentialing system? If reputation systems have been designed to help us identify what is good and censure what is bad, how do credentials fit into that? Why is a credential more effective at validating what a learner knows than pure reputation? How does a credential help "pierce the veils of deception, mediocrity, and banality?" And back to those power laws. How might web-scale peer-based or crowdsourced credentialing be influenced by power laws? If those credentials become meaningful, what rises to the top? What gets left behind? Or more importantly, who?
A credential implies authority, and some kind of qualifying act. Also, evidence, proof, legitimacy, credibility, validity, reliability, endorsement -- the usual suspects in reputation systems. In a badge system, what bumps it from reputation to credential? How much authentification is needed for that credential to hold up in, say, less receptive environments? Once you move outside your sphere of reputation, what kinds of tests does that credential have to pass in order for it to truly matter?
Perhaps you could plot different badge systems on a spectrum of reputation-centric participation and see that it morphs into a credential when there are qualifying acts and prescribed pathways. Maybe some expert-level recognition also needs to be there, to assess the qualifying act and legitimize it. Or maybe a highly reputable endorsement could trump any "soft spots" in the credential system.
Likewise, if you have thick reputation at every level of the badge system, could you overlook the lack of qualifying acts? For example, if the issuer, earner, and endorser are all credible and trusted, and have trusted relationships with the badge "accepter" or consumer, maybe the badge system could produce credentials that are carried more by reputation than not.
I haven't come across anything in the literature that describes the difference between reputation systems and credential / badge systems, so thanks to Michael for pushing me to think about this more. Chime in if you have some thoughts about the differences, and what those differences might mean for badge system design.