What reputation systems can teach us about badge system design

Dan Hickey's recent post Research Design Principles for Studying Learning with Badges  prompted me to reflect on the distinction between reputation systems and credentialing / badge systems. Why does this distinction matter? In part because the research on recommender and reputation systems that underpin the "anywhere, anytime" learning of the Web have much to teach us about how people participate online, how they establish trust, and how they find, define, and measure quality -- things we care about in badge system design.

Likewise, the information science literature has much to learn from education and learning science disciplines. We need both bodies of research (and many others) if we are to design effective badge systems that genuinely make learning better for the maximum number of learners. 

The most important reason for this distinction, though, is because there are asymmetric power-law distributions in online reputation systems and by paying attention to what is already known about technology-mediated social participation, we can consider how we might inadvertently replicate inequity through badge system design. Maybe that's a topic for another post. But there is an implied argument among advocates of reputation-based or peer learning that crowdsourcing credentials is better than the traditional system of credentialing.

Crowdsourcing is certainly different, and many of us in the Badges for Lifelong Learning initiative are advocates of peer-based learning and assessment. But if we care about the democratization of learning anywhere, anytime, we should not assume that crowdsourcing eliminates problems of equity, or that it is a superior way of identifying what is good and censuring what is bad. There is more to it than that, and putting the topic on the table will make badge system design better. Even better, there is plenty of research in the reputation systems literature to guide this conversation. 

Anyway, I digress! The point of this post is to respond to Michael Olneck's question "Can someone more fully explain to me the difference between a “reputation system” and a “credentialing” system?" 

Michael goes on to say that  "Credentials have power because they have reputations of validly indexing real qualities and capabilities. They are awarded by education institutions, which are “chartered” to produce “graduates” who are presumed to have particular qualities and capabilities, and are treated as if they actually have these. If crowdsourced reputational testimonials come to be trusted, and used to make selections, then they will have become “credentials. [See John W. Meyer, “The Charter: Conditions of Diffuse Socialization in Schools,” In W.R. Scott, Social Processes and Social Structures: An Introduction to Sociology HRW, 1970.]"

I'm not convinced I see the natural progression between crowdsourced reputational testimonials and credentials. If that were true, then a seller's reputation in eBay's Feedback Forum would be the same as a credential. Likewise, the top 10 search results in Google would be credentials. I'm going to try and tease out the distinction using Michael's comment that credentials have "reputations for validly indexing real qualities and capabilities." I think there's more to credentials than that, and thinking about this will help identify (clarify?) what badge systems do that make them unique.

But first, here are a few ways to think about reputation systems:

Paul Resnick's ACM article, Reputation Systems, has a simple technical definition: "A reputation system collects, distributes, and aggregates feedback about participants’ past behavior." There are different kinds of reputation systems, including eBay, which is based on buyers and sellers rating each other. Some reputation systems are based on sophisticated algorithms, like Google's Page Rank. Stack Overflow's reputation system is based on an algorithm that uses points, tags, and badges. I think of these as socio-technical systems designed to aid us in making sense of the (often overwhelming) democratizing effect of the Internet. 

Manifesto for a Reputation Society is also a useful primer for thinking about reputation systems. In their paper, Hassan Masum and Yi–Cheng Zhang write that "Reputation is a judgment of quality." In the traditional definition of a credential, it is awarded based on more than just judgment. A credential has been vetted by a so-called higher standard of de facto authority that defines some measure of competence or quality. Badge systems that function with many of the affordances of reputation systems have the curious task of (or making an argument for) formalizing judgment. I do wonder how the validity of that claim -- that peer judgement is a legitimate indicator of competence or skill -- can stand up as a credential outside whatever community bestowed it.  

Masum and Zhang write, "Being human, each of us has many limitations: time, access, ability, and experience. The main goal of developing enhanced reputation filters is to do as much as possible despite our individual limitations — to cooperatively pierce the veils of deception, mediocrity, and banality." 

How would you write those sentences to apply to a reputation-based or crowdsourced credentialing system? If reputation systems have been designed to help us identify what is good and censure what is bad, how do credentials fit into that? Why is a credential more effective at validating what a learner knows than pure reputation? How does a credential help "pierce the veils of deception, mediocrity, and banality?" And back to those power laws. How might web-scale peer-based or crowdsourced credentialing be influenced by power laws? If those credentials become meaningful, what rises to the top? What gets left behind? Or more importantly, who?

A credential implies authority, and some kind of qualifying act. Also, evidence, proof, legitimacy, credibility, validity, reliability, endorsement -- the usual suspects in reputation systems. In a badge system, what bumps it from reputation to credential? How much authentification is needed for that credential to hold up in, say, less receptive environments? Once you move outside your sphere of reputation, what kinds of tests does that credential have to pass in order for it to truly matter?

Perhaps you could plot different badge systems on a spectrum of reputation-centric participation and see that it morphs into a credential when there are qualifying acts and prescribed pathways. Maybe some expert-level recognition also needs to be there, to assess the qualifying act and legitimize it. Or maybe a highly reputable endorsement could trump any "soft spots" in the credential system. 

Likewise, if you have thick reputation at every level of the badge system, could you overlook the lack of qualifying acts? For example, if the issuer, earner, and endorser are all credible and trusted, and have trusted relationships with the badge "accepter" or consumer, maybe the badge system could produce credentials that are carried more by reputation than not. 

I haven't come across anything in the literature that describes the difference between reputation systems and credential / badge systems, so thanks to Michael for pushing me to think about this more. Chime in if you have some thoughts about the differences, and what those differences might mean for badge system design. 

 

 

molneck

More on Reputations and Credentials

I can see that there are important literatures, familiar to those working on badges, with which I am unfamiliar, and with which I will have to engage. For example, I did not previously know there was a literature on “reputation systems.” So, please forgive if my questions and comments seem naive.
 
Sheryl notes that “there is an implied argument among advocates of reputation-based or peer learning that crowdsourcing credentials is better than the traditional system of credentialing.” I would like to know a bit more about what the criteria for that judgement are? Most of what I have read from those promoting badges, but not necessarily working on badges (e.g., Kevin Carey) is that badges will convey far more verifiable and “granular” information than that conveyed by traditional credentials like degrees. (See e.g., Carey’s “College Graduates Deserve Much More Than Transcripts.” Chronicle of Higher Education, April 29, 2013. Available at http://chronicle.com/article/College-Graduates-Deserve-Much/138861/?cid=... utm_medium=en.) Are there arguments beyond this, e.g., about the desirability of broadening the population whose judgements are weighed?
 
Like Sheryl, I believe there is more to credentials than their having “"reputations for validly indexing real qualities and capabilities.” But, as I read Sheryl’s Comment, she emphasizes credentials as deriving from authoritative institutions, and indexing accomplishments that are prescribed, and follow established pathways. This is certainly one feature of what are commonly referred to as “credentials.” But, I would emphasize the USE to which what we call credentials are put, and insist that absent that use, what we call credentials would not, in fact, be “credentials.” Consider a college degree. It becomes a “credential” ONLY when it is used as a medium of exchange with a third party, to acquire opportunities or rewards. A credential is always a “credential FOR.” If college degrees were not accepted or recognized as credentials, they would merely be degrees. (Think of medical degrees from other countries that are not credentials that warrant licensing in the U.S.)  I think Sheryl recognizes this, when she asks “In a badge system, what bumps it from reputation to credential? How much authentification is needed for that credential to hold up in, say, less receptive environments? Once you move outside your sphere of reputation, what kinds of tests does that credential have to pass in order for it to truly matter?”
 
It is because online reputational assessments are envisioned to be used by third parties to offer opportunities and rewards that I have difficulty perceiving an important distinction between them, and traditional credentials. Whether eBay users have combined to rate a seller highly, or someone has earned a degree, both are cases of someone’s bona fides being established for purposes of further exchanges with third parties.
 
Re: “In the traditional definition of a credential, it is awarded based on more than just judgment. A credential has been vetted by a so-called higher standard of de facto authority that defines some measure of competence or quality.” Two thoughts here. First, a “de facto” authority is only an authority because of societal recognition or acceptance (possibly codified in law) of the institution’s authority. Its authority does not derive from itself. (Think of the poor king in  Ionesco's “Exit the King,” whose kingdom receded until it was no more, and he died.) The “community” or “communities” in which an institution’s authority is recognized may be very broad, broader than the community or communities in which certain badges are recognized, but that authority may be contested, and erode, and the authority of something else, e.g., badges, may expand into communities beyond the original community in which reputation was established.
 
Second, aren’t reputational systems based on something more than “just” judgement? Isn’t the presumption that those offering judgements are competent to do so, and have reasons for their judgements? Indeed, are “judgements” ever “just”? Judgements, presumably, are not mere preferences. 
 
In this light, I am puzzled when Sheryl asks “Why is a  credential more effective at validating what a learner knows than pure reputation?”  Perhaps my problem is semantic. I am trying to figure out how “reputation” can be “pure,” i.e. reduced to something that is not also entailed in a credential. But, if “pure” reputation means not, or not yet, codified or condensed into a credential, then I would take “pure” to simply mean reputation among a particular circle or network. 
 
I think that Sheryl’s point about “power laws,” i.e. “And back to those power laws. How might web-scale peer-based or crowdsourced credentialing be influenced by power laws? If those credentials become meaningful, what rises to the top? What gets left behind? Or more importantly, who?,” is terribly important. I see a good deal of reference to the “democratizing” power of badges. Badges (and MOOCs) may well democratize learning. But, once badges become credentials, offered in markets (e.g., labor markets), I am not readily seeing how they will avoid the fate of the declining value of college degrees when access to higher education expands. 
 
Finally, Sheryl suggest that “Perhaps you could plot different badge systems on a spectrum of reputation-centric participation and see that it morphs into a credential when there are qualifying acts and prescribed pathways.” My understanding is that badges that are awarded in recognition of what people “know and can do” intrinsically include “qualifying acts” and “prescribed pathways,” albeit these can be diverse even as they lead to the same badge. Isn’t that part point in the argument for preferring badges over transcripts, i.e. they convey more information?
 
I hope my comments and questions help us move toward further clarification, and don’t just muddy the waters.
dthickey

On credential, reputation systems, and learning

there is a lot of great stuff here to sort through.  I really found Michael's example about foreign degrees not being valid credentials in a particular country.  If I am not mistaken this brings us back the the distinction between purposes and functions that I have been obsessing for a long time.  Is it ultimately about the function that a particular recognition serves?

I too am strugging to make sense of some of these issues because I am not grounded in the information systems literature that Sheryl is shoing to be quite relevant here.  I think that some effort and explicating the underlying dimensions of recognition systems more broadly might be helpful here.  What is the essential difference between eBay and StackOverflow?  Of course because I am concerned fundamentally about learning I look for those differences.  There is genuine disciplinary learning happening in Stackoverflow right?  People are learning quite a lot about a particular discipline (coding).  People are still learning in eBay (they are learning about peoples reputations and that is not disciplinary knowledge so it seems less relevant.

There is also the distinction that Allen Collins make (in the book with Rich Halverson) between just-in-case learning and just-in-time learning.  Most of the learning for which formal credentials are issued concerns the former; a great deal of the learning in reputational systems is the latter.  IMO this is a big differene between in the latter there is usually some obvious evidence of accomplishment (as in videogames).  In the former there usually needs to be some sort of an assessment practice that has to serve as a proxy for the accomplishment or learning.  

Finally, I think that Micheal's last point is the most important one in all of this.  I guess I should not be surprized but a year in people still don't realize that the important point about badges is that they are web-enabled and contain detailed evidence of leaning.  Just today I was talking to a colleague who had been talking with a Foundation program officer about an educational program that would include badges. The officer was rather suspect.  My advice was to consider saying instead "web-enabled certificant that contained detail evidence of learning"  Niether credentials nor reputation systems have offered this so perhaps we might not constrain our vision by trying to make it fit into either of those schemes. .  

 

molneck

Purposes versus Functions

I have been reading a bit in the sociology of organizations about the dynamics and patterns of the diffusion of innovations. A couple of points are relevant to Dan's distinction between purposes and functions. First, those who adopt innovative practices adapt them to their own purposes and needs. They do not merely faithfully "implement" the innovator's vision. The result is that "the" innovation varies in practice. The literature on curricular and pedagogical reform in classrooms is also relevant to Dan's distinction, and to the badge project more broadly. For example, implementation of constructivist math curricula is shaped by what teachers learn and understand about new standards and models, which is inevitably a partial version of what the curricula authors had in mind. Implementation is also grafted onto prior practices, and is shaped by previously existing organizational imperatives, which may be based on principles different from those presupposed by the new curriculum. Those committed to the use of badges as part of efforts to promote "deeper learning" should not be surprised if what teachers do with badges proves less complete than hoped for.
jimdiamond

badges will not democratize learning

Michael (and Sheryl and Dan),

thanks for your thought-provoking posts over the past few weeks. I'm going to run the risk of missing the forest for the trees in this response (or availing myself of an opportunity to grind an axe...), but I’d like to respond to your point that, "Badges (and MOOCs) may well democratize learning." I don’t read your statement as an endorsement of that aspiration, but it is in keeping with what I find to be a powerful—and often unexamined—motivation underlying some of the current badging initiative(s).

When I hear it mentioned with respect to innovations in educational settings, I generally take "democratization" to mean "more access and choice.” At least in my professional world of "educational technology," the implicit argument is that many learners do not have enough opportunities to access the "right" things (for example, the things that are associated with their personal passions) to facilitate better, more meaningful learning experiences. By and large, I agree with the spirit of the sentiment. But access and choice in and of themselves aren’t enough. If open badges are to become widely-accepted alternative (or additional) indicators of accomplishment, then badge producers and the various communities in which they exist will need to ensure that the credential represents experiences that have value both for the badge recipient and those who will look to the badge as an indicator of some present or future quality. In that case, access and choice are only the first step toward democratization.

One way of considering the quality of any given badge system—as well as its role in “democratization”—might be to use John Dewey's theory of educative experiences. For Dewey, “miseducative” experiences were those that prevented the possibility of richer experiences in the future, while educative experiences always led in the direction of “growth” (the “continuity principle,” connecting present to future experiences). A second principle relates to “interaction” between a learner’s “internal conditions” and the “objective conditions” created by the educator and the learning environment. The role of the educator was to ensure that the objective conditions, the environment and tools with which a learner interacts, facilitated a learner’s ability to grow. To connect this back to badges: “educative badges” are those that promote growth, while “miseducative badges” are those that stunt it. The goal for the design of any given badge system ought to be promote continuous growth in directions the learner chooses. Further, (for Dewey…) a badge system that contributes to “democratization” would facilitate an individual’s ability to contribute to democracy as a “mode…of conjoint communicated experience.” For Dewey, the goal of education was to lead to growth, and growth could only be obtained when discourse was mutual and based on equitable interests. And the way to facilitate that growth was to develop learning experiences that encouraged forward movement.

All of this is perhaps to say the obvious: on their own, badges cannot make for more democratic learning experiences, at least if they are not evaluated by how well they meet an individual learner's needs, as well as the needs of the broader community in which the learner is situated. It goes without saying that the same critique is true of many of the credentialing systems we have now. For badges truly to contribute to democratic learning experiences, we will need to develop explicit criteria by which to judge how effectively they contribute to growth. Further, we need to find ways to "translate" those criteria across domains such that they really can contribute to the "conjoing communicated experience."