Blog Post

Intended Purposes Versus Actual Function of Digital Badges

I posted a longer blog about this topic on Remediating Assessment .  Here is a shorter summary that is also useful for folks who are interested in or are using digital badges.  One question about badges seems like a crucial issue as we grapple with different ways of characterizing and describing badges.  This post aims to add the category of badge functions to other badge taxonomies like the one by Carla Casilli.

Purposes Versus Functions in Assessment

I got interested in this distinction several years ago when thinking about assessment and transfer.  Many leading assessment scholars have followed Paul Black and Dylan Wilam’s lead in categorizing assessments in terms of assessment purposes.  In their groundbreaking work on formative assessment, they argued that assessment should fundamentally be put to formative purposes, which they defined as assessment for learning.  This was an important distinction and helped distinguish formative assessment from summative assessment, which they defined as assessment of learning.  This distinction is now widely embraced, along with the further distinction between evaluative purposes, which refers to assessments used primarily for evaluating curricula or programs.  

Over the years, I have come to the conclusion that categorizing assessments according to their intended purposes is also problematic.   Focusing on purposes overlooked the fact that the same assessment could be summative for some kinds of learning and formative for other kinds of learning.   For example, a formal exam that serves as a summative assessment of student learning can also be a formative assessment for the teacher to learn how effective the lesson is and how it might be revised.  Likewise, achievement tests are summative for teachers but formative for policy makers to learn whether policies have helped or hurt achievement.  For complicated reasons, this focus on assessment functions corresponds with the broader view of learning that is embraced by the DML initiative and that inspired the Badges for Lifelong Learning initiative. A nice articulation was here in John Seely Brown’s DML 2012 Keynote.

The Varied Functions of Badges

My interest in the functions of badges was spurred along when the MacArthur Foundation asked for help documenting the design principles for using digital badges that emerge across the 30 projects underway by the awardees in their Badges for Lifelong Learning project.  We needed to come up with a manageable number of categories.  Here is what we came up with:

Recognizing Learning.  This is the most obvious and arguably the primary function of badges.   David Wiley has argued cogently that this should be the primary purpose of badges.  If we focus only on purposes, then he may well be right.  His point is that badges are credentials and not assessments.  This is also consistent with the terrifically concise definition in Seven Things You Should Know About Badgesby Erin Knight and Carla Casilli.

Assessing Learning.  Nearly every application of digital badges includes some form of assessment.  These assessments have either formative or summative functions and likely have both.  In some cases, these are simply an assessment of whether somebody clicked on a few things or made a few comments.  In other cases, there might be a project or essay that was reviewed and scored, or a test that was graded.  In still other cases, peers might assess an individual, group, or project as badgeworthy. 

Motivating Learning.  This is where the controversy comes in.  Much of the debate over badges concerns the well-documented negative consequences of extrinsic incentive on intrinsic motivation and free choice engagement.   This is why some argue that we should not use badges to motivate learning. However, if we use badges to recognize and assess learning, they are likely to impact motivation.  So, we might as well harness this crucial function of badges and study these functions carefully while searching for both their positive and negative consequences for motivation. 

Evaluating Learning.  The final category of badge functions we are working with concerns evaluating learning. Digital badges have tremendous potential for helping teachers, schools, and programs evaluate and study learning.  At the minimum, just having a system for tracking all of the information included in all of the badges that a program awards might be very valuable.  Each badge has eight bits of information (“metadata”) which I presume will be recorded and easily accessible as a database.  And much of that information will be hyperlinked to even more information that will be accessible with just a little more effort.  Whether or not programs elect to use this feature will likely be up to them.  Additionally, some of the DML awardees are realizing that they can offer badges to learners for participating in their program evaluation plans as well. Awardees like Jim Diamond at the Education Development Center and Stacy Kruse at Pragmatic Solutions are just brimming with great new ideas of ways to use digital badges and all the information they offer to evaluate programs and study learning. 

Summary of Badge Functions

The following table summarizes the basic point of this post.  While the categories of badging functions are likely to evolve, I think the basic point will hold that there are a definable set of ways to use badges to support learning, and that functions are a good way to sort these things out and their interactions.  And while some of the functions, like recognition, appear essential to digital badges, other functions, like evaluating and researching learning, are better understood as potential functions.

 

 

114

26 comments

 

Here is the table.  We had some trouble posting it to the blog as a table

Recognizing/Credentialing Learning IS AN essental function

Assessing Learning IS A Probably Function

Motivating Learning IS A likely function

Evaluating/Researching Learning IS A potential function

99

 

 I would love to get input from folks as to whether these categories make sense and whether we have left anything out.  In particular I am wondering how the goal of defining design principles is served having a limited set of categories.  Alternatively we might consider the using more expansive and overlapping dimensions like the ones that Carla Casilli defined in a recent post about the open badge ecosystem.  My concern is that it might be unwieldy to derive and organize design principles without categories.  What do others think?

101

 

One of the reasons these categories appeal to us because they make apparent the interaction among the practices. Everybody will be using badges to recognize learning, so this will impact potential functions studying learning. For example if you don’t choose to recognize some type of learning you won’t be able to use the date from those badges to study that learning. But the relationship does not work the same way in the other direction.

103

I think Lora raises an important issue regarding our position in trying to frame the WHAT of badges. I can imagine that for an academic audience interested in assessment, those four might be great. But for informal educators, we might have very different NEEDS we are concerned with that drive what role(s) badges might play.

So while I have a hard time seeing how 3 of the six Frames from Six Ways to Look at Badging Systems Designed for Learning, I am not yet convinced I should expect to. One should your professional perspective on badges be the same as mine?

Frame 3: Badges as Learning Scaffolding

Badges, as a form of scaffolded learning, reveal multiple pathways that youth may follow and make visible the paths youth eventually take. Scaffolding in a learning environment refers to providing guidance for youth to encounter learning opportunities that engage them at their level of ability before taking them to the next. In other words, badges not to assess or motivate, but to guide. Within an organization, that might mean using badges to provide youth a way to understand the options offered within a particular organization, customized to their current interests and abilities; e.g. introductory level badges might be pursued to unlock another set of badges. Across organizations, it might mean using badges to provide youth access to similar opportunities within their community.

Frame 4: Badges to Develop Lifelong Learning Skills

Badges are viewed as a tool for developing the metacognitive skills required by today’s youth to succeed in the classrooms, workforce, and civic spaces of the 21st Century. Badges support learners to give language to and value what they are learning, by offering names for their new competencies and providing a venue that recognizes their importance. As badging systems can aggregate the various formal and informal places youth learn, badges support learners to make connections amongst these places and develop strategies for negotiating and eventually shaping their emerging learning ecology. Badging systems also offer new communities within which participants may learn to negotiate their own learning identity, based on the theory that how well people learn and their motivations for learning are based on their identity as a learner. The 2012 HASTAC competition employed this frame in its title.

Frame 5: Badges as DML Driver

Badges are viewed as a praxis to undermine the deficiencies within current learning environments and spread Digital Media & Learning practices. Badging systems require participatory learning environments, offering peer-based learning communities in which youth don’t just receive badges but comment on them, share evidence around them, and more. Badging systems can reach youth throughout the Hanging Out, Messing Around, and Geeking Out framework (HO-MA-GO). Badges should disrupt existing practices. Badges should lead to a better use of digital media for learning.

 

 

99

 

Another question this post raises is whether it makes sense to characterize badge functions in terms of learning.  Others make a distinction between supporting learning with badges and other functions like socializing or making friends. My theoretical orientation leaves me with a very broad view of learning, so that knowing what your friends like or have accomplished is just a more specific kind of learning. I like the broader view because it provides coherence.  In particular I worry that setting learning off in a separate category from other social activity conflates it with teaching

95

. . . for now.

If badges are really going to become useful for learners, they have to be respected by the audience that the learner wishes to accept the badge (educational institutions, employers, certifying agencies, etc.).  For this to happen, badges should be used strictly to indicate mastery of some subject matter.  It is the equation of the badge with concrete mastery that will give badges currency in the world outside Mozzilla-land and academia.  Once that currency has been established, then (and only then) can the broader view be applied.  (IMHO.)

95

I think the idea of categories is useful; perhaps a taxonomy or hierarchy of desciptors would be helpful?  For example, while I agree with the importance of maintaining learning and teaching functions separately, the way I am thinking (now :) ) is that a robust learning trajectory with a recognition and assessment can not only be motivating, but seems to be a construct inextricably linked with a badge purpose "for" learning......moving the learner towards understanding versus measuring the outcome "of" the learning process.

Furthermore, some badges, even with an expansive view of learning, still seem to be aligned with the old concept of "seat time" ("screen time") or social capital versus a learning function (the equivalent, perhaps, of digital LOTS (lower-order-thinking-skills).

If the functions of some badges are so intertwined, as they are in some cases, that a "primary" function is not immediately apparent, the perhaps the construct confers a kind of "uniqueness" to a badge purpose (learning trajectory + social discourse for learning + recognition + motivation + formative feedback + summative assessment).... so other categories, which are construct-based, could be useful for identification.  (Or, for example, recognition + motivation + social capitalizer :) ). Perhaps part of the reason we are all struggling with these classifications is because the differences are nuanced and sometimes badges have aggregate functions. Definitely MHO :)

76

I appreciate the work you are doing here to define functions of badges Daniel.  The taxonomy of potential purposes and functions begins to define what is intended by designers of badges.  As we know, what is intended is often not what actually results so your broader focus to include practical usefulness is wise at this early developmental point.  Since my careers have all been in practical applications, I find myself leaning towards points Peter is making and the ultimate value of the technology.  There are values associated with pedagogy and learning...but actual gained skills and abilities in practice that are represented by badges are likely to be critical in gaining broad acceptance and use of badges.  

80

Hi Dan,

I think that the four categories you have identified are a comprehensive place to start. Here are, briefly, my thoughts on each:

Recognized Learning: The strength of a badge's reputation and trustworthiness as an accrediting device seems to come directly from strength of the reputation of the issuer. This means it's essentially a branding issue.

Assessing Learning: This has to do primarily with the rigorousness of the curriculum that must be completed in order to get a badge, and how thoroughly the learner is tested for knowledge retention.

Motivating Learning: How do we appeal to learners who want to compete, explore, achieve, and socialize while learning? How can badges tap into the learner's desire for accomplishment that traditional classroom instruction often leaves out?

Evaluating Learning: This is the big one, and the one that is certainly only possible with the launch of the DML Design Principles Documentation Project--tracking how we know what we know.

The only other big issue I can think of that isn't covered is this: the September workshop reminded me how incredibly different from each other some of the badge projects are--should be be keeping track of and evaluating in some way the huge variety of learning experiences that can be augmented with badges?

59

Hi Dan and all,

These categories map nicely onto the design of digital badges for the HYPE youth media program.  My theoretical orientation, from within the field and scholarship of youth media, leads me to ask (in the context of HYPE):  how do the youth participating in HYPE describe the function and purpose of badges, in their own words?  Discussion about functions and purposes seems to me to be very much contingent on where we are locating ourselves in the conversation.  I appreciate that these are urgent questions to be asking from the perspectives of developers, administrators, designers, implementers, educators and evaluators.  And the categorization offered here helps me  a lot in thinking along these lines.  But I am also aware that the learners engaged in HYPE's badge system have their own notions about how badges function for them, and their own perspective on the purposes badges serve for them.  So we're trying to approach evaluating our badge system with the same methodology of "collegial pedagogy" that informs how we organize the spaces, practices and relationships of youth media making at HYPE.  The term of course is drawn from Lissa Soep and Vivian Chavez, and captures our intentionality about shifting power relations between yougn people and adults.  In this context, for us, one of the greatest promise of badges is to what extent and how they can help create and sustain that shift over time.  How might these categories (of design and analysis) shift if we take up the locations of learners and badge earners?  It may mean a greater multiplicity of categories and an awareness of the places where youth and adult perspectives meet and where they (productively) diverge.

93

I have been trying to get to a longer post to respond to this for a while.  But alas.... What I wanted to say was how lovely it was to NOT know about Soep and Chavez.  I asked a colleague who studies culturally relevant pedagogy and he was not familar with them either.  But we dug a bit deeper and began seeing a lot of commonalities and cross references with the scholars who study comparable issues in context closer to home.

My point is that another part of our project is going to be assembling the relevant research literature and attaching it to the specifically relevant principles.  Case in point, in the comment that Barry made below he is highlighting the fact that we are certain to end up with a design principle about "democratizing learning".  We need a system whereby others like yourself can help us add references and help other see their relevance.

Much of the most relevant literature will come from diverse areas, so it really will be a communal effort if it is going to work.  More later but I just wanted to respond to tell you how excited I was to get your comment and that I plan to use it to frame the next post about our project.

70

Dan,

What a great idea.  You've stirred the makings of a system with this ongoing and robust discussion!  You're right about the array of fields, disciplines, ways of knowing that will combine to create something like a literature review  on badges and democratizing learning.    Reading the replies here is helpful in deepenig my awarenes of my own discipline-specific location in relation to the badge project.  

Shaping my current research on badges in the local context of HYPE, I'm drawing heavily from scholars in and aroudn the field of youth media.  Voice, agency, and collegial pedagogy help organize this analysis.  The project I'm developing is interested in the potential for digital badges as mediating artifacts or tools that help young people develop awareness of their agency as learners.  Glynda Hull writes usefully of the crafting of "agentive selves," through multimodal storytelling.  At HYPE, we're thinking about badges as digital tools that intersect both with the formal documentary storytelling and with the authoring of agentive selves.  

 

It's very promising to read here about democratizing learning as a design principle.   I understand this purpose to be one of the drivers of the DML / Mozilla open badges effort--and thinking together about how that takes shape as a design principle is an exciting invitation.  Badges hold further promise, it seems to me, as tools for valuing the processes of democratizing learning.  This is intrinsically connected to "voice" --  can digital badges, in effect, give voice to a learner's story about what they know and how they know it?  Can learners leverage badges to give voice to learning that otherwise might remain unseen/untold? To have a voice in what counts as learning and representations of learning?

 

This is, of course, my representaton of what I hear the teens at HYPE asking from badges, and wanting from digital media more broadly.  Expanded opportunities, tools and spaces for makign their learning visible and connecting it to other contexts and communities. Badges offer them a cool way to catalyze this kind of connection and recognition. 

103

Our project's task is to simply document the design principles that emerge from the various DML projects (more at http://hastac.org/node/104928).  Personally I have a bunch of opinons about these things but our job is to document what happens across the awardees.  As Carla Casilli explained so nicely in one her posts, there is going to be a bit of chaos when designing systems.  I suspect that the lovely chaos needed to get badges dialed in will be much more challenging in an accredited academic unit. My  university appears poised to pilot them in a non academic unit (OpenFlow networking workshops) before allowing them in an academic unit.

Look for example at the practices that are emerging in the work in informal contexts by global kids.  The report of their badge pilot this summer was full of great ideas and refinements: http://www.olpglobalkids.org/content/global-kids-summer-badge-beta-report

But in the end we just need to find the places where they are apprpriate and then identify the design principles that need to be further refined in that context

95

Angela, that was very intriguing.  One of the things that Andi Rehack has been working with on our project is the process of categorizing the DML project.  This is going to be cruical down the road for helping people find design principles and examples that are most directly useful for issuing badges in their particular context.  We are working with some of the categories that Carla Casilli defined here http://carlacasilli.wordpress.com/2012/08/07/mozilla-open-badges-another-take-on-the-shape-of-the-ecosystem/  So the notion of "construct-based" badges is an interesting way to consider categorizing them.  We will consider it as we slog through our continual process of refining categories.

95

Angela raised a really interesting theoretical point. Yes badges are certain to have mulitple functions.  But my situative theoretical orientation makes it almost impossible to envison the functions in an aggregative sense, because of the way the functions interact with each other. 

For example, summative assessment functions often undermine formative goals:  is this someting like Formative feedback - summative goals?

But rigorous summative assessment is likely to increase the value of badges as recognition of accomplishment which in turn increases their motivational function.  I guess this is something like Summative Assessment x Recognition x  motivation.

I actually have no idea how to even characterize these relationships verbally much less mathematically.  But to your point about nuances.... they are all nuances at this stage, but once we dive in and start issuing badges those nuances become very salient aspects of badgeing practices, and wicked problems are likely to emerge very quickly.

thanks!

95

== drawing upon a couple of decades of cataloging and classifying "things" :)  --  I've found that sometimes a new "class" is needed when it's difficult to place things in one category (or lots of descriptors....). Perhaps there are qualities of badges which are not exactly an aggregation but unique/intrinsic attributes to the digital badge concept -- 

Best!

107

Great point Angela.  We have all been pondering what to call things and it is both fun and aggravating, but it certainly is a sign that you are incharted territory.  

When you say cataloguing and classifying things, do you mean like the work of Bowker and Starr?  Their notion of "boundary objects" seems pretty relevant here.  Pam Moss (Moss Girard and Haniford) helped me understand how it all relates to assessment, becuase assessments take on meaning in their particular educational systems.

Because of (a)  the amount of information they contain, (b) the amount negotiation required to define that information, and (c) the ease with which that information can be shared, badges can take on a lot more meaning than traditional assessments or tests.  This lets badges carry a lot more meaning across a lot more boundaries.  And while we are getting all anthropological, we can shudder to imagine the potential prolepsis that badges might afford.  I still can't wrap my head fully around prolepsis and we still don't know enough about badges to speculate about a fraction of it.  Maybe you can.  I have never studied anthropology.

100

:)  The Bowker and Starr work looks very interesting, but my background is in cataloging and classification of knowledge objects and describing them (like metadata).... and thinking deeply about how others may perceive or receive, or retrieve that info. Taxonomies, controlled vocabulary, descriptors, meta data, key terms -- through some of my work as an information specialist (Masters in Info & Library Science... a couple of decades ago). In the most superficial kind of this work, argulably, people think of attributing "subject headings," but the long view incorporates the users' perceiptions, needs and mental models -- knowledge seeking  and acquisition habits as well as understandings.  

Once, long ago, I was charged with creating such a system for a slide library; classifying the images involved an understanding, not only of the work of art itself, but also its attributes:

  • construction: media or materials;
  • context: school of..time period..."movement" etc
  • purpose: and/or meaning
  • iconography: or symbolic meaning(s)
  • cultural history: place, time, "actors"
  • subject matter: thematic elements
  • significance
  • artist or architect, photographer etc
  • "type" of art {painting, sculpture, textile, etc)

then, having considered those points (for a 2  x 3 slide, as they were then :) ), thinking of how others may wish to retrieve or use the image..... and assigning descriptors or other intellectual access points for those. The "others" looking for the images were local audiences knowledgeable of art history, the "whole world," through a database, and others who were just looking for an example of x, y or z.

It was a huge task to organize thousands of images with very diverse properties :) Contributing to the difficulty was the fact that the system I was instructed to use had serious limitations for both storing and retrieving this kind of data. However, there are similariities with a "badge" project..... with similar constraints. A new, large system, which is still evolving... with some who would care to consider the attributes, as well as a seeming majority of folks who care about the end products because it has value to them and the potential badge owners or audiences.

My point is that by carefully considering the various attributes and purposes, intended or not, I suggest this will  eventually add depth to the badge concept because of this thinking and the application of theoretical elementss (a framework). A concern is the development of a classification scheme which does not plan for the expansion of categories right from the start...and here I'm talking about classification of badges, I think, versus a project to find a commonality of design elements :)  When I worked on the image classifying project, it was necessary to create "new" descriptors which were in a sense, a gesalt. of other attributes.... but it took a while to consider what these new categories should be.

Prolepsis.... very interesting :) although I confess I don't really know what it means (yet :) ).

90

I just realized that Alex Halvais did a nice bit on badges boundary objects on a great post over here: http://alex.halavais.net/badges-the-skeptical-evangelist

103

As we elaborated here http://hastac.org/node/104928 our project is actually not an evaluation of the initiative or the projects.  In my opinion that would be premature, because the goal of proving that badges "work" would interfere with the goal of envisioning and refining the design principles needed to use badges appropriate in particular learning contexts.  Our job instead is tracking the emergence of research and evaluation practices within each of the projects, as they take advantage of the opportunities that emerge in their context.

As a researcher and evaluator, I am insanely excited to watch people discover new opportunities for evaluating their own projects by using the evidence that badges will generate.  Take a look at the report from the pilot that global kids did with BadgeStack this summer: http://www.olpglobalkids.org/content/global-kids-summer-badge-beta-report  as they point out they never used to gather detailed data about what their programs accomplished.  The data in the badges was very useful for both finding out what they did AND communicating that to others.  I have dabbled in program evaluation for years and I know how difficult it is for many informal organization to gather this kinds of data and then present it.  It is like magic IMO.  Likewise look at how easily global kids was able to actually study the patterns of activity in their projects.  

It is premature to report on what we are finding in our interviews with projects, but there is a lot more where this stuff comes from. It is going to be amazing.  If we can find a way to make sure to link the principles to people and projects who take the risks and refine the practices from where the principles emerge, we should be able to create some sort of self-sustaning social network.  I have no idea how we are going to do that but it sure looks to be fun to figure it out.  Great to have folks like you excited about it.

87

that was stunning. Yours was one of the best response I have seen to the concern that that the badges that many people are envisioning will be controlling and support only very mechanistic forms of learning.  Global kids involved their youth very directly in the design of their badges even before they were digital several years ago.  One of the DML projects organized a very sophisticated meeting with targeted youth to define both what forms of learning that would be recognized with badges and how that learning would be assessed.  They even brought in professional designers to work with the youth to help them design professional looking badges that embodied their values.  From my perspective this sure seems consistent with collegial pedagogy.  That brings me to another point worthy of another comment about tracking the relevant research but I am going to save it for another blog post.

 

And, let me say to all of you, thank you for your comments and sorry for the lenghty delay in resonding.  I turned off the notifications at a point when I meant to turn it to a digest.  You may have surmized that I am not a "digital native".  Thanks for your patience!  

73

Anonymous (not verified)

Dan, I think Mark Prensky would call you a "digital immigrant." :)

91

Doh! 

I just realized that I could have put each of these replies as actual replies to the comments.  I as such a dope.  Or maybe a dohpe?

76

I'd agree here, and add that I think it is not just in the badge/curriculum development process but the long term relationship with youth's learning that is addressed here, regarding the nature of power between youth and adults.

To quote from the "frame" documennt from Global Kids:

Frame 6: Badges to Democratize Learning

Some badge systems are designed to democratize the learning process, to change who does the assessment and what affect the learners have over their learning environment. Learners can shape the content of their badging system and perhaps even the structure itself. They might propose or add new badges, or the missions required to earn them. Learners can participate in the assessment process, ranging from recommending peers for accreditation to earning the right to be the accreditor.

94

Barry--

that is a great point.  This gets at a point that Carla Casilli has raised and hope that she will weigh in on here.  Carla indicated that sometimes people will assign social purposes to badges.  It is like your suggestions of democratic purposes.  

One one hand we might just allow the purposes to keep expanding the purposes.  But that seems like it gets unmanagable.  I find group of 3-5 to be about all organizations can manage.

Right now my project is addressing this from a theoretical perspective.  For me, these are purpose are aspects of the formative and transformative assessment functions.  This is because I think that changing "who does the assessment and what affect the learners have over their learning environment."  is itself a form of learning.  It is the social learning that Seely Brown and Mimi Ito talk about.  Likewise I think the purposes that Carla talked about are also learning (discovering what your friends "like" is learning right?).  My sociocultural/situative perspective argues that all forms of social change are learning, and that "academic" learning is just a special case of that learning.  I think this distinction is hugely important.  By not treating the forms of change that you and carla are talking about as learning, we open ourselves up to a narrow approach that elevates academic learning and downplays other forms of learning that are ultimatly more consequential

there is a short post about transformaitve functions here: http://hastac.org/blogs/dthickey/2012/03/18/some-things-about-assessment-badge-developers-might-find-helpful.  A longer one here: http://remediatingassessment.blogspot.com/2012/03/some-things-about-assessment-that-badge.html  If anybody is interested I was asked to prepare a longer version of this for an assessment jounral and I would love to hav somebody read it and give (acknowledged) comments before I submit it.

106

A bear and a bird meet in the woods, the bear on the ground and the bird in the tree. Posted on the trunk of that tree is a sign, one word above the other, reading "leadership, through badges, can be learned." The bear says, "Ah-ha! Badges are about learning. it is the word closest to me." The bird say, "Silly bear. Badges are about leadership. That is the word closest to me." At that point a squirrel pops out her head from the tree and says, to no one in particular, "you are both right. It is all a matter of perspective, determining what you background and foreground. No perspective is absolute but all live in an ecology of meaning." The bear and the bird both look at the squirrel and ask, "Who are you taking to?"

93