5 Buckets for Badge System Design: "You Are Here"

Badge systems, like other sociotechnical systems, are ways of building order in our world – but for now, at the early stages of design and development, we’re in an innovation free-for-fall, which can be both exciting and daunting to newcomers. To build out the badge ecosystem, it’s important that we create a full library of toolkits, templates, and other signposts to guide people. 

I’m interested in minimizing the daunting part of badge system design. We need to take what early innovators are learning about the design process, and make that knowledge useful to others designing badge systems. What does Intel’s badge system have in common with Girl Scouts? How does the badge system at UC-Davis compare to Nature Badges at the Smithsonian? If we can figure this out, then we can find some solid footing or starting points for the design process. We can create a way for organizations to find their "You Are Here" pin on the map. Knowing where an organization is in relation to others will help make sense of the badge design process, and it will also light up opportunities and possibilities.

I spent a few weeks with piles of post-it notes and the 30 Project Q&As that our Badges for Lifelong Learning grantees shared about early stages badge system development. I started sorting bunches of phrases into bigger bunches, looking for categories that made good buckets. (Thanks to everyone in Duke’s Phd Lab for letting me take over their wonderful space for a short period.) Once I had a sense of how the lessons looked in those buckets, and I saw which projects were connected to which advice, I started to see some really clear badge system classes. I’m calling them classes because I have an information science background and that word choice makes sense to me, but they’re basically just buckets or categories.

I went back and forth about whether to define the badge system classes in more detail, and opted to make them as succinct as possible. Partly because I’m really interested in hearing your feedback and want to see how the classes resonate with short descriptions. And partly because I want to avoid any hint that one class is better than another. To my thinking, some organizations may also be between classes. They may start off as one class, and see how their decision points change if they shift to a different class -- by developing new learning content or opting to rehaul their current technology platform, for example.

Here are the original 4 badge classes:

  • New build. The badge system, learning content, and technological platforms are designed simultaneously.
  • Integrated build. The badge system and learning content are co-created and integrated into a pre-existing technological platform.
  • Layered build. The badge system is layered on top of pre-existing learning content and pre-existing technological platform.
  • Responsive build. The badge system responds to pre-existing learning content, and the technological platform does not yet exist, is optional, or is distributed.

I’ve been talking to others in the badge community about these classes, including Lucas Blair of Little Bird Games (check out his awesome Badge-Based Curriculum Design Process graphic) who not only gave excellent overall feedback, he suggested a 5th class:

  • Badge-first build. The badges are designed first and the learning content and technological platform are designed around the badges.

Lucas rightly pointed out that many learning organizations are currently drawn to this 5th design approach. As soon as he pointed it out, I now see examples everywhere of this kind of badge system. I plan to dig into the conversation Lucas and I had about these classes and how they might evolve over time, but for now, if you’re interested in badges, badge system design, the badge ecosystem, or all of the above, please feel free to weigh in.  What’s missing from these classes? Are there other classes we should include? How might these classes be used? 

I’ve also begun linking the badge system classes to different examples in the Badges for Lifelong Learning projects, but before I make those links public, I need to revisit some of my notes to see if they do indeed match the classes I’ve sorted them into. If your organization is among one of the Badges for Lifelong Learning projects, do you see a build that reflects your design approach?

Eventually, I can see how badge system classes might act as an organizing principles for the work Dan Hickey and his team are doing with the Design Principles Documentation research, and perhaps it will also help with the way we think about toolkits for badge system design. More to come on these thoughts -- this is a first post of many as I gather more feedback from the community.


 

Angela Elkordy

Badge Design and a view of the K12 space

Sheryl, I think you bring up a great point that we are at the nexus of "innovative and promise" as well as "trial and error...(then "discard or regroup" according to level of traction) a system  in digital badge design.  This will probably continue as we work through the purposes (intended or not) or digital badges in our various communities of practice. I would like to eventually see a more robust digital badging platform available in the K12 (more specifically, the secondary level) which permits alignment with more than one set of standards as well as flexibility to describe the attainment of cognitive tools or ways of thinking. For example, in my own badge design, I'm seeing the limits of aligning to Blooms, and also, limitations on how to articulate the culmulative knowledge acquired through a learning trajectory of several badges? (I've designed a 5 level digital badge learning trajectory: InfoMaker, which is aligned with several standard sets (which naturally don't encompass all the knowldge or skill sets  -- intentionally :) ). Through the badge series, kids (grades 6-10) learn "design thinking" as well as the Next Gen SS Practice of "Obtaining, Evaluatind and Communicating" information. A system built around badges may help with this -- but special builds are tend to be proprietary -- which impedes access for us to move forward with these issues as a community.  Interesting observations, thanks for sharing!

slgrant

Interesting what you say

Interesting what you say about the limitations of aligning to Blooms. Can you say more about that? It's particularly interesting to me given the original intent of the taxonomy, which was to improve communication across educators and designers. Sometimes, the frameworks become so easy to use that people forget what they really mean, if that makes sense. That's one of the reasons we're being careful to stay away from concepts like "best practices." 

jimdiamond

Badge classes

Sheryl,

GREAT post and AMAZING work! The pragmatic developer in me appreciates that there might be a "If you are here with respect to badges, do this" decision tree at some point, while the researcher in me is very interested in knowing whether there are qualitative differences related to learning or implementation outcomes based on which bucket you fall in. For example, the fact that, "many learning organizations are currently drawn to this 5th design approach" makes me nervous (perhaps wrongly), as it reminds me of any number of projects I've seen where the development of the tool preceded the clear articulation of the learning goal/need, which typically leads to a mismatch between the doing and the learning. But then, I can also imagine badge frameworks (such as the OBI) that are generic and meta enough to accommodate an incredibly broad range of objectives, especially as the badges themselves aren't the tools that facilitate activity, but rather markers of the activity.

Regardless, this would be an interesting framework for some evaluator to pick up at some point. Thanks a lot for sharing it, Sheryl.

slgrant

You picked up on the exact

You picked up on the exact same question I have about the 5th bucket -- although I have some additional questions about sustainability, and perhaps credibility as well. I'm thinking of Carla Casilli's post about validity, in which she looks at credibility, reliability, and validity. Badges are going to be under much more scrutiny than other credentials, but we also don't want to make badges like other credentials in order for them to be credible. It's a tension that will probably continue for a while, at least until we start to see how design approaches affect some of these larger claims. 

dthickey

The perils of the fifth bucket

Sheryl--

Awesome post and great work.  This is a great example of how systematic study across multiple design efforts can generate generally useful knowledge.  I started a reply and it got too long so I put it up at as new post at Remediating Assessment.  The upshot of the post is the perils of starting with badges.  We are doing a workshop next week to share out our findings from the design principles project and realized that you categories are the clear starting point for working with people who want to implement badges.  We worked on a slide to represent your categories and this is what we came up with:

BUILDING BADGES: Technology + Content + Badges

  • NEW: Technology, content, and badges all designed simultaneously
  • INTEGRATED: Start with technology
  • LAYERED: Start with technology and content
  • RESPONSIVE: Start with content
  • BADGE-FIRST: Start with badges

Your post helped us reframe the big point we were working with for the workshop.  Now it seems like it should have been the title of the event:

It's not (just) the badges.  It's the ecosystem!

So the question for all of us who want to help people build badge systems is how do we help people who start with the badges reframe their efforts to consider how the introduction of badges will allow them to re-envision both content and technology.  I think there are some nice examples in Nichole Pinkard's characterization of layering (to use your category) of badges into their existing Digital Youth Network content and platform helped them transform the platform and especially the content.  It is a nice contrast to some of the projects that seem to have focused so much attention on the badges they never manage to work out the much larger issues with content and technology.

Your post is really timely for us and will help shape the final round of interviews on our project.  Thanks!

slgrant

Perils of starting with badges

To say that badges-first is perilous means that something is important. What is the something that's important? Is it ease of design? A credible or rigorous system? Narrower choices deeper into the process? Better learning outcomes? It's hard to stay value neutral about these design approach buckets, but if we are going to say one is better than another, then I think it's important to say why. In general, though, I  think we'll see more badges-first builds now while things are so new, and then we'll start to see more optimal builds emerge as the norm. (Another shout out to Lucas for having that conversation with me a few weeks ago.) 

I've received great feedback from the Badges for Lifelong Learning Competition grantees about which bucket they see their projects belong to, and I'll write a follow-up post with some of their comments later. What I'm particularly interested in is how much drift between buckets occurred. That tells us just as much if not more than knowing which bucket people identify with. 

Glad you're able to use this with your work, Dan! 

jimdiamond

This probably isn't the right metaphor, but

"starting with badges" reminds me a bit of "teaching to the test." (As I say, I'm not sure the metaphor captures it.) My sense of unease probably relates to the "credible or rigorous system" something that's important. Let me see if I can type this out loud—

I find that I'm often trapped in "educational game design think," where I'm interested in the alignment between game elements (typically mechanics) and learning objectives such that I feel like the doing in the game is giving learners opportunities to practice certain skills or wrestle with a concept, either in the game directly or with a teacher using the game play experiences in the classroom.

But badges are different—they're not about the doing, so much, as about the done. And that's typically what worries about me most about them: what's to prevent badges from becoming the same old degrees and tired credentials that many of us here seem to be unhappy about? Well, it's most likely about the process that leads to the achievement of the badge. I'm temporarily setting aside the valid assertion that badges are motivating in and of themselves, though it can run afoul of concerns about extrinsic motivators). The process is the really important piece. But if we start with the achievement—that is, if we start with the badge and then build backwards—we should be incredibly clear about the process that the badge is supposed to represent, rather than the outcome. In other words, define the badge in terms of what the earner can DO and then articulate all the supports that will help the earner get there...

Does this make sense? Maybe not...But that's what comes up for me when you ask about the "something that's important."

dthickey

Reciprocity

Jim--

The thing  I like about the first bucket is that it requires innovators to see badges as one part of an ecosystem.  The relationship between the badges, content, and tech is reciprocol (regardless of whether the reciprocal influences are is recognized or the extend to which they allowed to influence the design of each element).  The reason most of us react so strongly to "teaching to the test" is because that usually implies teaching to distal standards-oriented achievement tests that were never meant to be taught to.  I know it makes me a minority in the assessment community but I don't think we should be teaching to classroom performance assessment either (whether it be following Wiggins and McTigue or whomever).  One of the reasons that I think badges will be so transformative is because it will help many differnet stakeholders realize that it is all about the ecosystem.

Put differently, I think that starting with the badges presents the same challenges as starting with the test (or the performance assessment).  I am not a big fan of "backward design" because it organizes learning ecosystems around the narrow and individualized ways knowing can be represented on a formal assessment that can be efficiently and objectively scored.  Sheryl's framework helps us appreciate that the connection between the badge and content is most likely via an assessment.  Backward design from an assessment means that the function of the bades is constrained by specific (and likely narrow) representation of the knowledge on the assessment.

The magic of badges IMO comes from the fact that the badges contain actual claims and specific detailed evidence (and links to evidence) about those claims.  This means that someone who is considering the crediblity of that claim can look at the evidence and see for themselves.  Without badges, our assessment practices must adhere to traditional assessment validity practices.  This means that the credibility of traditional scores is premised on the assumption that institution assured the assessment or the test was reliabile and the claims were valid.  This rules out many of the kinds of innovative assessment practices such as peer assessment that have the most potential for transforming ecosystems.  

This is why Carla Casilli's insight about the way credibility is judged in social networks is so important to us. Because badges contain claims and evidence, their credibility is more contextual & situational and less formal & institutional.  The first bucket ensures that this unique affordance of digital badges is allowed to influence and be influenced by the content and the technology.

 

 

slgrant

Yes -- this makes perfect

Yes -- this makes perfect sense. I've had to read outside my field to keep up with all the educators involved in badges for learning, including developing a deeper understanding of Bloom's taxonomy (you cannot avoid Bloom's!) 

I was reading that Bloom's taxonomy was initially created as an assessment framework, not an instructional one (from Daniel Schwartz and Dylan Arena's excellent MacArthur report: Measuring What Matters Most.) We know that badges and assessment are a degree different from each other, but in terms of "teaching to the test" I think you're right that badges could very easily have the same effect as assessment on instructional design. 

Schwartz and Arena write, "in the way that assessments always manage to do, it (Bloom's taxonomy) has commanded the instructional enterprise." 

The only difference I see is that a big part of the badge movement is to recognize learning anywhere, anytime. I mentioned in response to Cathy's comment that we are connecting different hefts of learning, and perhaps the "teaching to the test" issue is moot for some of these learning situations. 

I really appreciate comments like the one you're making because it gets us to talk about learning when we're talking about badges. I'm curious to hear back from people who have educator backgrounds how you think the badge-first approach might be valuable in some of these other types of learning experiences. 

 

Angela Elkordy

Bloom's "Limitations" or "Opportunities"

I've found in the badge development for my research, that despite having several levels in a learning trajectory, cognitive objectives overlap so often as to become in practice, not very helpful -- particularly when engaging in problem solving, critical thinking and analysis as well as "producing" as a result of these processes. In addition, the broad categories become "fuzzy" and lacking in specificity -- especially when translating into practice, in my experience -- 

bjoseph

Useful frame

This is a really nice idea. I'd love to see a list of HASTAC grantees, or others, and see where they align, and what lessons we can learn from that.

slgrant

Thanks Barry! I just sent an

Thanks Barry! I just sent an email asking the Badges for Lifelong Learning grantees to let me know what bucket they see their projects belong to (more or less, since some of them will be more like one bucket than another, and some will have started in one bucket then moved to another). I've already received lots of responses, so I'll loop back with the community and show those results. 

Sarah Blattner

4 Middle Schools - Lens from the K-8 community

Sheryl,

I am working with 4 middle schools on developing badge learning programs.  I'm happy to share how each school is approaching their badge learning design.

I really appreciate how you are considering not only the different classes of badge system design (your categories above are quite comprehensive), but you are also paying attention to how daunting it might be for newcomers.  From working with a variety of schools with different resources and infrastructures, I am learning that one of the biggest obstacles for the early adopter schools is finding easy to use tools and finding ways to connect their badge learning ideas to a system that is not too cumbersome or challenging for teachers and students to manage.  Many schools already have technology infrastructure in place and some have learning management systems.  They must find a way to make their badge learning programs compatible with what they already have and often on a very small budget and few staff to do the heavy lifting.  It is usually the "edupreneurs" in these schools who are the early adopters and blazing new trails.  I would also like to mention that the teachers are really committed to making their badge learning programs tied to deep, relevant, project-based learning. 

Wonderful post! I look forward to seeing how these classes evolve and how other projects define themselves.

slgrant

Badges in the K-8 community

We have to figure out badge system design for this community!  If we can use badges to lessen some of the damage of multiple choice testing and end-of-grade testing, that's an enormous step forward. Badges will probably exist side-by-side with multiple choice tests, rubrics, and quantitative metrics, is my guess. But to really see if that's true, it has to be remotely possible to design in this environments. 

What you say about the user-experience of badges for teachers is so important, and having platforms that snap (relatively) easily into pre-existing content and technology (plus limited bandwidth) is going to determine how widespread badges become in schools. One of our grantees -- LevelUp -- is piloting a system like this in the Adams 50 School District in Colorado. What impresses me so much about their system is how much thought they're putting into the teachers' experience (one of the developers is surrounded by teachers in his family, so that helps!).  We can't just count on edupreneurs and early adopters -- or if we do, we need to make it easy for them to demonstrate why badges matter. 

You have a unique perspective introducing badges into different environments, and I would love to hear what that's been like for you. If you write a blog post about it, please let me know so I can follow your work. Or feel free to share here, or email me if it's easier. 

April.m.moore

Layered Build -- for K-12 Student Info Systems

As someone working on badging from within a large urban K-12 school district, I believe our program fits into the "Layered Build" class of badges. We designed our student badging portal to integrate seamlessly with our existing student information system and issue badges for reaching a variety of student mastery and achievement goals that already exist in our system. We began with 12 badges, from grades 4-12, highlighting critical benchmarks on a student's Passport to Success, our college and career readiness initiative.

Since implementing this badging program districtwide (with a soft launch last spring, and a full launch this fall), we are now exploring additional badging opportunities at both the school and district levels. Each school has an "Ambassador" who works with site teams to develop badges that have the potential to motivate and recognize students at their particular school. Our hope is to increase motivation from students who may not be motivated by a grade of "A" on their report card but are motivated by their peers. Thus we're hoping to leverage to social component of badging. For the student perspective on badging, see the newspaper article published recently in our local newspaper.

As we add badges, I'm not sure if our program would still be considered a "Layered Build" and am interested to hear your thoughts. We're planning to add badges for student growth (improvement on standardized tests, English learners redesignated to English proficient. etc/). In addition, we're adding bages for "soft skills" such as completion of community service and other extra-curricular elements. These badges can be issued through our student information system or directly through the portal we have designed.

Our K-12 badging portal 21st Century Badge Pathways launches open source November 7.

slgrant

This provides an interesting

This provides an interesting layered build example. I'll have to look more closely at the platform -- I haven't looked too closely at different platforms, but it raises a good question about whether some platforms are more or less suited to different kinds of design approaches. I hope we hear from the platform builders about their thoughts on these categories. Sometimes it's hard to tell from looking at the marketing material how much customization they are able to do, or how much collaboration is involved in setting up the badge system. 

Thanks for sharing the link to the newspaper article about your students. I agree with the quote you made in the article, you are definitely pioneers in this area! Especially a badge system operating in a school system. I've only heard of two, and they're both Badges for Lifelong Learning Competition projects.

Starting with limited numbers of badges during the pilot and then building from there -- that reflects the advice that came out of the research I did around the badge system design approach buckets. I'll have another post up once I've had a chance to go through that data in more detail, but it sounds like your team discovered that 12 was about the right number (similar to what others have said) of badges to start with. 

In terms of whether you remain a layered build or not -- will you be creating the badges first and then developing curriculum around them? 

April.m.moore

Layering buckets?

Sheryl, you concluded with a great question, and one to which the answer is as yet unknown. So far our school ambassadors continue to layer badges upon our existing system. But as we exhaust the recognition components within our current systems, the creative process is moving outside the "box" of our k-12 education system. Given the forward momentum of our conversations, we are likely to be layering badging systems (using our pathways portal that Karen described so well). At the school and district levels, I imagine we'll continue to use the layered buckets. However, at the classroom levels, teachers may be creating badges first and then designing the curriculum to match them. This is particularly likely with our new Common Core State Standards. We have found our badging initiative to be very timely with this new style of teaching and assessing student learning. We’ll keep you posted as we continue to implement and explore the badging potential within k-12 education.

jimdiamond

two questions about the newspaper article

Hi April,

Thanks very much for sharing these links with the group. It's very exciting to see a badge system being implemented across an entire district. That's the first case I've heard of it.

As someone who's involved with digital badges mostly on the research side of things, I was especially interested in two items in that newspaper article that I wonder if you'd be willing to share additional information on—

My first question relates to student motivation and any theories you might have about how badges relate. The achievements mentioned for middle school include meeting math standards and taking college prep exams. Is the idea here that a badge will incentivize students to invest in those achievements in ways they might not otherwise? Similarly, an achievement mentioned for high a high school badge is earning certain SAT or ACT scores. Again, would students be less inclined to score as highly? Or, are the badges targeted toward students who seem to be less motivated to invest in these achievements. In that case, have you seen any change in behaviors yet among that population? It would be very interesting to no more about why lower-performing students might be interested in those badges.

My second question relates to the quote from the student regarding "advertising to employers." Does the district have an agreement with employers through which the badges represent certain skills that students without a college degree might not ordinarily have? The ways in which such an understanding was reached would be incredibly value to many others in this community if it's something you're able to share.

Thanks again April.

 

kbjeffrey

Passport to Success as a Pathway System

I've had the privilege to work with April and the team at Corona-Norco.  One of the aspects of P2S that makes it so innovative is that it is really a System of Pathways integrated with real life activities.  The P2S pathway system provides a very rich combination of motivations for students, their parents as well teachers and other school staff.

When a student completes a pathway, they earn the corresponding "pathway badge".  Each pathway has its own age appropriate goal:

  • Elementary School:  Ability to organize their work, manage their time, complete assignments, and communicate and collaborate with others. 
  • Middle School:  Acheivement in Mathematics and English Language Arts as well as the development of an Individual Learning Plan to promote personalized pathways for student success in high school and beyond.
  • High School:  Engagement in deep learning to become strong thinkers.

Each of the badges in the pathways, is really a pathway on its own and each is earned over an extended period of time.  Here is an example badge for each pathway:

  • (Elementary School) Outstanding Work Habits:

    Strong students are able to organize their work and complete assignments on time and in an excellent manner. It is important to establish a quiet working place at home for students to read and to work on assignments. Outstanding work habits are exhibited by students who write down assignments, bring home required textbooks and notebooks, complete and turn in all school work, and study for tests.
     
  • (Middle School) Individual Learning Plan:
     
    With the guidance of parents, teachers, and counselors, students complete an Individual Learning Plan (ILP) using Career Cruising as a resource. This four year plan is developed from various assessments which help to identify the interests, skills, and learning styles of each student. Each ILP maps out course selections, career options, and college preferences for each student. To make it a working plan, ILPs are monitored and adjusted each year.
     
  • (High School) A-G Courses Fulfilled:
     
    Both the University of California (UC) and California State University (CSU) require freshman applicants to complete a sequence of courses in high school with a grade of C or better to meet the minimum level of academic preparation to undertake university level work. These courses are referred to as 'a-g courses' because of the letter each subject area is assigned: "a" History/Social Science, "b" English, "c" Mathematics, "d" Laboratory Science, "e" Language Other than English, "f" Visual/Performing Arts, "g" College Preparatory Elective. Since these courses foster critical thinking and study skills, private universities also support the a-g course sequence as a gauge of meeting a minimal level of academic preparation for college.

As the P2S website says to parents:

"We encourage you to monitor the progress of your student in achieving these Passport Stamps. The journey to obtaining stamps will differ from student to student. ...  After all, students possess a wide array of talents not captured by these Passport Stamps."

April.m.moore

Thoughts on Student Motivation

Jimmy, in regards to your first question, we have already seen quite an interest among the video-gaming population of students at the intermediate and high school levels, predominantly our pre-teen and teenage boys. These students may not naturally see meaning and value in an “A” grade, but they thrive on social, peer recognition and already understand the concept of digital badges (in fact, they are teaching their peers about them).  It is too early in our implementation to see quantifiable changes based on increased student motivation. However, in our early adopter schools, administrators have noticed increased student knowledge and ownership of their educational pathways, what it takes to achieve the next badge/benchmark/college readiness indicator. Students are more active in the process rather than passive.

Regarding the “advertising to employers” concept, we are encouraging our students to use their badges as components of their digital portfolio, showing off their skills to potential employers and universities. We believe that a fully completed Passport 2 Success indicates a student that is college and career ready. As our district administrators meet with our business and industry partners, community leaders, and local universities, they are showcasing the readiness of our students and using the Passport 2 Success as evidence of student readiness. This is very much a work in progress.

The next phase of our badging implementation, which we hope to launch this spring, includes tangible rewards in terms of redeeming badge “points” for rewards. In conjunction with our local businesses, community partners, and student leadership, each school will offer tangible incentives for which students may redeem their points. For example, one high school may provide movie tickets, free homework passes, and discounted tickets to an athletic event or school dance while an elementary school may offer tickets for a free ice cream or frosty, lunch with the principal, or other age-appropriate incentive. Our badging system has the capacity to offer these now, but we are delaying implementation so that each school has time to build up their list of incentives, in conjunction with their local business and community partners. Early research indicated that offering these incentives districtwide (with over 54,000 students) would be too much of a strain on our community and business partners, so we built the badging system to allow for local and differentiated “points stores” tailored to the specific school and community. We hope these tangible incentives will further draw interest to our program and spur student motivation.

aubrecht

Earthworks Rising Badge System & website

I think Sheryl's way of conceiving of badge development is fantastic. Just yesterday, I was asked to help design a badge system for a conference where I have created and organized the game-based learning track for three years. By looking at Sheryl's list, I could identify that they are describing a Responsive Build. That helped me to know what information I need from them and to know what we need to discuss and decide upon before enlisting the developer's input. It was really helpful! and I shared the HASTAC resources with them, which are extensive. Great job!

The Earthworks Rising project fits within the first category: New Build. Because it was a new build, we first had to find a suitable way to structure the content. We hit upon the idea of a card system, but never figured out how to integrate the content with earning badge slices outside of a one-size-fits-all approach. In retrospect, it would have been best to agree upon how to scaffold the learning content as it relates to badge earning, even if we didn't have time or money to build the structure. Knowing this is central to the overall design of the badge. Being a pie badge, the levels must be built into it as one unit as opposed to multiple badges that are awarded in sequence. Trying to fit this within a particular design tool's constraints also complicated the decision-making process. In retrospect, I think the badge must be conceived and designed before determining which tool to use for the build. Otherwise, it's just too complicated and too easy to loose sight of the what earning badge slices means to the overall build.

Thinking of it as Sheryl has conceived it, made me realize that our difficulties as a team came from not being able to clearly articulate and agree upon how the content and the badge interacted. A new build is a daunting task. One cannot narrow the scope until it has been blown-up large. Through grappling with the various aspects of this process over the past year and now stepping back, I can more clearly see the relationships among the various parts and how to restructure them. We've figured out how to integrate the learning content with the badges.

Last week, my colleague took all of our 20 topics and 20 challenges and created a path with branching so that we could see where the badge slices will be awarded and how they level up in relation to the topics and the "guided journey" that earning the badge slices creates. It's all laid out on a "huge" piece of paper and I'm making a digital mind map of how that might look which will allow us to play with it further and figure out what is missing. It will also provide a way to find out from our users if the path that is both linear and non-linear brings the user along in a meaningful and engaging way, providing information as it is needed and functions as we conceive it. We imagine it like a spiral with bursts of strands that separate and expand around an idea (such as pilgrimage) then weave back in to the core topic of earthworks built by American Indians. Each burst or expansion provides the non-linear path and then unlocks more challenges for the user. The earned pie badge slices will represent some required and some elective challenges. In this way, the learning is scaffolded and through the process, certain themes and topics are revisited, hence the spiral image. In this way, we are concurrently desining the badge and the learning content within the context of an overall website/ platform design of a card system, social options for users, and things to do and make...which leads back to badges to earn!

 

slgrant

This is great feedback -- I'm

This is great feedback -- I'm glad it will be useful for you going forward, and also gave you a frame to look at your own design approach. It will be interesting to see over time how people experience new builds. They may seem particularly daunting now because everyone is figuring it out for the first time. Hopefully we'll get enough feedback from projects like yours so we know what the right questions are to ask, and how to identify pathways that make sense for organizations in similar situations. 

I also love the idea of visualizing the branching paths of your badge system. I always appreciate that "view" when I'm in a game-based environment, and I can see how a learner would get the same benefit. 

Cathy Davidson

Mozfest Badges Hit the Basics

I love this conversation--it's one of the most in depth, engaged, we-can-do-this conversations I've seen.   Thanks for your wonderful post, Sheryl, and to everyone for these great comments.   Sheryl, your work on the badge systems in the DML Competition--your meta analysis of thirty+ different systems--is just inspiring.   It is so hard to think about this that the taxonomies you are providing will be an enormous step forward.  In addition to the work others are doing right now, including Dan Hickey's group at IU and Mozilla, we are getting closer and closer all the time to being able to start operationalizing badging systems as something that could displace the inflexibilities and inequalities of existing systems---and could offer something new to worlds of learning that we don't even know how to "count" right now.

 

As an amusing but also instructive side note:  I happened to go to the Mozfest site just now (thank you, Twitter) and I'm intrigued by what our collaborators Carla and Sunny have come up with as badges for MozFest.    There are lots of categories with good graphics.  You click on the badge and it gives you more information about what the badge is for.  And each "station" at Mozfest has code numbers it can give to attendees.  Once they have that code, they can then redeem the code and get the badge.  

 

This is a much simpler form of credentialing and exchange and transparency than most of you are talking about here but for an existing institution with a known identity (i.e. Mozilla), on a specific occasion (Mozfest), with a specific hierarchy of badge conferers and badge earners (session chairs and audience/participants), and a technical infrastructure in place (OBI), this not only is doable but shows how a system can work and what some of the most fundamental components are:  

  • existing institution
  • specific, limited scope or occasion
  • clear distinction between issuers and those wishing to have badges
  • badges that clearly define the scope (transparency) and, for want of a better word "tone" or "serioiusness" of the achievement, clear picture of what, exactly, was earned ("Mozfest Reveler" or "Mozfest Karaok-er" or "Mozfest Session Leader"!)
  • technological infrastructure for holding the metadata  (note:  no promise for how long, etc--but for how long do you need the world to know you were a Mozfest Reveler?)

It's quite simply, illustratively brilliant and inspiring to see this in action at this scale and is great to think about it in terms of the complex arguments about learning and replacing the whole 100-year-old apparatus of multiple-choice testint our institutions of education worldwide have made their one and only standardized interoperable (and utterly non-transparent: i.e. your score reveals how well you did on the test being scored) system. 

Here's the link to the Mozfest badging system

http://schedule.mozillafestival.org/#!/badges

 

dthickey

On the Study of Educational Innovations

Sheryl--

I spent much of a long bike ride yesterday thinking about this.  You have provided a much more nuanced and far reaching framing of something that we learned when interviewing projects about how their intended practices in their proposals were transformed when they projects attempted to enact them in the reality of their ecosystem. It did not end up as one of our design principles for recognizing learning .  But we touched on it as a "design implication" at the bottomand now it is is emerging as part of the "big conclusions" about the way that the system constrains the learning that can be recognized, and those decisions in turn constrain the assessment that is possible, 

I frankly think that the insights that you uncovered are actually more useful than what we uncovered.  In addition to being smart and persistent, you were in the center of all of the hard work at HASTAC to help these projects actually design and implement their badge systems.  We discussed this at lenght as the Design Principles Documentation project was taking shape.  Mimi Ito was quite insistent that this project needed to be very ethnographic and not provide guidance to DML awardees beyond helping them gain access to insights emerging from other project and the more general design principles across projects.  

I was a bit resistant at first because I stopped doing more naturalistic or experimental work a decade ago because it did not seems to deliver the insights need to change practice. As Ellen Lagemann has said, the most scientifically valid insights for reforming practice come from efforts to reform practice. In addition to my confidence in Mimi's judgement, three things she said/implied  made me happy to frame Indiana's project as a very naturalistic (rather than interventionist) effort.  The first was that and HASTAC had been charged with supporting the projects and that we would be stepping on each other's toes.  The second was that my project would not have the resources to provide that support.  The third and most important was that a more naturalistic project would get at unique insights that would otherwise not be possible.  

Our different and highly complementary takes on the same challenge is a great example of that last point.  It is also why it is awesome to have three differnet teams (HASTAC, Mozilla, and IU) looking at the same issues.  I am looking forward to working with everyone over the next eight months figuring out how to package every thing that we learned about badges so that is useful for everybody going forward.  But also think that we learned some important lessons about fostering educational innovation more broadly.  And I hope we can think and write about that as well.  It has been an amazing ride so far,

 

dthickey

And--

I think we should also recognize another pretty amazing thing.  Nearly all of our knowlege building around badges is taking place outside of the traditional peer-reviewed context. What does make it out of this process and into the peer review is likely to be mostly (a) correlational studies about someting about badges and individual differences (they exist!) that are theorized post-hoc, or (b) experimental or quasi-experimental studies showing that badges "work" (they do!).  

Thanks to Cathy and David's leadership of HASTAC, we have this shared public space where we can hammer these things out in real time.  While it may not be peer reviewed in the traditional sense, it is definitely resarch and our peers are certainly reviewing.

jimdiamond

But where's the evidence?

Hi Dan,

It's hard for me to tell whether your parenthetical exclamations in the first paragraph are tongue-in-cheek, but those things are actually important to determine, aren't they? Certainly none of us should be impressed by any study that concludes that "badges work." I would be impressed, however, by a study that concludes that there is some systematic, logical, and likely valenced relationship between a PROCESS associated with the achievement of some badge in some population and some dependent factor. Is that kind of knowledge building occurring now in this community? It seems like some design issues are getting hammered out for sure, but are issues related to (sorry to bring these up again) validity, reliability, and mobility being hammered out, as well?

By no means am I arguing that correlational studies are the only things worth looking at. But for any of us to begin suggesting that "badge systems" (which is really how I'd prefer to start referring to the things) should replace (or even coexist with) the more standardized systems for recognizing achievement that currenlty exist, there's a lot more systematic work to be done. And while I appreciate Cathy's example of the Mozfest badges, I also don't care much about those "low-stakes" badges (and there will be "high stakes badges," for sure...) and I wonder how much Mozilla would if there was an expectation of some future outcome associated with them.

I guess that I'm just trying to understand more about what's being hammered out here. I agree with you that transparency is absolutely key and that one of the most attractive elements of the badging ethos, but I'm also mindful of the transparency associated with portfolios, which have yet to replace the current system. If we're (speaking very broadly) going to present badges as alternatives to standardized systems, then I think we do need to demonstrate things like validity and reliability, to say nothing of sociological studies that look at how these systems are assimilated by those with power. And this isn't just for badge naysayers, but for the populations whom we're suggesting might benefit most—those that are being left out or behind by standardized measures. The SAT, after all, was supposed to be completely meritocratic, removing class biases from determining who should go where for college.

Something I've been thinking about lately with repsect to badges is the fact that they are yet another set of credentials. If we are already worried about the obsession with credentials in our society, then why are seeking to introduce another set? One answer is presumably that this new set will be better than the old one. But we'll need to see the evidence.

bjoseph

Badges are not a noun but a form of communication

Jim, I agree. From my earliest work with badges, I always insisted on calling them badging systems, in part to break peoples from the 1:1 association many held with the Scouts but, more importantly, to shift the frame of reference from a trophy to be collected to a token within a communications network.

As I am trying to find words to explain to my colleagues at work, badges are not an object to be collected but a signifier of social capital which can only be spent within a learning ecology, one that situates a particular source of badges (e.g. a learning space) within youth’s other learning spaces - such as school and out-of-school learning spaces - and, at the same time, is connected to the broader ecologies of higher education and the workforce. In other words, a badging system would need to afford youth the opportunity to aggregate badges across many sites of learning (the way an adult uses a resume to design a list of places of employment) and feel confident such a presentation would be valued by those they are seeking to convince.

But today that is still a hard case to be made - yes, tools and infrastructure are being built, but I can't say to a youth in the museum where I work that in, say, two years any employer or higher ed institution is going to care about a badge we give them today. And without that promise, or the ability to aggregate their badges with other institutions where they learn, I find many (but not all) youth don't feel it's worth the effort.

In addition, that is why I too am not as interested in badges at conferences as a strong use case. If there were a Foursquare of badges - some tool we are all using that connects the events in our lives with both those sharing the same event AND those within our distributed social network (friends, family, etc.) I can see that event badges can accrue meaning. The would be a way to communcate laterally (with those we are with) and in a distribute way (across our remote and distributed networks). But, abscent the use of a ubiquitous tool, I don't understand what a badge earner is communicating with through an event badge - when submitting evidence, when earning it, when sharing that it was earned. I am not saying it is not there, but I think those three areas of communicating are much strong in an on-going and/or extended learning space, such as in an after school program within a permenant institution. And I think we will learn more through emphasizing practices learned in those contexts.

slgrant

Darn, I wrote a much longer

Darn, I wrote a much longer response to your comment, Dan, and my super-slick mouse performed some weird automatic function and I lost everything. 

I just wanted to iterate what we spoke about on the phone, that I'm so grateful you took a naturalistic approach. In thinking about toolkits and exemplars, we'll be able to look at processes and choices instead of "this system is an examplar" and "this one is not." That distinction seems profoundly important at this stage -- to exemplify the process of design. Use buckets to create map pins, and then look at instances of exemplary design approaches. I think that will be far more useful to newcomers just getting started.

Also, I would be remiss if I didn't mention how hugely influential your work has been on my thinking. You can see in the photos if you look closely that I started to try and connect my work with the design principles that your team has been working on, hoping to see some patterns. I ran into the same problem I was encountering in the methods section of my dissertation proposal, which is to figure out a way to justify which badge systems would include in my comparative case study analysis. I figured if we could figure out a simple organizing principle, then it would be easier to manage the variables and complexity. 

It has been an amazing ride, I agree -- and will continue to be. I'm looking forward to seeing how this conversation around the 5 Buckets for Badge System Design evolves. It's great to see that it's resonating with people.  

 

 

slgrant

Excellent point, and thanks

Excellent point, and thanks for linking to Mozfest, Cathy. This is exactly why we need to stay as value-neutral possible about the different design approaches -- the context in which a badge is being issued makes all the difference in deciding which approach is best. I don't know if Carla and Sunny started with the badges first (they're a little busy running MozFest right now (!), and I'm especially interested in hearing what Carla has to say -- she wrote to say she would weigh in when she has a moment) -- but if they did, it would make perfect sense for that context. Maybe, too, people will gravitate toward a badge-first build for certain types of badges, and certain kinds of outcomes (like participation or engagement and more bevioural and identity types of badges). 

Badges-first builds may be ideal for learning experiences like MozFest and maker fairs, or for learning experiences in which the badge fulfills different functions than, say, badges issued in a K-12 school environment.

Conferences are one of the potential learning environments for badge-issuing that I'm excited about.  They're a critical part of lifelong learning, both inside academia and outside, and yet most of us who participate in them don't include them on resumes unless we presented papers or were on panels. Yet conferences are often some of the best ways to think across disciplines and begin new lines of inquiry. If we say that learning happens anywhere, anytime, then we need badge design approaches that can truly adapt to anywhere, anytime, including something as ephemeral and distributive as a conference or festival. 

That's one of the reasons I made the choice to use "learning content" instead of curriculum, and "technology platform" instead of "learning management system" to describe the badge system design approaches. Jim's point about being cautious, and Dan's use of "perilous" to describe badge-first builds are super important for the kinds of learning environments where people are going to demand serious amounts of credibility and rigor. As our field grows, hopefully we'll start to hear back from people about their design experiences and which approaches work best for different contexts. 

 

Cathy Davidson

Great point about "real time peer review," Dan!

Dan--you raise a fantastically important point here.  Thank you!  I hadn't even thought about it but, if the original post and each of these comments were happening in refereed journals, it would be three or four years at least in the unfolding.  Each would be not a conversation that presumes people are reading the antecedent remarks but each would require a rehearsal, as fair as possible (and we KNOW how that sometimes goes in literature reviews), of past research, and then a making of one's onw mark in the "mistakes" or "blindspots" or "failings" or "limitations" of the previous. 

 

And in the time to publication, two or three other side-shoot articles may have appeared that then are not answered by the other one, and so the conversation is lopsided, fraught, and often comes with a baggage of implicitly ignoring or not responding to "public knowledge" just because of print or peer-review lag time.   

 

Instead of all that other-presentaiton and self-presentation, in this conversation I feel as if I have learned so much, partly from reading you and Sheryl and Michelle and Karen and April (and apologies to whomever I inadvertently left out:  all of the above!), certainly some of the most knowledgeable people on the planet, working this through together, in public, not as a pose but as problem solving.

 

You are inspiring.  All of you.  And thanks, Dan, for reminding us that it doesn't always happen like this.  It's not just about having a tool, it is about having built an engaged learning community ready for participation and welcoming to the public at large.   Thank you so much for such a great insight on the advantages of NOT always having work peer reviewed while it is in process like this. 

Carla Casilli

Different lenses and the value of context

Really fascinating approach to thinking about how badge systems come into being, Sheryl. Thanks for investing the time and post-it notes on this. :) Reading through this and the related comments provided me with much insight and enjoyment. 

Having now worked with a variety of orgs who range from one end of the spectrum to the other, I can say that all of these phenotypes exist, to borrow a phrase from the field of biology. And now that I write that, I'm wondering if there is another cut at this from a genotype / phenotype approach. Will ponder that as I've been having some fun thinking about badges, badge systems and genealogy. More on that in a later blog post.

Your thinking here corresponds with my own perceptions: there are multiple ways to categorize the thinking about badges and badging systems. We've seen a lot of the layered build and the responsive build. In fact, I'd like to suggest that the responsive build is typically what we're seeing in the field of academia and accounts for a lot of the concern about badge validity.

The five types you've described constitute a great lens for folks who are considering implementing badges as well as those who are aiding them. I'd suggest that there are additional ways to view these systems, too. Thinking about assessment types (include 1 assessment type, several, all) or goals (highly focused system goals vs. loosely constructed goals) or possible pathways (linear, multiple, none) are others—obviously, all of these lenses exist on continua.

As for where to start in the building of a badge system, in an effort to demystify and simplify the process for folks who are starting from a new build, at workshops and seminars we have had the occasion to encourage them to start small, often with anywhere from one to three badges. While this may not seem like a fortuitous or dynamic beginning to a badge system, getting people to create something—anything—shifts their perceptions of what's possible and makes something out of nothing. And this shift is seismic: because for the new build teams, beginning to think about and create individual badges is half the battle. The other half is building a meaningful system.

As for badges that are considered "low stakes" or "high stakes" here it's necessary to consider that context plays a very large role in the interpretive value of a badge. For example, an attendance badge may seem meaningless to someone with little to no knowledge of what went into earning that badge. Indeed, for someone who lives in a major metropolitan North American city, a badge earned for attending a conference may carry little meaning—for the earner or for a badge consumer. And yet for someone who lives in sub-Saharan Africa, a badge for attending a health seminar in a town may have required 3 days worth of travel and can prove significantly meaningful to the earner and to a badge consumer. Consequently, we must be careful about dismissing badges because of our own limited lenses or predispositions.

As for the MozFest badges, I'm pleased that Cathy enjoyed them. They were designed and created as participation badges: badges that certify the cohesive value of common experience. It seems to me that some of the best and most valued badges are about recognizing one's role in a community—even if that community and role are temporary. Perhaps especially if that role is temporary. Of course there's much more that can be said about badge types but that will have to remain for another conversation. 

Thanks for kicking off this great and rich discussion and thanks to all who have commented. The open badge ecosystem is a wonderful and varied place. 

bjoseph

badges to shape identification with peers at a temporary event

Carla, I appreciate what you just shared above. Thank you. And I do see your point about, as I might frame it, using badges to shape identification with peers at a temporary event. Might you have any figures post-Mozfest about how many people requested badges and how many badges were conferred?