Research Design Principles for Studying Learning with Digital Badges

Summary

Web-enabled digital badges are quickly transforming the way that learning is recognized in schools and in informal learning contexts. But there are few examples or models for studying digital badges. This post introduces six design principles for studying learning with digital badges that are emerging in the Design Principles Documentation Project. These principles distinguish between summative, formative, and “transformative” research, and between using conventional forms of evidence and using the evidence contained in digital badges.  This post is quite long.  I pondered posting a short version here and a longer version at Remediating Assessment.  But I just did not know what to cut so just cross posted the whole post.  And I believe that putting the entire draft here will foster some healthy discussion.

Background

The Design Principles Documentation Project is capturing the design principles for using digital badges that are emerging across the 30 HASTAC/MacArthur Foundation Badges for Lifelong Learning projects funded to develop badge systems. Previous posts introduced the principles the project uncovered for using badges to recognize learningassess learning, and motivate learning. This post introduces the principles for studying learning. While the more inclusive term “study” is used here, this is really about what is often referred to as “research and evaluation.”

Of course, research and evaluation are contentious topics in education. One of the reasons for this is that people disagree on what counts as “evidence” and what methods count as “scientific.” A 2001 report by the National Research Council laid out the argument that the “gold standard” of scientific educational research is randomized experimental trials. But the NRC also recognized that many of the most important ideas that might be tested in experimental research are unlikely to be discovered in experimental studies. This seems certain to be the case with digital badges in education. 

The principles summarized in the other three posts provide a nice context for thinking about where the science behind badges is going to come from.  These principles are more general descriptions of the specific practices that emerged as the various project figured out how to use digital badges in their contexts.  We expect these principles to be quite useful for others who wish to use badges—particularly when they are linked to examples in project and to relevant research resources.  But most of these principles are not framed as “hypothesis” that could be tested in an experiment.  Even if they did present testable hypotheses, the results would probably not generalize from the context where the experiment was conducted to other badging contexts where those findings might be applied.

Research and Evaluation of Digital Badges

Thanks to the DML competition and extensive media coverage, many schools and programs are considering using digital badges. This means that many are also beginning to ask about the research evidence concerning the effectiveness of digital badges for student outcomes. Digital badges are so new that there are very few published studies and just a handful have yet to make it through the peer review process. Sheryl Grant’s annotated bibliography provides  a nice summary of what badges research there is, along with a lot of other relevant resources from other contexts.  After the initial badges competition, HASTAC announced a separate research competition to study digital badges and made awards to five badges research projects. Some of these will be discussed below.

The DML 2012 badge awardees are still tweaking their badge systems. Our interviews confirmed that most projects started with a pretty clear idea of the learning they wanted to recognize with digital badges. But many are still sorting out how to assess that learning, and most are just beginning to consider how their recognition and assessment practices would impact motivation.

Few of the projects included any formal research or evaluation studies in their original proposals. Notably, the DML 2012 competition did not require that proposal include detailed evaluation plans. I think this was a wise decision. It is my opinion that requiring detailed evaluation plans would have led projects to prematurely search for “summative” evidence that badges were effective before they had a chance to figure out how to best support learning with badges. But our interviews with project leaders revealed that many were starting to think quite seriously about the sorts of studies they might conduct now that their badge systems were taking shape. However, many of them were also unclear as to where they might start, and few of them had even begun to grapple with the far-reaching idea of using the evidence contained in the digital badges in their research.

Important Distinctions for Studying Digital Badges

In our initial efforts to understand the studies of digital badges that were taking place or might be carried out, we uncovered three dimensions that seem helpful:

Systematicity. Arguably, the distinguishing feature of “research” is that it is systematic. Research involves systematically gathering some sort of evidence and attempting to document things in a way that could inform others. The design principles that the DPD project is identifying for recognizing, assessing, and motivating learning are mostly not coming out of systematic studies. Rather the project is attempting to capture more informal knowledge as it emerges as teams get their badge systems up and running. This fourth strand of the DPD project is concerned with more systematic efforts to create new knowledge concerning digital badges.

Purpose. Building on the assessment literature, one can distinguish between summative studies “of badges” and formative studies “for badges.” Summative studies are more naturalistic examination of the way the world is, while formative studies are more interventionist efforts to change things. While most summative studies are intended to be formative, they do so less directly. One can also distinguish transformative research that examines how entire learning ecosystems are changed or created around badges.

Evidence. We are distinguishing between studies that don’t use the evidence of learning contained in digital badges and studies that do use this evidence. What makes digital badges unique is that they contain the actual evidence (or links to evidence such as artifacts produced by learners) to support particular claims of proficiency or accomplishment. There is usually a lot of negotiation involved in deciding what learning should be recognized with badges and how that learning will be assessed. As such, the evidence contained in badges will embody the values of the program or organization that issued them. This means that the database of issued badges has enormous potential for studying learning.

Focusing only on systematic studies and crossing three purposes and two types of evidence yields the following six research design principles. '

These descriptions draw on selected examples from the DML competition as well as the studies being conducted the awardees in the Digital Media and Learning Reseach Competition on Badging and Badge Systems Development. This will be updated over the next year as addition systematic studies get underway.

1.  Research OF Badges 

Summative studies of digital badges are likely to be the largest category of badges research. Some will rely more on interpretive methods and qualitative evidence. For example, HASTAC Badges Research awardee Katie Davis (University of Washington) will study how students and teachers in the Providence After School Alliance (PASA) experience the badges used to give high school credit for expanded learning opportunities. Katie will use interviews, questionnaires, and observations to explore (a) how badges fit in the academic and peer culture, (b) the role that badges play in motivation and achievement, and (c) whether badges connect in-school and after-school experience. Likewise, one of the studies being carried out by HASTAC Badges Research awardee Jan Plass (New York University) falls in this category. Plass and colleagues will video record game play in publicly available games with and without digital badges. They will then analyze those recordings for trends and insights into participants’ perceptions and valuations of badges, and for changes in gameplay patterns due to badges.

Other summative studies of badges might rely more on correlational methods and focus on individual differences and variables. In one of the first published peer-reviewed studies of digital badges, Abramovich, Schunn, and Higashi (2013) explored mastery-based and participation-based badges in an intelligent tutoring system for teaching proportional reasoning in mathematics. They measured self-reported motivation toward mathematics before and after the game, pre-achievement of proportional reasoning, and opinion toward badges. Correlational analyses revealed both positive and negative effects of badges on learner motivation, and that these finding interacted in turn with student ability and types of badges. The Badge Impact Survey (BIS) that Jan Plass is planning to develop based on the results of the initial observational study promises to be quite useful in this class of studies.

Other studies of the impact of digital badges will use experimental methods, such as creating different versions of the same types of badges issued. For example, the final study that Jan Plass has proposed will modify a geometry game to examine the impact of two different types of badges.  They will compare mastery badges (based on players’ own progress mastering learning goals) and performance badges (based on players’ performance relative to others). They will examine impact of the different badges on a range of individual outcomes, including motivation and learning. This study promises to provide generalizable principles about the impact of these two common types of badges in game-based learning environments. Other summative studies will be more consistent with typical program evaluations. While DML awardees were not required to include formal evaluations of their badging programs, some of them are evaluating the programs as part of their larger organizational mission.

2.  Research FOR Badges

Other studies will formatively intervene more directly in badge system design. One distinctly formative effort is the study proposed by HASTAC Badges Research Awardee Jim Diamond of the Educational Development Center. Jim has already been working intensively with the DML/Gates 2012 Awardees Who Built America? (WBA) teacher mastery badge system. Jim’s study is asking some of the same questions as Katie Davis’ study of PASA. For example Jim is asking about the role that WBA badges play in teacher professional development, and examining the ways that badge-related activities influence the development of an online teacher professional development community. What pushes this research into the formative category is that Jim is asking these questions while directly participating in efforts to build the badging system and the online professional development network.

Studying things as they are changing gets messy really quickly. And studying one’s own practice makes it hard to be “objective.” Jim certainly recognized this in his proposal. This is why he is using design-based research (DBR) methods. As articulated by Paul Cobb and colleagues in 2003, DBR builds “local” theories in the context of iterative refinements of practice. Generally speaking, DBR studies start with some relatively general design principles for getting from the current state of affairs to the desired state of affairs. The back and forth process of translating the general principles into specific features yields specific design principles. Importantly, this process also reveals the key aspects of the learning context that support the specific design principles. It is this “embodiment” of the design principles in learning contexts that is presumed to generate useful insights that others can readily build on (Sandoval, 2004).

Two ongoing explosions of badging efforts should offer numerous opportunities for systematic formative research of digital badges. A number or researchers and graduate students are involved in efforts to design badge systems for the 2013 Summer of Learning in Chicago and ongoing efforts of Hive NYC.While it is beyond the scope or timeframe of the DPD project to track of all of these efforts, it appears certain that new models of practice for formative studies of digital badge systems will emerge from these efforts.

3.  Research FOR Ecosystems

Of course, many projects are using digital badges to create new learning ecosystems or transform existing ones. Some of the projects are beginning to study this process systematically. Consider the pilot study carried out by Global Kids of a new badging system for their youth programs. A DML award paired them with DML Badge System awardee Learning Times to implement BadgeStack in Global Kids’ Race to the White House and Virtual Video Project programs. The report of the pilot study provides some examples of what this might look like. The report of the pilot study describes how badges impacted the educational programs that Global Kids had already developed. For example, they found that:            

Global Kids youth leaders received confirmation 48 times that evidence submitted of their work met the requirement of one of thirteen different educational objectives in their programs. At the same time, youth leaders received confirmation ten times that their evidence did not meet the requirements. Both took extra time—for the youth to submit the evidence and the GK staff to review and evaluate—but the goal of providing formative assessment was significantly advanced (page 6).

The report explains that this sort of assessment had never been carried out in the educational programs that Global Kids offer.

Other systematic studies of the transformational effects of badges on ecosystems are likely to emerge in the Summer of Learning and various Hive projects. One ambitious example is the dissertation research of Global Kids Alumni and Indiana Learning Sciences student Rafi Santo. A grant from the New York Community Trust is supporting Rafi’s extended study of the diffusion of innovations in the Hive NYC. Rafi’s study is not focusing specifically on digital badges.  But a DML award to Global Kids should help ensure that badges are systematically implemented across the Hive NYC community. This and other such efforts promise to provide more specific research design principles for studying the creation and transformation of learning ecosystems via badges and other specific innovations.

Formative studies of entire learning ecosystems are incredibly complex. There are many variables to consider, numerous principles and features to be refined, and many methods that might be used. And there is the incredibly complex relationship between mentor/teacher learning and mentee/student learning. While Jim Diamond’s study certainly has some of these characteristics, it seems like he made a wise decision to tame some of that complexity by staying within the DBR framework. However, as the badges community matures, it is certainly going to need to tackle this complexity. Fortunately, a new strand of DBR known as Design-Based Implementation Research (DBIR, Penuel et al., 2011) aims to address these additional challenges. In particular, DBIR explicitly addresses (a) the existence of multiple stakeholders with different perspectives, (b) the crucial and unique role of educators and mentors in DBR, and (c) a concern with developing capacity for sustaining change in systems.

4.  Research WITH Badges & OF Badges

 Using the evidence contained in badges offers new opportunities for summative research of badges. This includes studies of the credibility of claims made in badges. This question naturally has come up a lot around digital badges. A 2012 article in US News & World Report suggested badges might someday overturn the monopoly that colleges currently hold on formal credentials—but “only if badges are proven credible.” As badges begin to function as more formal credentials, employers and college admissions officers are wondering about the reliability of the assessments behind the badges and validity of the claims made in badges. Some have noted that the credibility of conventional credentials (grades and transcripts) are seldom systematically scrutinized. Nonetheless, more formal badges are likely to trigger studies using conventional criteria from educational and psychological testing (e.g., internal reliability, construct validity, generalizability, etc.). Mozilla’s Carla Casilli has written convincingly that the fact that badges are web-enabled means that the validity of the claims made in any badges will ultimately be crowdsourced. This means that evidence from formal reliability and validity studies might be meaningless if relevant personal or professional networks collectively ignore or dismiss that evidence.

The evidence contained in digital badges has many other potential uses. The aforementioned pilot study of badges at Global Kids provides initial examples of the how programs can use the evidence to evaluate and study their programs. Before Global Kids introduced badges, their primary evidence of learning in program evaluations were summaries of blog entries that students were asked (but not really required) to make. With BadgeStack, it was simple to link to a detailed description of the badges that were offered to program participants. Additionally, the details of who earned what badges provide a surprisingly comprehensive complete picture of the learning that was supported by the program. Examining the order in which badges were earned also allowed Global Kids to begin studying the paths that learners took through their programs. Given the challenges that many schools and programs face in evaluating and studying learning, the introduction of digital badges seems poised to unlock enormous potential in this regard.

5.  Research WITH Badges & FOR Badges

The evidence contained in digital badges also has the potential for systemic efforts to formatively improve badge systems. Consider for example, the work of Stacy Kruse, Creative Director of DML 2012 awardee Pragmatic Solutions. Stacy and a team including assessment guru David Gibson are collaborating with the Digital On-Ramps project in Philadelphia and several educational initiatives at the Corporation for Public Broadcasting (e.g.,Road Trip Nation). As Stacy put it, “before I started working with digital badges, I was working on learning analytics.” This kind of experience has left Stacy and colleagues quite enthusiastic about building learning analytics directly into the badging systems they are building, and using those results to dynamically refine what badges are available, how they are displayed, etc.

Our interviews with other DML awardees uncovered some other promising efforts to use the evidence in badges to transform badging systems. GoGoLabs CEO Lisa Dawley and the Planet Stewards project are using badges to connect educational content from the National Oceanic and Atmospheric Administration to the Next Generation Science Standards. One of their challenges is mapping the game-like curricular “quests” to the standards. Such mapping is notoriously difficult and a major obstacle to standards-based reform. Curricular activities naturally touch on multiple standards, and systems need redundancy so that students and teachers can select from multiple activities. Because badges can be more specific and because they contain actual evidence of learning, they open up entirely new formative possibilities for mapping. This same evidence can then be used summatively to examine the learning trajectories that students take.

6.  Research WITH Badges & FOR Ecosystems

Eventually researchers are likely to begin using the evidence in digital badges to systematically study and improve entire learning ecosystems. In this way it seems possible that digital badges might ultimately transform the entire learning analytics movement. But this seems unlikely to even get started until clear research design principles for summative and formative studies using the evidence in badges emerges.

 

References

Cobb, P., Confrey, J., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13.

Penuel, W. R., Fishman, B. J., Cheng, B. H., & Sabelli, N. (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40(7), 331-337.

Acknowledgements

This work was supported by a grant from the John D. and Catherine T. MacArthur Foundation’s Digital Media and Learning Initiative.  Thanks to all the DML awardees for their continued participation in this project.  Katarina Schenke, Cathy Tran, and Rebecca Itow contributed to this work and this report.

 

 

dthickey

TLDR?

My first question is whether this is indeed too long.  If so would it make sense without the examples?

dthickey

Learning Analytics

My second question is about learning analytics.  One of the reasons I am so interested in digital badges is because the data they contain should quite immediately be evidence of learning.  Everybody is interested in learning analytics these days.  Check out the videos from last weeks Learning Analytics Summer Institute for a really current look at what is happening.

But much of the data the people are collecting and analyzing needs to massaged and transformed before it can be used to document or support learning.  It seems to me that LA where badges are involved will be entirely different because the data contained in the badges willl need little or no massaging or interpretation.

Is anybody else out there looking at this?  I would love to help hook up some of the DML Awardees with some of the big thinkers out there in Learning Analytics.  Perhaps a symposium at the LAK 2014?

For what it is worth, my Educational Assessment BOOC will feature digital badges and should tons of data to explore. I am not even sure where we will start analyzing but it looks promising

katie_davis

helpful framework

As I gear up for my research in Providence, it's helpful to have a framework like this to think about the different ways of studying digital badges and what they each have to offer.

In reading through the various approaches, it seems clear that they need to be in dialogue with each other if we're really going to understand the potential of and best practices associated with digital badges. With my own research, I hope that our findings will not just be used in a summative way but will also be used to inform future iterations of PASA's badge system.

The addition of the "I" in DBIR is important. If digital badges are going to gain traction, we need a good understanding of how various stakeholders view and experience them and what's needed to make them sustainable.

 

dthickey

on research functions and research purposes

Katie--

Great points.  You got right to the central issue I struggled with when I was writing this.  Your study is definitely going to provide tons of valuable information that is quite formative for the Providence After School Alliance.  But that function will be relatively indirect.  In many many cases, the findings from what I am labeling summative studies here don't have any formative impact at all.  That is very unlikely to be the case with your study. 

The distinction here is that you are studying the system as it is and as it evolves, but are not directly involved in transforming the things as you are studying them.   That has significant advantages in terms of generalizabiity and objectivity. (This is what Richard Shavelson and others have pointed out in their critiques of DBR. )

Jim Diamond on the other hand appears to be quite actively changing the stuff that he is also studying.  Regardless of the outcomes of the study, practices will be changed and refined as part of the study.  That also has significant advantages, but in some ways (and from some perspectives) it makes it harder to generalize.  This is why I think the emergence of the DBR framework was so crucial.  It makes the distinciton between research that is expected to have a foramtive function and studies that are carried out with a formative purpose.

I have come to realize why these sorts of taxonomies are always written at a very abstract theoretical level.  It gets really complicated when you get really specific.  But I assume it is what the field needs.  And it was certainly what we were tasked to do on our project so we are going to just dive in.  Looking forward to hearing back from you and others.

dthickey

Crowdsourcing Validity

I had a conversation with someone today that made me realize that the notion of "crowdsourcing" creditials can mean something quite deliberate.  Linked In has tried to do this and it apparently has not worked very well.  I understand that there are some professional networks where Linked-In endorsements are value.  But they are so meaninless in my own professoinal community that I might question someone's qualifications for simply asking me for an endorsement or presenting those endorsements to me.

But Stackoverflow has done a great job crowdsourcing credentials.  They did it very differently.  My first question is what does this mean for badges?

This exchange made me realize that have been thinking all along about a very different type of crowdsourced credibility as it relates to digital badges.  I have been thinking about it more tacitly. Consider that a digital professional network might unwittingly and tacitly  undermine (or boost) the validity of credential issued by the official body representing that profession.   Like for example, imagine that there is a listserv for Human Resources in Accountancy people who recruit and screen accountants for a living. If that commuity collectively concludes that a credential from some accrediting body is less valid than the accrediting body claims it is.  The  value of the credential is diminished, regardless of what they claim.

Badges may make this happen at warp speed.  In other words, because badges are so web-enabled, I presume that they will be more more connected to web-based professoinal network.  Even if it is not at warp speed it may happen too quickly for a credentialing body or their representative to respond.  

dthickey

For example, Microsoft certification

I just had another conversation about this with somebody who provided a good example.  Mark Bell is a Telecom Student at Indiana University and is finishing hs PhD thesis on deception in online context.  It was a quick convo and I am going to add in a bit but here is the issue.

Mark pointed out that the Microsoft Certification several years ago was a credible and valued credential.  Then three related things happend.  The first is that everybody started getting them.  The second thing is that advice about how to prep for the test or openly cheat began circulating.  The third thing is that employers became aware of the first two things. And the credential lost it value.  I don't know the details but I suspect this happened over an extended period of time. 

If people are earning web-enabled credential and pushing them out to facebook and twitter, this turn of events is likely to happen much more quickly.

This brings me to a really important counter example: Cisco.  John Berhens and colleagues created one of the best formative assessment/summative credentialing system that I think has ever been created for a professoinal credential.  They did not leave it up to others to figure out how to prepare for the test.  Instead the created a first-rate curriculum guidelines and web-enabled formative assessments and gave it away for free to voc ed and community colleges everywhere.  By using sophisticated evidence centered design methods, there were able to create summative assessments that could not be readily compromsed by people stealing or memorizing items and then sharing them.  They dramatically expanded the pool of Cisco-certified technicians.  But to my knowledge that certification is still widely value by employers.  

I think there are some lessons for us to be learned from these examples.  Can anybody fill in any more of the details for us?  Do we have any examples of this sort of stuff where digital badges are involved?

slgrant

Reputation and Credentials

"But Stackoverflow has done a great job crowdsourcing credentials.  They did it very differently.  My first question is what does this mean for badges?"

Stack Overflow is technically a reputation system -- not a credentialing system. It's a fine distinction, but a critical one at the design level, and likely impacts whether crowdsourcing will occur at the level of success Stack Overflow experiences.

You can have a badge system that isn't built on a reputation system, and a reputation system that doesn't use badges. You can also have a reputation system that uses badges, like Stack Overflow, but the badges do not function as credentials in the pure sense of the word.  When badges become credentials, there is an assumption that a claim is being made (as opposed to an opinion given), which triggers issues of validity. The purpose of the badges in Stack Overflow is to encourage specific types of engagement, and to also communicate what kinds of behaviors are valued in that community. 

Stack Overflow has no curriculum, and there are no "learning pathways" inside the system, at least not in the more prescribed ways we talk about them here. That's important when comparing the crowdsourcing that occurs in Stack Overflow to the kinds of badge systems we're talking about in the Badges for Lifelong Learning initiative. Stating an opinion (reputation) is a less taxing form of participation than verifying a claim (credential), and that distinction is likely to have a rather dramatic effect on crowdsourcing credentials. 

thiemehennis

Reputation & the StackOverflow model

I have written a book chapter in which I compare different kinds of reputation systems and present a reputation model specifically for peer-based learning environments. It criticizes the very direct way of 'endorsing' people as in LinkedIN, and proposes a model that is able to interpret and use different kinds of evaluations (ratings, endorsements, likes, citations, etc.) from multiple sources (people, papers, communities). You can check the PDF here: https://www.researchgate.net/publication/233842141_Reputation_in_Peer-Based_Learning_Environments

What I find interesting about SO is that they also have a strong connection with the market: their careers.stackoverflow.com website offers companies to look for talented programmers. This 'market demand' may - in learning environments - be combined with the question and used as an input for a curriculum. 

dthickey

Outstanding Paper!

Thiemme--

I finally got a chance to read your paper--or actually papers as there are really two entire papers in there.  I thought the review of the difference systems was exemplary in the way it connected to the relevant research literature.  A lot of really  throughtful work went into this.  And the model and conclusion lay out an impressive like of inquiry, a nice career trajectory I think.  For me the opening paragraph of the Future Research Directions on page 118 was a nice summary of this

Centralized approaches in knowledge management usually fail (Benbya & Van Alstyne, 2010) and centralized approaches in education are unable to cope with the increasing demand for high-quality education worldwide (Schmidt, Geith, Håklev, & Thierstein, 2009). Self-organization of knowledge management and learning is a promising approach to tackle problems in centralized environments, but is only possible with supporting technologies.
 
This is a great example of what I was hoping would happen around these posts and hope can keep happening over the next year with our project.  This is a directly relevant paper to anyone who gets serious about peer awarded badges.  It has great suggesions for innovators who wants to build a system systematically, and is a good model for a innovator/researcher who wants to systematically contribute to the knowledge base in this area as well.  I hope anybody who is interested in peer awarded badges will consider how useful analysis like yours of a scalable system would be.  I hope you will pursue some collaborations in this regard, and invite you to consider looking at the two-part peer recognition system we are using in my for-credit and open online courses ("endorsent" stamps for completeness without judging quality, and "exemplary" for particularly helpful contribution).
 
I really like the set up around Siemens and Downes' course, through I confess I do have some opinions about the framing  of the learning theories.  I personally think it is quite helpful to distinquish more sharply between constructivist, socio-constructivst, situative theories of learning.  IMO. conflating them makes it difficult to see and exploit the value of knowledge that is primarily social and only secondarily indvidual.  It does not seem to be a problem in the review, but I worry that it will be a problem in carrying out coherent research.  But I might be mistaken.  In particular I wonder if the system you describe in the second part of the paper results in repution indices that are an aggregate of indivdual scores, or whether they take into account the socially-constructed nature of reputatons.  
molneck

Comments and Questions

Dan claims that “Web-enabled digital badges are quickly transforming the way that learning is recognized in schools and in informal learning contexts.” I wonder how valid this claim is. Is there an ongoing effort to monitor the incidence of badge use in schools, among employers, or in other sites? Or, does the claim of rapid transformation refer to the progress you are seeing among the MacArthur award winners to successfully design and implement their projects? If the claim is true with respect to schools, it would represent an amazing exception in the history of education, which was once described by the historians David Tyack and Larry Cuban as “Tinkering Toward Utopia.”
 
Dan notes that “most projects started with a pretty clear idea of the learning they wanted to recognize with digital badges.,” and that the projects then engage with how to assess that learning. One set of  questions I have is how the very use of badges might react back on what learning is sought, how learning might be redefined, how definitions of “teaching” and “learning” might be altered by use of badges. For example, does the use of badges inherently represent a commitment to “competency-based” learning? 
 
More generally, I would be interested in, and think that those within the badge movement would also find it relevant, how use of badges interacts with what some have called the “grammar of schooling” or the model of “real school.” That is, how and where do badges “fit” into the institutionalized set of organized practices we know as “school.” Another way of asking the same thing is to ask what would have to change for badges to be incorporated as an integral, not merely “alternative,” aspect of schooling? For example, as Bill Watson at Purdue has noted, he still has to give grades in the course in which he awards badges. I anticipate that Katie Davis’ work in Providence will provide some answers to these questions. It will be especially interesting (to me, at least) to learn how teachers view badges. Do they judge them by effectiveness criteria? Appropriateness criteria, e.g., they are too much like games and play, school is serious?  [See David Tyack and William Tobin, “The ‘Grammar’ of Schooling: Why Has it Been so Hard to Change?” American Educational Research Journal 31 (1994): 453-479, and Mary Haywood Metz, “Real School: A Universal Drama Amid Disparate Experience.” Politics of Education Association Yearbook, 1989. Pp. 75-91.]
 
Dan reports that the Global Kids project reported a preponderance of the work done for badges met the “requirements” of the program. I am wondering whether these findings, or the experience of using the badges, had, or might, in the future, have any effect on what the "requirements" are, or what the institutionalized boundaries between "outside" and "inside" the official program are.
 
I frequently see reference to “the badges community.” I am curious if anyone has a handle on, or could establish, the parameters that describe the network that constitutes the (no doubt, ever changing) “badge community.” I’d be particularly curious as to how this / these networks overlap with other, related networks. For example, I noticed that Cathy Davidson is included in a depiction of the network of “top players” in the MOOC movement (http://chronicle.com/article/Major- Players-in-the-MOOC/138817/). One clearly related community is the “learning analytics” movement, something of which, from my insulated vantage point as a sociologist, I had never heard until reading about Dan Hickey’s work. Perhaps, there are other terms that refer to the same activities and communities, e.g., “instructional design” (???),  a term to which I was only recently introduced. Interestingly, I had not encountered either of these terms in the school reform policy literature I have read, e.g., that dealing with No Child Left Behind. (At least I knew about Rich Shavelson’s work through reading about accountability in higher education. And, having been passingly familiar with evaluation and curriculum research over many years, I knew of Bob Linn, Lorrie Shephard, Lauren Resnick, and Lee Shulman - but I wonder if these folks are folks whose work informs the “learning analytics” movement. [Dan refers to Shavleson as a critic of DBR.])
 
Can someone more fully explain to me the difference between a “reputation system” and a “credentialing” system that Sheryl Grant has distinguished? Credentials have power because they have reputations of validly indexing real qualities and capabilities. They are awarded by education institutions, which are “chartered” to produce “graduates” who are presumed to have particular qualities and capabilities, and are treated as if they actually have these. If crowdsourced reputational testimonials come to be trusted, and used to make selections, then they will have become “credentials.” [See John W. Meyer, “The Charter: Conditions of Diffuse Socialization in Schools,” In W.R. Scott, Social Processes and Social Structures: An Introduction to SociologyHRW, 1970.]
 
Dan notes that “because badges are so web-enabled, I presume that they will be more connected to web-based professional  network.” I would think that following the NMI badge project will yield evidence about this.
 
I am interested in the comparison between the differential fate of Microsoft credentials and Cisco credentials. If Cisco credentials have become widely distributed, but not fallen in value, this suggests that demand has kept up, and that the dilution of the value of the Microsoft credentials comes from the cheating associated with them, not from their large number in circulation. Or, do they operate in different markets? More generally, I am curious whether the inflationary dynamics that occur with college degrees will occur as well with badges. Education credentials are of value, at least in part, because they produce distinctions among people. Their value depends, again, at least in part (sociologists would say more; economists would say less) on their relative scarcity. The “open” badges movement not only celebrates a proliferation of badge issuers, but a democratically wide base of badge acquirers. Especially in a low-demand economy, this will predictably lower the value of each additional badge issued. Following the NMI badges, in particular, will offer evidence about this. Possibly badges will hold their value for niches in the economy where there is high demand.
 
 
 
dthickey

On transformation

Funny how so much of this discussion is about credentialing, but it did not happen on the post that introduces the principles for recognizing learning that we uncovered   But a lovely discussion it is and I am happy for all of this interaction.  

As for Micheal's first question about transformation, great point.  I am definitely talking about the transformation around the badging systems.  In some of the projects there has been very little transformation caused by the introduction of badges, but in others it seems pretty profound, as groups of stakeholders come together to decide what should be recognized with badges and how that learning should be assessed. But I have worked hard in my career over the years to get help organizations be more systematic about recognizing and assessing learning.  So it might look more transformative to to me than it does to somebody else (like yourself perhaps) who might see such changes absent of major policy changes as trivial.  I have always focused on practice and only recently began to appreciate the importance of policy.

Regarding Tyack and Cuban--do you think that they would characterize the changes in higher ed with MOOCs as "tinkering"?  How about the explosion of online education across contexts?  I think that much of the change in learning is actualy trivial and some is quite negative, but for some learners in some sectors the landscape has changed quite a bit.  But again, I am concerned fundamentally about how learners interact with each other and with disciplinary resources.  That is the primary type of learning for me, while the cognitive and behavioral residue that gets left behind might be less different.

dthickey

Cuban on MOOCs

I should have looked before I asked about Larry Cuban.  Yes it does in fact appear that Cuban considers the changes in higher ed as more tinkering.  Here is one of his more recent posts on the topic.  I think I still think that because he is not looking closely at learning (and particuarly at interctive learning) that he is looking past some of the changes.  It turns out he has written quite a bit about this topic and there are some lovely discussions on his posts.

I liked one of Steven Downes' responses to the Cuban's general argument here.  My view of what matters in learning resonates a lot with Downes' notions of connectivism, so I am inclinded to agree with him.

molneck

Extent of "Transformation" Associated With Badges

There is a distinction I think we need to keep in mind when talking about the extent of "transformation" badges have effected. That is how transformative a new model is compared with an old model (say, of teaching, learning, curriculum) and how widespread and faithfully implemented a reform is. When I said that badges would be unique in the annals of the history of school reform if they were as transformative as Dan seemed to be claiming, I had in mind the second. (See David K. Cohen, "Teaching Practice: Plus Que Ca Change..." http://www-personal.umich.edu/~dkcohen/downloads/CohenTeachingPractice.pdf.) I would venture a hypothesis here. The extent of "transformation" in the second sense has historically, and will be in the future, inversely related to the extent of "transformation" in the first sense.
dthickey

Faithfully Implementing Routines vs. Spreading Practices

Michael--

I don't think I claimed they are highly transformative but rather that that might be.  However, you have articulated some tacit assumption about transfromation that IMO will obscure and obstruct the kind of tranformation that I am thinking about.  It seems to me that all of the failed reforms that you and others refer to have failed because reformers have aimed for, in your words, reforms that are "widespread and faithfully implemented."  Because of the highly social and contextual nature of schools, this appraoch is bound to fail.  In my writing on this topic I call this appraoch "distributed instructional routines"

The kind of transormation I am thinking of has much more in common with what Henry Jenkins has characterized as "spreadable media" in his latest book.  Back in 2009 when Henry posted the reports that turned in to that book, I did a couple of posts at his blog that used those ideas to propose an alternative notion that might be called "spreadable educational practices"  The post is here but below is the table that juxtaposes what I think you mean by "faithful implementation" and the enactment of spreadable practices.  

My point is that we foster foster and study transformation using the notions in the left colume, we will get little transformation and find even less.  I believe the using the notions on the right will result in more transformation and allow us to see transfromation that occurs organically.

http://www.newmedialiteracies.org/wp-content/uploads/2009/03/Picture%207-thumb-500x658-882.png

 

molneck

Comment on Faithfully Implementing Routines vs. Spreading Practi

Dan,

 

Would that the "failed" reforms I had in mind- i.e. reforms that failed to take root (rather than failed to accomplish their stated goals) - were of the Disseminated Instruction Routines model. Sadly, the reforms that made less lasting impression on how we do school are those that are based (without the contemporary vocabulary) on principles consistent with Spreadable Educational Practices.

See e.g., Stories of the Eight-Year Study: Reexamining Secondary Education in America [Paperback], Craig Kridel (Author), Robert V. Bullough (Contributor) SUNY 2007; and Reform and Resistance in Schools and Classrooms: An Ethnographic View of the Coalition of Essential Schools [Hardcover], Donna E. Muncey (Author), Mr. Patrick J. McQuillan (Author) Yale, 1996.

I think it is important to ask what features of school organization, and of the institutionalized models underlying school organization, would have to change in order for reforms exemplifying SEP to take root. 
 
This may well be where digital media, online communities of practice, etc. with which you are familiar, and are working, have the potential to change the story from what is has been historically. 
 
 
dthickey

On spreadable practices

Okay, we are really going to have to write someting together because you know a lot more about school reform than I do.  I am respoinding specifically to the failure of things like fetured things in "What Works Clearinghouse" to impact practice.  The kinds of disseminated instruction routines like drill and practice test preparation that fared well at WWC (albeit under different names and formulations) were already widely used.  Those sorts of things can be "faithfully implemented" because it just involves parking kinds in front of computers during their free periods; it helps when drop the kids who don't conform from the experiemental sample and then explain that way under implemention fidelity).  

But we have never seen any widespread efforts to refine and spread innovative practices in the way that I am thinking about it.

Your earlier comments led me to dust off some earlier arguments and put up a new blog post on The Transcendent Potential of Digital Badges.  But now I have to stop blogging and get busy verifying participation so I can get grades for the summer term in by tomorrow's deadline!

slgrant

Innovation and Transformation

Your comment was on my mind while reading Nine Questions for Evaluating Education Innovation, a report from Nesta and New Schools Venture Fund that Tony Wan wrote about in EdSurge. Tony writes, "The work is based on a recent book by Michael Fullan, Stratosphere: Integrating Technology, Pedagogy, and Change Knowledge, which emphasizes that any sustainable efforts to impact education at scale must also focus on pedagogy and system change, in addition to technology."

I'm curious to what extent designers have taken into account the constraints and affordances of existing systems when deploying their innovations, and how this impacts transformation or innovation at scale. Not a whole lot, I'm guessing. I'm just now looking into literature about technology adoption, not just in schools but in a variety of institutions and organizations. 

One of the smart things about the Badges for Lifelong Learning Competition is that developers and learning content providers were matched -- collaboration was built into the design process. You can see how that influenced the lessons learned in the 30 project Q&As we just posted, and it gets to Dan's point about the intended purposes and actual functions of digital badges. It would have been a lot easier to fund 30 technology teams to build badge platforms, but instead, 30 collaborations were funded. I believe that gets us a hair closer to transformative badge systems simply because they are designed by, for, and within the systems they are seeking to impact. That's my cup half full perspective, right there. 

A couple of other thoughts about transformation that fall under unintended consequences: Many of us working on the Badges for Lifelong Learning work find the scrutiny of existing systems to be one of the upshots of the badges movement. Cathy made the comment, "I'd love some more commentary on how existing systems could conceivably be changed to measure the range of other skills, competencies, interests, and formal and informal educational accomplishments that a learner amasses and that are often more important than actual GPAs or GREs for lifelong occupational success and advancement." 

Me too. Let's use the badges work to get that commentary going. 

I'd also like to know more about the interplay of policy and reform. Current policy conditions make the badges conversation particularly fascinating (if you haven't already, check out the Policy, Political Trends, and Challenges webinar we did a few weeks ago). It speaks to the "What are badges for?" question that I don't think we talk about enough. 

molneck

"Real" MOOCs versus "Actually Existing" MOOCs

Stephen Downes asks "As for missing the whole point, the question is, what were the Stanford and MIT people trying to imitate. They were not trying to imitate our MOOCs ..." What MOOCs "are" is a matter of definition: Are MOOCs what Stephen Downes envisions them to be, and works toward them becoming, or are MOOCs what Udacity, Coursera, and EdX are offering? I do think that the popularization of badges by folks outside the badge movement in venues like The Chronicle of Higher Education, and Forbes, very much overlook the effort of some to use badges to redefine and reform learning and knowing. But, if badges "take off," I suspect that "purpose" will take a backseat to other "functions" to which adopters will put them. To avoid this will require carefully designed badge projects which integrate pedagogical and curricula reform with credentialing functions. The U.C. Davis Sustaining Agriculture and Food Systems major may exemplify this kind of "double duty."
dthickey

I agree

Michael--

You are absolutely right.  In his 2012 State of the University Address, Indiana University president Micheal McRobbie (under IU Online halfway down) quoted a 2012 letter from USC President Max Nikias to the USC faculty on this topic that should resonate with all of us.  The upshot for me is that there an edu.com "bubble" right now.  MOOCS and badges are a big part of that bubble.  And that bubble is going to burst.  Just as companies in the dot-com bubble that were not generating true revenue evaporated, educational companies and organizatoins that are not generating "true academic value" will evaporate as well.  

Of course, this begs that question of what is "true" and what is "academic."   From my perspective on this, the equally important point is that educational organizations that dismiss online open courses (MOOCS and beyond) and web-enable information-rich recognition systems (badges and beyond) might end up like the pubishers and retailers who dismissed the internet in the 1990s.  They won't disappear overnight, but they will end up fighting for their suvival over the medium term and may not exist in the long term.

Peter Fairweather

One of the Hidden Dangers Facing Colleges. . .

To follow up on Dan's point: "educational organizations that dismiss online open courses (MOOCS and beyond) and web-enable information-rich recognition systems (badges and beyond) might end up like the pubishers and retailers who dismissed the internet in the 1990s.  They won't disappear overnight, but they will end up fighting for their suvival over the medium term and may not exist in the long term."

One of the real dangers facing 2nd tier institutions is that they may respond to their revulsion at gamification by focusing on the "traditional residential campus experience" (i.e., largely clinging to the traditional one-way classroom lecture thinking that it is the time-honored way to generate "true academic value.")  Such an approach keeps faculty and adminstrators in their comfort zone (and in so far as it rejects the gamification of learning is easily justified).  But it prevents the institution from enriching the on-campus learning experience--either through technology or by retooling--e.g., flipping--the classroom.  They would seem to be fighting the good fight but, as Dan says, are likely to be doomed in the long tem.

Very interesting discussion.

slgrant

In thinking about your

In thinking about your question: "Can someone more fully explain to me the difference between a “reputation system” and a “credentialing” system that Sheryl Grant has distinguished?" My response started to look more like a blog post than a comment, and I didn't want to hijack Dan's post, so I moved my thoughts here: What reputation systems can teach us about badge system design

 

bpenuel

Badges and Validity Arguments

 

For me, this post inspires a broader question, “In what sense are badging systems new kinds of assessment systems, if at all?”

I add “if at all,” because (to lay out my own bias here), I see badging systems principally as a means of building young people’s social capital by tying together institutions that recognize them for different kinds of accomplishments. I agree with Carla Casilli’s view as represented here that “evidence from formal reliability and validity studies might be meaningless if relevant personal or professional networks collectively ignore or dismiss that evidence.” What matters most for badges, I think, is that recognition for accomplishments in one setting can be granted in another, in a way that broadens access to learning opportunities, especially for youth and adults from nondominant communities. Accomplishing this goal may or may not require badges to become assessments in the same way that standardized tests function as assessments. 

If we do want to think of badges as forms of assessment, it doesn’t require us to give up well-planned studies of the sort reviewed here, nor giving up the project of validation, even ones that are perhaps premature. Rather, Caarla's quote reminds us that validity and reliability are not properties of instruments. Instead, what we need is to assemble evidence about particular kinds of claims we want to make about learners’ competencies into a coherent argument. Ultimately, too, that argument is about how we want to use the assessment data.

One way to sharpen our arguments about badges is to make the claims about what kinds of futures we want learners to be able to access. For example, we could say, “A learner who designs a web site using Word Press should be able to get an internship in a web design company.” When we make such claims clear, it becomes immediately apparent that we need to qualify those claims, “…if they have other relevant skills required of the web design company.” Or “…if the learner wasn’t simply given a template and content to follow, but had to exercise some creativity.”

An external organization hiring a learner can’t look under the hood of the project easily to know whether all those conditions are met. A badge alone is not likely to do the job. A fleshed out argument might do better in convincing a prospective employer, but that’s likely to include a larger set of artifacts and interactions—a resume, an interview, etc.

Perhaps I am asking a lot of badges right now, but the promise and attention are high, and I think some claims are being made about the transparency of badges that just don’t ring true to me. There’s no getting around the fact that some badges will be compelling as sources of evidence to some people but not others, and that these judgments will be consequential. It’s true that the same scrutiny is not likely to be given to grades and standardized tests, in terms of their validity for use in sorting and selecting students. But if we want to create an alternative, it would be useful to ask just how these different things became “natural” forms of assessment over time (The Big Test does a nice job of this). It becomes immediately apparent is that badges do not contain the evidence claimed for them; instead, people promoting badges will have to convince others that they do.

dthickey

RE: Badges and Validity Arguments

Bills comments were just the kind of concerns that need to be raised here.  He said privately that it might not have the kind of input I was hoping for.  but it is exactly the kind of conversation we need to be having if this innovation is going to evolve into something meaningful and transformative.

While I am indeed excited about badges, it is mostly because of how they might trigger larger innovations. Mostly I ahve been concerned about innovation around assessment, but Sheryl and others are helping me see all the potential changes in recogntion systems as well.

I am going to respond to one of Bill's points about getting under the hood of projects to make my point.  I have been working for a while to transform the way we assess student generated artifacts and projects (this post  is now four years old but I think it still holds).  Bill says that "a badge along is unlikely to do the job" and suggest that a larger set of interactions and artifacts might be needed to convince someone.  But what if that set of larger artifacts and interactions are indeed embedded in the badge itself?  In other words the whole point about digital badges within the OBI is that there is no such thing as "a badge alone"  

I am pretty excited to find ways of putting some of the other evidence that I used to validate artifacts as evidence of learning direclty into the badges.  this primarily now "reflections" but they could also be traces of engagement of the author of the artifact with other students and teacher about the knowledge represented in the artifact.  Additionally if one has stacked badges where the project badge is accompanied by another badge that indicates success in a more formal on demand context (in my case, a conventional exam) that could be quite convincing.

I got some nice initial evidence of my initial model in my onlne class on educational assessment this summer.  I hope to learn more this fall when I scale it up as an open class for up to 500 studnets thanks to Google.  Here is the current description of badges in that course.  it has changed a bit this week but this is mostly it is current.  So far so good, but integrating the OBI and google is not as easy as I was hoping.

 

 

jimdiamond

transparency and translation

Hi Dan and all,

Thanks for the great posts over the past week or so. There's so much to discuss here, but I'm especially interested in Dan's comment that, "It seems to me that LA where badges are involved will be entirely different because the data contained in the badges willl need little or no massaging or interpretation" and Bill's comment about the transparency of badges not ringing true.

Both are germane to the badge project with which I'm affiliated right now: the American Social History Project's history teacher professional development badge project. For the past few weeks, the project team has been determining how best to represent the evidence of professional development and achievement through the evidence and criteria URLs that Mozilla's backpack requires. In the process, it's become clear that a great deal of interpretation will need to occur among a badge's various consumers (in this case, school and district administrators and professional peers) to determine just what it represents.

Rightly or wrongly (likely the latter), what many traditional credentialing systems have going for them are something along the lines of face and external validities (both very broadly defined here). In any given community of credential consumers (right down to an n of 1), a credential will live or die based on general perceptions and/or the views of those with power of what the credential "means" and how useful it is as a predictor of some future outcome. A driver's license suggests some proven ability to drive; a college degree suggests some proven ability to write papers and pass exams; and a law degree suggests some ability to practive law. What all three of these have in common is "tradition," as well as accreditation bodies: some authority that determines the definition of the credential and who's allowed to grant them. However broadly, it suggests a level of standardization that does not yet exist for badges. And that could be a very good thing, though that's likely the subject of a different conversation.

To return to the badge project I'm working on: it will not be enough simply to click on a badge and see a.) a percentage score on a multiple-choice quiz for content knowledge; b.) a rubric score for the alignment of teacher lesson plans to Common Core literacy standards; and c.) a rubric score that judges the effectiveness of teacher annotation to student work as a means of providing feedback on how well a student is practicing certain skills. Any administrator interested in determining exactly what s/he might be about to grant continuing education credits for will have to go deep to understand what those scores mean. Further, they'll need to understand whether the "achievement as defined" by the grantor matches the "achievement as desired" by the consumer. This has always been true of professional development credentialing. It's just that it's not yet apparent whether and how (or whether they even should try) badges are going to overcome this challenge

To answer Bill's question, I don't think badges are some sort of new assessment system. Rather—at least in the K–12 context in which I work—they seem to be an extension of "portfolios." Badges come with the same promise and drawbacks of educational portfolios: they allow for an in-depth look at the nuances of longitudinal performance, but they require a level of focus and translation that usually isn't associated (though it ought to be) with standardized scores.

It seems to me that one extremely valuable contribution that the current badging initiative (I've commented elsewhere about the problem with lumping all of these things together—they're cleary not all equal, just like any other credential) can make is not necessarily in the realm of "creating new credential systems." In some ways that's done, though the focus on using badges to credential those with achievements that often go unrecognized is extremely important. Rather, it seems to be about the translation: given the desire to recognize achievements and passions in non-traditional contexts, how can this initiative work toward facilitating the transportability of badges? My guess is that at some point badge accrediting bodies will start forming, though that runs the risk of perpetuating the same biases built into many of the current systems. To get there (or to obviate it), how can the various stakeholders in this community begin to work together such that badges can translate from one system to another? How (and why?) will stakeholders begin to see various types of validities among one another's credentials? And if some sort of "system(s)" manifests among the various stakeholders, can badges/portfolios become successful alternatives to standardized scores? Or at least sit alongside them. I hope the answer is yes, but we need a bunch of research to get there.

dthickey

Re: Transparency and Translation

Jimmy--

Great subject heading for a great post.  I am excited about the possibility of transformation (of ecosystems) and transcendance (of existing contraints on recogntion of learning).  This issue of translation is indeed a big one and I am glad that you and your colleagues are working on it in such a widely relevant context.  I assume that your teacher continuing education context is similar to the one in Indiana.  Indiana University used to be responisible for accrediting organizations so that their workshops and in-service programs could count towards teachers' continuing education requirements.  

But that all got turned over to the school systems some years ago and I understand that it has become quite a mixed bag.  Indiana has a detailed Professional Growth Plan that calls for 90 points/hours of continuing education to renew their licences. For me, the most interesting thing is the Growth Plan  Experience Documentation guidelines that say what sorts of documentation is needed.  I wonder how your American History Project would fare.  There is an option for curriculum development. but the criteria is "must be serving on a formal committee organized by a local school, district, state, national, or international agency.".  Would you qualify for that?  Can it qualify as in-service credit?  In our case that calls for a "credit approval slip" or a "course attendance slip" and that it "may be verified by the school district."  It seems to me that if your work hard to label medadata fields and put meaningful data in those field that we could make a lot of progress here.  Whatever you find I looking forward to helping you help others learn from your efforts.

I am facing a somewhat differet challenge with my open online course on educational assessment this fall.  The documentation experience guidelines says that one credit hour of college coursework - 15 points.  So one three-credit course counts as half of the 90 total points needed.  My open online course will cover most of the content in the for-credit version that I normally teach.  The only difference is that I made about a third of each wikifolio optional.  If students complete the optional parts of each assignment and engage in discussions of their wikifolios with the other members of their networking group and get a passing score on the three quizzes and final exam, they will have learned as much as the students who took the course for credit.  

The problem is that my open course is not "earned from an accredited college or university".  My university has been very supportive but the attorneys and adminstrators want it to be very clear that my course is not accredited.  They asked me to change the "enrollment page" to the "registration page" and are happy that I refer to "participants" rather than "students."  I am getting in touch with adminstrators to sort out just what we can promise participants, but it sound like all we can promise is that we will give them a "web enabled certificate" since they don't understand badges yet.  I might go so far as to promise a specific certificate where we use extra project resouces to audit their participation, and perhaps try to get the course approve in advance by some of the larger districts.   I am honestly not sure, but it is going to be fund finding out. The upshot is that your your point is well taken: regardless of the amount of evidence in the badge, we are gonig to have to do a lot of translation before it will count.

Here is another aspect that I wish I knew more about.   One worrisome thing I hear  is that adminstrators use the credits to get work done, and that it often does not involve professional development at all.  I have not seen any evidence but I hear rumors that some systems implicitly tie them to things like recess duty (can anyone verify or elaborate on this?)  The bigger point here is that adminstrators presumably want to be in control here, and that most are going to be relucant to award  continuing education credits for a badge from an online program.  So we have to work extra hard if this is true.

soletelee

Badges and Validity Arguments

Hi Bill, 

Can you expound on this statement please? " It becomes immediately apparent is that badges do not contain the evidence claimed for them; instead, people promoting badges will have to convince others that they do."

We have taken evidence into consideration with open badges. One of the metadata fields for the open badges digital badge standard is in fact an evidence URL. 

You also mention, "An external organization hiring a learner can’t look under the hood of the project easily to know whether all those conditions are met. A badge alone is not likely to do the job. "

This is precisely the problem we're trying to help mitigate by packing a bunch of information into a portable digital badge unit -- evidence, criteria, issuer information, learner information, etc are embedded within a badge such that it allows stakeholders to "look under the hood" of a competency or skills claim made. 

"A fleshed out argument might do better in convincing a prospective employer, but that’s likely to include a larger set of artifacts and interactions—a resume, an interview, etc."

Curious to know what your thougths are on self-reported nature of resumes that aren't necessarily verified or evidence-based. 

 

soletelee

Badges and Validity Arguments

Hi Bill, 

Can you expound on this statement please? " It becomes immediately apparent is that badges do not contain the evidence claimed for them; instead, people promoting badges will have to convince others that they do."

We have taken evidence into consideration with open badges. One of the metadata fields for the open badges digital badge standard is in fact an evidence URL. 

You also mention, "An external organization hiring a learner can’t look under the hood of the project easily to know whether all those conditions are met. A badge alone is not likely to do the job. "

This is precisely the problem we're trying to help mitigate by packing a bunch of information into a portable digital badge unit -- evidence, criteria, issuer information, learner information, etc are embedded within a badge such that it allows stakeholders to "look under the hood" of a competency or skills claim made. 

"A fleshed out argument might do better in convincing a prospective employer, but that’s likely to include a larger set of artifacts and interactions—a resume, an interview, etc."

Curious to know what your thougths are on self-reported nature of resumes that aren't necessarily verified or evidence-based. 

 

bpenuel

Evidence and Argument

Happy to elaborate, Sunny, from an assessment perspective.

I think what I see as conflated in "badge-talk" is the distinction between evidence and a warrant for that evidence. When I look for in a warrant is some statement that allows me to link claim and evidence of learning. Meta-data just doesn't do the job.

Jim Diamond's post today I think captures my own view, that for some established assessments, we allow this sort of "stand-in" of the certificate for a warrant, when it comes to grades and test scores. But asserting the same for badges doesn't work because they are not widely institutionalized. The power of an assessment needs all of the things you say, but also some broader argument for why the badge is consequential for future opportunity. Especially the power of a *new* form of credentialing or assessment.

If we look at the history of testing -- I mentioned The Big Test as good account -- we'll see that there was little of this conflation of evidence and warrant. James Conant and his collaborators at Harvard knew that they had a world to convince about the benefits of testing. The argument, too, was not about the properties of the test, issuer alone, but much more than that. It was about consequences and use. Contrary to how it's been taken up in schools, the point was to broaden access to education, making admission into Harvard and other elite institutions less based on patronage and legacy, and more on merit. 

For the record, I don't think badges should go the route of being alternative forms of assessment, but I think they will, and if they do, then the same criteria and critiques will need to apply to them as we apply to standardized tests of various kinds.

Cathy Davidson

GREAT conversation

Hi everyone,  You have no idea how happy this conversation makes me.   It is exactly this kind of serious convergent conversation, from so many disciplines and concerns and investments, that we need, powerfully, to make alternative, formative, motivating assessment systems to the dominant systems possible.   This is some of the most thoughtful, engaged thinking I've seen and the dialogue form really helps to elaborate questions, queries, and possibilities.   

Interestingly, when I started research in this area in around 1998, when I was Vice Provost for Interdisciplinary Studies at Duke, my motivation was almost entirely from the learning side, open source and peer-to-peer.   We began HASTAC in 2002 out of a convicton that higher education, in particular, needed to change and be more open to peer-to-peer learning models and an ancillary conviction that K-12 could not change until higher education did.   Assessment seemed key since (sorry to be such a broken record) you cannot have a system that counts what you say you do not value.   The counting--all the assessment standards and credentialing metrics--end up driving the values and that, to my mind, is the death to learning more broadly construed and education more formally construed.  Bill, for me, that is how the motivational learning and the alternative forms of assessment talk go together.  

 

Now that I spend a lot of my time with the business community, I see, though, about the same standardized, inflexible, summative, and un-motivating and often highly rigid and antiquated systems of school assessment and credentialing can plague HR systems.   Many companies are seeking a kind of well-prepared, smart, energetic, engaged employee (especially in technical areas but also in design, graphics, creative and artistic areas) who may have dropped out of standardized educational systems (literally or as achievement) long ago but who may have much to offer.   Badging works to help identify those people, to coalesce interests and skills and talents and ad hoc employment experiences into something that could be a career, that could work for job placement.   And then, for existing employees, badging adds a component that most HR rigidities miss, including the collaborative workforce component that comes from peer-recognition.

 

What fascinates me is how these different areas (a) go together and (b) how much they resemble the same processes that happened in the 1875-1925 period as society, education, and industry were all going through enormous changes and many things were going through parallel and yet convergent changes at once and ( c ) how the standardized and institutional credentialing models of that late Industrial Age (statistics, deviation from the mean, IQ, disability scoring, multiple choice testing, all evolving together)  parallel the "big data" possibilities and critical thinking of "big data" of 2013.  

 

It feels like an epoch moment, another paradigm shift time, when many things come together at once.   I don't think badges are 'the answer' to all these issues but, like the Kansas Silent Reading Test or multiple choice test that Kelly developed in 1914, or the ABCD grading system that Mt Holyoke adopts in 1890s,  badges may be the quantifying symbol of a lot of epic changes.  What seems so simple on the surface has extreme depths of implications that extend throughout school, work, and society.  

 

The argument unfolding above about "validity" as defined by being "widely institutionalized" is a bit tautological and needs a bit of pulling apart.   There has to some intrinsic standard of verifiability in the metadata or a new system won't be institutionalized.   The fact that such things as SAT scores are "widely institutionalized" makes them valid relative to themselves--i.e.  you can make comparisons of test scores both among individuals and, aggregated, among institutions.   However, that institutional acceptability of a measure does nothing to:

  • really test whether the test itself is intrinsically valuable,
  • whether it actually addresses the skillset it is supposed to test
  • whether it actually addresses the actual skills (important in rapidly changing environments) needed by the institution that is accepting students, for example, based on their scores
  • whether the scores themselves "match" the ability level in all relevant domains of the person's taking the test

In other words, wide institutional acceptance allows for standardization but it does not do much in itself to answer the question of whether that which is standardized is actually that which we want to measure (i.e. we're back to the valuing what we count and counting what we value issue).

 

I think the arguments above were starting to blur these different ideas of validity, intrinsic and extrinsic (although those terms themselves are blurry).  I'd love some more commentary on how existing systems could conceivably be changed to measure the range of other skills, competencies, interests, and formal and informal educational accomplishments that a learner amasses and that are often more important than actual GPAs or GREs for lifelong occupational success and advancement.   

 

To me the key question is:  Is it possible that the chief importance of badges will be to push wholesale reform of existing credentialing systems?   Or is the present system too much rooted in an antiquated view of disciplines, competencies, expertise, authority, credentialing, ability/disability, hierarcy and data to be as useful as badging potentially is for new ways of defining the talents needed in the world we live in now? 

 

I don't know the answer to those questions but I know those are the right questions because our current systems are not working on many levels at once.   How we get to what works better is, I think, the issue we all share.  This conversation and others like us help to clarify that joint commitment and its points of overlap and disagreement.

 

Thanks for such an inspiring morning of reading!   I can't wait to see what comes next.

bpenuel

Intrinsic standard?

Cathy, can you clarify what you mean by "There has to some intrinsic standard of verifiability in the metadata or a new system won't be institutionalized"? I ask, because from an assessment perspective, I'm unfamiliar with thinking in terms of validity as something intrinsic to the measure. In fact, that notion is heavily critiqued in the assessment literature. But I think I am misunderstanding.

I think I am making a slightly different point from the one you critique, and I don't think it's tautological. What I am arguing for is to look at the history of how things become institutionalized as a means toward understanding how badges might become so important and consequential that people in different settings use them readily. 

Cathy Davidson

More precisely

HI Bill, I mean that the metadata has to be complete enough and precise enough to be relied upon or thought to be reliable.   If the actual characteristic/metadata of the badges and the badge system are not carefully constructed, they are not useful to institutions and therefore won't be used for their intended purpose (whether that purpose is as an alternative cv, a new kind of Human Resources evaluation, an internal merit or reward system, and so forth).  

 

Ah, I love your second paragraph. I agree completely.  I thought you were saying something else, about institutional adoption itself being a measure of validity.   I see now that you are interested in a different aspect and I share that interest.   The history of how things become institutionalized is one of my pet scholarly research interests.   In 1926 when the SAT was established that pretty much ensured that multiple-choice testing would become the way of ascertaining knowledge-acquisition in America and is the beginning of the institutionalization of summative, end-of-grade grid-readable (later, machine-readable) testing as the norm for assessment.  

I have really found your comments illuminating, Bill.  Thanks so much for participating in this conversation.

dthickey

On Stack Overflow

How awesome that I am having trouble keeping up with the discussion that has erupted here.   This is a good example of how badges might eventually allow some and force others to transcend prior models of research.  

Great point that Sheryl makes about stack overlflow not having a curriculum.  But the comment makes me wonder about about the argument about "hidden curriculum" made by Giroux and Mary McCaslin's notion of the informal curriculum.  the lack of a formal curriculum at StackOverflow does not mean that there is not an informal one right?  And the formal curriculum in many badge projects does not mean that there there is not an informal one operating as well.  It is not so much the formal intended curriculum that matter as much as the informal enacted curriculum that emerges in an ecosystem.  that is the one that is mostly likely to be shaped by badges and then serve as the avenue for impact.

I might have more to add after reading Sheryl's other post and Thieme's paper.  There is a lot of interesting stuff here 

Tony Cook

On Learning!

I've read a number of posts in the last few days including the string above and my comments may apply across a couple of blog posts.  There is a line of thinking that continues to come to mind in regard to how the discussion has generally been around external values of badges.  Of course, that's what our world has been inclined to think of with education in that one must obtain credentials in order to progress in society or the workplace.  I'm also reminded of a quote I have used from time to time, "What is it that we do when we teach? Is it to impart knowledge to a learner or is it to build capacity of a person to learn and apply knowledge and understanding?"  If we are to improve how learning occurs then maybe there should be some additional thought in regard to what learning means intrinsically to the learner aside from what it means externally.  The workplace needs workers with capacity to function which includes the knowledge, skills and abilities to do work but also to learn to do more or to do things better.  What if we could capture in badges, evidence that the person has shown capacity to function in a role but also to learn, adapt and grow?  Well, that's one thought and it still goes to the value of the badge to those who one would share them with.  If it's reason to give more attention to what is provided in metadata and more specifically the evidence, then it's a good point to make.  How would we capture in evidence the things about a person that cannot be captured in traditional measures?  What forms would that take on in addition to what we might capture in a video story or similar creations.

To get back to the point of the value of badges to the learner themselves, how often do we seek to learn something, be it in a course or training or anything else, simply because that learning would have value to us alone and not necessarily to others.  Or, when we take part in a learning experience to build our personal capacity or to grow ourselves, how does that add to our "portfolio" of learnings and the meaning of those experiences to us as individuals?  Another thought here would be in regard to the value of a "learning," or badge in our discussion, as a "marker" of a learning experience as a means of "recall" that helps the person more readily respond to the need at hand.  So, I'm curious as to how an individual learner engaged throughout their life in what may be, to them, personal or personalize learning would see their "collections" or portfolio of learning experiences.  How might they find such a thing useful in their life?  Of course, and back to how one would share those things externally, much of what is in one's portfolio might be shared which means they need to be of quality and value externally.

We might be mindful of the value and meaningfulness of badges to the earner on a more personal or individual level, or motivations at that level, while we work to assure that badges have a place as a credential, evidence of ability or capacity, certification, or other meaningful uses that may evolve.

One last point here in regard to 4-H, since that's where part of my perspective comes from.  In 4-H and other youth serving organizations, the young person often collects representations of their learing experiences in ribbons, certificates, trophies, etc. and, especially in 4-H, organizes them in a "record book" or what we might see as a portfolio in a digital environment.  Now, these record books can become extensive and I've judged some myself that were inches (several) think.  They have been used as evidence submitted for scholarships and such things.  Sometimes the evidence was very redundant much one could glean through the material and learn something about the young person.  This was, and is, a traditional way of assessing the worthiness of a person for the reward at hand.  What would this process look like in a digital environment?  This is what brings me and some of my colleagues to the digital badges concept.  Not to replace what 4-H values in traditional approaches to evaluating young people but to offer an alternative and something to evolve toward as the world of digital learning environments continues to grow.  And to get back to my other point, these 4-H record books, as thick and extensive as they can be, have considerable intangible value to the young person that put them together.  They represented things they hold dear and of significant meaning that they experienced.  To what extent would a collection of badges be valued in the same or similar way aside from how some of it would be used externally?

I just thoughts I would share a few of these thoughts in response to the other writings and comments.  There is so much to consider in this area.

dthickey

RE: On Learning

John--

Great points.  In 4-H provides us with a great opportuntity to explore what happens when badges are introduced to an existing semi-formal accediation system.  While 4-H is not as formal schools, the value and meaning of the recognition was constructed alongside the organization itself.  I could not find much about it in the official history that I found there might be some in the articles referenced at the 4-H wikipedia entry.  

From my perspective, the 4-H community and the 4-H recognition system co-constructed each other.  This meant that explicit values  (the meaning of the Head, Heart, Hands, and Health) and implicit values (perhaps the values are widely embraced among communities defined by farming and rasing livestock) wove themselves into the  4-H recogntion system over the last 110 years.  So while you refer to value of badges on a "personal and indivdiual level" there value is actually stretched across the larger 4-H community and ultiamtely extends out over the longer history of the organization.  While this was a complicated process, it is much easier to appreciate the much more complex construction of schools, grades, and transcipts over roughly the same amount of time.

This is all buidlup to respond to your specific question: 

To what extent would a collection of badges be valued in the same or similar way aside from how some of it would be used externally?

It seems to me that if badges are merely used to replace or supplement ribbons that they would function in a very similar manner.   We see a lot of this in schools when teachers introduce Moodle or other course management systems; often they are designed to make it easier and save some paper.  But the basic interaction does not change.

What I think is different about badges is that they might make it possible for a recogntion system like the one at 4-H to evolve and advance much more quickly.  For example, elswhere on this discussion Bill and I discussed whether the links in badges might more readily provide evidence of what somebody actually learned in a particular project.  I suspect that 4-H success means a great deal more in rural communities than it does, say, in the non-agricultural business contexts.  I assume  that some of the recognition at 4-H concerns the business of farming or livestock; badges might be a great way to quickly and efficently convey this learning to these other business contexts.  

4-H choose to implement digital badges alongside a brand new curriculum on robotics.  As we leaned from interviewing you and as you articulated in your recent Q&A, this was a lot of work.  You tried to do a lot more than just introduce badges.  You also introduced a whole new curriculum and a much more ambitious assessment practice.  In this sense your badges have already been transformative.  I am glad that you and your colleagues at 4-H dove into this challenge with gusto, and think we are going to learn a lot.