Project Q&A With: Computer Science Student Network Badge System

Project Q&A With: Computer Science Student Network Badge System

Carnegie Mellon University’s Computer Science Student Network (CS2N) is an online learning environment where students, teachers, and hobbyists can earn badges and certifications as they play with, compete in, and learn about computer science and STEM-related topics (CS-STEM). Badges visually document progress to establish concrete curricular trajectories from introductory-level tutorials to industry-recognized certifications.

The goal of the CS-STEM Badge pathway project is to establish a set of Open Badge pathways within CS2N designed to lead students to study Computer Science, Science, Technology, Engineering, and Mathematics (CS-STEM). As a Badges for Lifelong Learning Competition grantee, we expanded CS2N’s technical infrastructure to enable CS2N to automatically award and export CS-STEM Open Badges and establish badge portability between CS2N and other OBI partners (i.e. Chicago Summer of Learning’s unique backpack).

Who were you addressing with your badge system design?

Middle school through adult hobbyist, student through formal and informal educators.

What were your initial goals for the badges? Did those goals change at all throughout the design process?

To develop a badge pathway that stimulates and certifies valuable CS-STEM learning.

What types of badges are you using (participation, skill, certification, etc.)? Are there levels or pathways represented in your badges?

CS2N uses a skill-based badge system to motivate students, help them to define pathways and set goals, and indicate that they have mastered concepts. In CS2N, badges are used to capture and record student learning; CS2N badges measure skill and certify proficiency. CS2N Badges are designed with 5 major goal areas in mind:

  1. Motivation
  2. Assessment
  3. Guidance
  4. Identification
  5. Credentialing

These goals are explained in detail in the following section:

1. Motivation: Modern digital badges are descendants of both the merit badge systems from the Scouting tradition (e.g. Boy Scouts, Girl Scouts), and the achievement systems of online video gaming. Both connect accomplishments with a compact, portable form of recognition designed to motivate earners to pursue further achievement.

Our initial research has focused on motivation, and found that badge’s ability to motivate learners is real, yet nuanced. Certain types of learners respond differently to certain types of badges, both positively and negatively. The implication for badge design is that the details matter: a hap-hazardly-designed system could end up hurting some learners’ motivation rather than helping! CS2N Badges have been redesigned to include only the types of recognition that were found to have net positive gains in student motivation.

2. Assessment: A badge stands for an accomplishment, and must communicate this clearly and credibly to both the earner and anyone who could potentially be evaluating the earner’s qualifications. Further, since badges can be awarded in real-time, they can be used as milestone markers in formative assessment. Two of the main ingredients in making these things possible are Hierarchy and Evidence.

Hierarchy refers to the arrangement of badges into organized categories indicating purpose and significance. It is important that a viewer be able to instantly distinguish an achievement representing a single lesson completion, from an achievement that represents the successful completion of a months-long consolidated project. By structuring badge designs accordingly, a viewer can immediately begin to ascertain the scope of an achievement and begin to understand it in context.

This hierarchical structure also allows CS2N to build specialized tools that exploit the automated real-time awarding of small and medium-sized badges designed to give the teacher an up-to-the-minute way to monitor student progress shown in the example below.

Evidence. CS2N’s badge system includes opportunities for students to submit and display computational artifacts designed to show evidence of badge earner competence. Allowable Evidence in CS2N takes many forms, computational artifacts (for instance, attaching source code to a Programming Badge), instructor endorsement (digital signature by a certified instructor), performance-based assessments of AI-driven tutoring systems, and online exam scores.

3. Guidance. CS2N Badges are uniquely qualified to both suggest and document flexible learning trajectories toward meaningful milestones. Flexible trajectories are increasingly important with anytime/anywhere learning. CS2N provides Mapped Pathways, which offer several key advantages:

First, the Pathway is a concrete sequence of badge-slots that lead to an end goal. The trajectory begins with an introductory-level experience; a startlingly high percent of learners who engage in introductory-level experiences never progress beyond that level. The presentation of this suggested trajectory informs the learner about further opportunities, and provides a viable “default” path forward.

Second, the goal and the steps leading up to it are aligned directly with a specific learning framework, in this case the Computer Science Principles standards under development by the College Board in collaboration with the National Science Foundation. Progressing along this trajectory is, by nature, progressing in the pursuit of recognized Computer Science skillsets.

The Basic Programming Concepts badge in the Badge Pathway is designed so that the Basic Programming slot can be filled with a badge earned in any programming language, with any physical, digital, or simulation-based curriculum tools, so long as they cover the applicable skills and document them with the applicable evidence.

The circular slots in the bottom half of the Pathway are a portfolio-style collection of project-scale activities that the learner selects to serve as evidence of his or her proficiency in key topic areas. The seven slots in align with the seven Big Ideas in the CS Principles Standards Framework.

4. Identification. Part of the core functionality of Badges is their ability to call attention to the qualifications of their owners. To that end, it is important that the badges be both portable and machine-discoverable.

Portability. CS2N Badges are designed to be part of a larger ecosystem of delivering, assessing, and credentialing learning. Our project is guided by the Open Badge Infrastructure (OBI) and its extended badging community. CS2N uses the OBI standard data format so that students who earn CS2N badges will be able to display them in other badge backpacks.

Machine-discoverable. In a world where relevance of information may be contingent upon the ability of search engines to uncover it, a digital Badge provides its earner a unique advantage. CS2N badges embed an invisible packet of specially formatted LMRI metadata information so that they can be easily discovered, understood, and filed correctly by major search engines and learning management systems. CS2N uses the widely supported LRMI format to show the badges alignment with educational standards. This means that not only are LRMI-tagged badges discoverable, but they tell search engines and educational databases directly what Standards-aligned skillsets the badges represent. CS2N large badges, and most medium badges, contain embedded LRMI information that allows them to be discovered, indexed, and associated with the correct Standards by any online system. See the example Robot Algebra badge below.

5. Certification/Credentialing. CS2N badges are records of accomplishments, but not all accomplishments are equally relevant. The top tier of CS2N Badges includes certifications, which represent real-world credentials that are already valued and defined by industry, academia, and other key stakeholder groups. By aligning Pathway outcomes with milestones that already have cultural and representational validity, the value proposition of pursuing completion becomes clearer, and the risk of “reinventing the wheel” is removed.

The Certifications are aligned with the newly emerging NSF funded Computer Science Principles framework – the forerunner work for what is expected to become the entry-level Advanced Placement Computer Science exam – and the National Instruments Certified LabVIEW Associate Developer Certification, an industry-standard first-level certification in the visual programming language that powers LEGO MINDSTORMS robots, but also sophisticated aerospace laboratory equipment and self-driving cars.

How were the criteria for the badges determined?

Previous research including:

  • An NSF funded Robotics Corridor Project which enabled the principal investigators to conduct DACUMS with industry representatives to determine the level of programming the entry level technicians would need to enter the robotic field.
  • Ongoing unfunded research that aligns with the emerging AP Computer Science exam.
  • A partnership with National Instruments and LEGO to develop a badge pathway that leads to the CLAD Certification (Certified LabVIEW Associate Developer).

What pedagogies (if any) informed the learning and badge system design?

Our project builds on existing motivational theory, as well as our own research involving robotics and computer science curriculum development and research-based teacher professional development offerings. The section below is taken from one of our own responses to an NSF RFP.

Badges and Motivation Theory
While the claims and assumptions made about badges being motivational seem credible at first glance, there remains considerable skepticism about whether they are promoting the right kinds of motivation, and there is not yet a clear answer to the question of exactly how badges can fit productively into both the motivational and assessment spaces simultaneously.

Traditional Theories
Traditionally, behaviorist learning science theories cite motivation to learn as originating extrinsically from the learner while constructivist learning science theories propose that the best motivation to learn emerges intrinsically within a learner (Greeno, Collins, & Resnick, 1996). Many research studies in the 70s, 80s, and 90s found that purely extrinsic reward systems (e.g., paying participants in the study for success) actually lowered performance and learning outcomes because they reduce intrinsic motivation (for a review, see Deci, Koestner, & Ryan, 1999). For example, Harackiewicz et al. (1984) conducted a study involving an adjusted pinball machine that offered targeted feedback along with external rewards for high performance. The external rewards reduced intrinsic motivation. This kind of prior research is commonly cited in attacks on badges for learning: if one views badges as extrinsic motivators, then the prediction is for badges to produce a reduction in intrinsic motivation and thereby reductions in learning outcomes. If a goal of reformed CS education is to produce increases in interest in CS careers, a reduction in intrinsic motivation would be particularly counterproductive.

However, the simple intrinsic-extrinsic dichotomy cannot fully explain the complexity within learner motivation regarding badges. Badges are not directly rewarding like money or food. In addition, they can have rich content of relevance to learning objectives. Indeed, even in the Harackiewicz study named above, when the criteria for the reward was made explicit to the learner prior to performance, intrinsic motivation actually increased. Further, motivation and learning research has found it necessary to progress beyond simple intrinsic-extrinsic dichotomies to better explain learning outcomes, and a theory of badges should be rooted in that more modern theorizing. We choose to root our work in two well researched and highly influential theories of motivation: Achievement Goal Theory and Expectancy Value Theory.

Achievement Goal Theory
Achievement Goal Theory (AGT) has identified critical learning goals lying within a two-by-two matrix with Mastery and Performance on one axis and Approach and Avoidance on the other (Elliot, 1999; Cury, Elliot, Da Fonesca, & Moller, 2006). Mastery approach goals reflect a desire to master something based on self-interest in the subject or skill being learned. Performance approach goals reflect a desire to perform demonstrably better, while performance avoidance goals reflect the desire to avoid the appearance of underperforming. Mastery avoidance goals, reflecting a desire to preserve mastery in a skill or subject, exist theoretically but have yet to be identified in most real-world contexts.

Research evidence has shown a positive correlation between mastery goals and academic performance outcomes, with students engaging in deeper processing that enhances learning; there is also clear negative impact of performance avoidance goals on academic performance because students avoid practicing or engaging in disruptive worrying during learning when they have performance avoidance goals (Elliot, Cury, Fryer, & Huguet, 2006). Performance approach goals have had both positive and negative correlations with academic performance, depending on other variables (Elliott, Shell, Henry, & Maier, 2005). A common finding is that combination of high mastery approach goals and high performance approach goals (learners can have both goals at once) produces the best learning outcomes. Overall, AGT has proven to be a good predictor of academic performance in various academic subjects (Pajares, Britner, & Valiante, 2000).

When viewing badges through an AGT lens, several connections are possible. Students could be motivated to earn larger, more meaningful badges based on adoption of mastery goals. However, students could also be motivated to earn any type of badge, no matter how small, by a performance approach goal orientation to have more badges than their peers. But there is also some risk: While a mastery goal orientation leads to a positive learning outcome and a performance approach goal could lead to learning, students could also have a negative learning outcome by adopting a performance avoidance goal of earning just enough badges to avoid being identified as a low badge earner.

Theoretically, badges could be designed and altered to promote mastery and performance approach goals while minimizing the chance a student would adopt a badge-related performance avoidance goal. Badge system designers can create a variety of badges that blend opportunity to earn badges reflecting mastery of a skill or content with badges that are awarded for effort, participation, or even luck. However, it is still unknown whether the goal orientation of a student affects the badges they earn or if the badges being earned can change a student’s goal orientation.

Preliminary studies show promising results. For example, we conducted a preliminary study where we introduced badges to an Intelligent Tutoring System used for mathematics instruction in a 7th grade public school classroom (Higashi, Abramovich, Schunn, & Shoop, 2012). We found that students responded with interest to badges just as they were coming to difficult moments with the tutor. Further, we found that our badging intervention reduced the levels of performance avoidance goals (that typically leads to students doing poorly). In addition, the number of badges earned by individual students correlated with drops in performance avoidance goals. In other words, our preliminary study indicated that the number of badges earned in an online learning tool predicted a decrease in negative learning goals.

Expectancy Value Theory
Another highly relevant theory to badges is Expectancy-Value Theory, which can be applied to unpacking the role badges play in performance and identity—we likely must make changes to students’ identity if we are to increase and diversify the number of students pursuing CS careers. In simple terms, expectancy-value theory assumes learners engage in behaviors only if they think it is “worth it,” and expectancy value theory details the elements that enter into that ‘worth it’ calculation.

At the top level, there are two parts: an expectancy (how likely will the learner be successful) and a value (is the outcome valued). These two are multiplied such that a learner must have both a reasonably high expectancy of success and some reason to value the outcome in order to engage (Wigfield and Eccles, 2000). And there are three different possible values, any of which is sufficient to drive engaging an activity: 1) intrinsic value—the thing itself is worth something (e.g., inherently interesting); 2) utility value—the object is useful for other reasons (e.g., helps get into college or is useful in chosen career); and 3) attainment value—the thing is connected to the person’s identity (e.g., doing poorly would threaten who the learners think they are).

Beginning with the expectancies component, badges could impact students by increasing expectancies for success by providing positive performance feedback throughout learning. However, if the badges are perceived as too easy to obtain, then students might attribute the badge success as indicating the character of the badges rather than their own skill development.

Focusing on the intrinsic value element, educational design often intuitively builds on this kind of a framework by trying to build more interesting activities and materials. However, some attempts at increasing interest have resulted in negative learning effects. For example, the addition of interesting details that are unrelated to the primary meaning of a passage of text resulted in lower student understanding (Wigfield, 1994). Yet a badge can be tightly integrated with targeted skills or content, thereby reducing the potential for distraction but still providing additional interest. If a badge can be designed to increase student interest then there is high likelihood that the same badge will also cause improved learning performance. Intrinsic value and mastery goals are highly correlated, and thus we include the two as one element in our theoretical framework.

Focusing on the utility value element, it may be possible to design badges that show a connection between component skills and larger competencies and careers, thus revealing the larger value of the given coursework. Over time, as students experience success towards larger competencies and experience interest along the way, their identity may change, which in turn increases the attainment value of learning activities in the given topic. Larger badges could be designed that name students with particular identities, further playing into this attainment value shift.

What are three things you learned about badge system design?

  1. Not all badges are meaningful to earners. They must signify things that the badge earner finds useful/significant/relevant.
  2. It is not a trivial task to develop, support, and populates a badge system that track:
    1. Who earned the badge
    2. Who issued the badge
    3. Examples of work (computational artifacts) that illustrate the type of work a student that earned the badge is capable of doing
    4. Who recognizes the badge
    5. The length of time that the badge is recognized
    6. Linked evidence of earner work, where applicable
  3. Learning Resource Metadata Initiative (LMRI) based Standard Alignment tags.
  4. Badge pathways make a long journey achievable.

What would you do differently if you were to start over?

Hire a graphic designer earlier in the process.

What is left to do? What is left unanswered?

This answer is broken into three parts: research, development and testing, and partnerships and articulation agreements.

Continue to build a research base that determines the utility of badges.
Which particular badges are perceived as desirable, easily understood by students, and are accurate indicators of performance?
Does our Badge Theory predict associations of particular badges with particular motivational states?
Does our Badge Theory predict pathways of motivational variables to larger outcomes (skills and career goals)?
Does CS2N’s overall badge ecosystem increase: learner persistence, CS content learning, and CS career interest?

Development and Testing
Over the period of the last year we have been on a sprint to develop and refine “badgeable” course materials that teach Computer Science principles CS novices using multiple CS related applications (robotics, animations, modeling, engineering, simulation). The work needs significant testing and iterative development for it to be an optimal solution.

Partnerships and Articulation Agreement

For badges to become a valid game-changer in education and employment, badges need to be recognized. The badge community needs to align badge efforts with educational and industry needs and develop articulation agreements where badges are recognized as “proof” of competence. Partnerships need to be developed between education institutions, employers, and badge developers. This will take substantial resources until critical mass is reached.

What are the 3 main challenges to widespread adoption of your badge system for your organization?

  1. Recognition of our badges by formal authorities – schools, industry, people that hire.
  2. Recognition of our badges by people that award credit.
  3. Rigorous testing that authenticates that badge earners know the competencies that the badge suggests that they know.

What is your badge system testing strategy? How have you or will you be testing your badge system prior to deployment?

Testing and evaluation is an integral part of our project. Our system has already been deployed. Our testing happens at multiple levels:

  1. Badge Motivational Testing with the University of Pittsburgh- funding for this project ends this summer.
  2. Internal Technical Testing by our technical developers and the CS2N Community. As of today CS2N awards 178 badgeable achievements and has awarded 40K badges.
  3. Internal testing within our teacher professional development courses. Since 2004 the Robotics Academy has trained over 100 teachers per year in weeklong summer training programs. This project has allowed the RA to add “certifications” to the training. Teachers take pre and posttests that enable them to become certified. A certified teacher is able to offer certifications to students through CS2N.
  4. Testing with this summer’s CS2N “Robotics Summer of Learning Project”.

What factors is the success of your badge system contingent upon?

  1. Partnerships/ecosystem acceptance.
  2. Branding and marketing.
  3. A research base that provides evidence of the effectiveness of CS2N’s badging system.

What will you do with the results of your evaluations? What research, if any, will be based on these evaluations, and do you plan to publish the outcomes?

  • Abramovich, S., Schunn, C.D., Higashi, R. (2013) Are Badges Useful in Education?: it depends upon the type of badge and expertise of Learner. Educational Technology Research & Development, March 2013. DOI: 10.1007/s11423-013-9289-2. [Paper PDF]
  • Higashi, R., Abramovich, S., Shoop, R., Schunn, C.D.(2012, June) The Roles of Badges in the Computer Science Student Network. 2012 GLS Conference [Paper PDF]
  • Abramovich, S., Higashi, R., Hunkele, T. Schunn, C.D., Shoop, R. (2011, July) Achievement Systems to Boost Achievement Motivation. 2011 GLS Conference [Paper PDF]

Please describe any impact your badge system may have already had on your organization and your learners.

More teachers are asking to become “Certified Instructors.”

How would you characterize the impact your badge system will have on the badge ecosystem?

Since we are a worked example that offers relevant and rigorous badges we could become a significant contributor to the system.

What plans do you have to scale your badge system?

In our plans “Certified Instructors” will have the opportunity to certify students. We are also interested in pursuing opportunities to partner with expanded dissemination networks, including technical programs in community colleges, and training centers.

How successfully are you getting institutional buy-in, or adoption from your learners?

We have multiple types of partners each having their own reasons for “buy-in”:

  1. Teachers – teachers like CS2N’s badging system because it helps them to market their course (they can offer an additional certification). When a teacher earns a certification it adds prestige, and they also like the automated assessment tools that we have integrated into the system.
  2. The Boy Scouts – we continue to work with the Boy Scouts by helping them to see the alignment between CS2N badges and their new Robotics and Computer Science Merit Badges. Scouts that earn CS2N badges are able to apply the badge to requirements found in the Merit Badge.
  3. Robotics Competition Sponsors – We have marketed certifications to the Robotics Competition Community and over 500 coaches have signed up to take the certification course.
  4. Administrators in schools – They like that their teachers can earn a certification.

Once your badge system is built, how self-sustaining is it? How much do you anticipate maintenance to be?

We plan to continue to sell our teacher training coupled with the new badges and certifications.

For those who want to follow the development, implementation, and adoption of your badge system, what social media sites will you be posting updates to?
Twitter: @CS2Network, @RossHigashi

No comments