Blog Post

Good Badges, Evil Badges? An Empirical Inquiry into the Impact of Badge Design on Goal Orientation and Learning | Jan Plass

Good Badges, Evil Badges? An Empirical Inquiry into the Impact of Badge Design on Goal Orientation and Learning | Jan Plass

There has been a lively debate recently among members of the badges community about the impact of badges on people’s motivation. Some are concerned that badges might stifle students’ intrinsic motivation and cause them to be more focused on winning new badges than on the work they are doing. Others support the use of badges, considering them superior to grades for evaluating student performance (Openbadges List, 2012). Yet whatever the differences, there seems to be agreement that the way badges are designed, and how they are interpreted by the learner, will ultimately determine the effects they will have on users. Dan Hickey refers to this as the purpose versus function of badges (Hickey, 2012), which mirrors similar ideas between the intentions of the designer versus the users’ interpretation of the design by Norman (1988).

Clearly, this is a critical question that calls for empirical evidence. We propose to collect such evidence to inform the discourse on the purpose and function of badges, and to develop theory-based, empirically validated design patterns that will support badge designers and issuers in their design decisions.

In particular, we propose to:

(1) Observe user performance with and without badges of specific functions and investigate corresponding patterns of behavior and performance.

(2) Develop a Badges Impact Survey (BIS) based on the results from Part 1 and a theoretical framework of situated learning, situational interest, and achievement motivation. This scale will measure the effects of badges on goal orientation and perceived learning.

(3) Conduct an experimental study on the effects of badges on goal orientation and learning outcomes by modifying a geometry game to issue badges with specific designs and by measuring their impact on goal orientation and learning.

Theoretical Rationale

Situated Learning

Learning as a process that is highly contextual and involves the co-construction of knowledge in a community of practice (Lave & Wenger, 1991). For adult learners, the context in which learning is situated is often their real life, but it has been suggested that for middle school students, video games can provide similarly rich contexts in which meaningful learning can take place (Gee, 2003). We are interested in how badges shape this context, and what aspects of the learning process are affected by badges with specific designs. Of particular interest here are learners’ individual and situational interest in the learning activity and topic, as well their motivation to learn.

Individual and Situational Interest

Interest and motivation are often described as interdependent (Krapp, 2005; Schiefele, 1991; Ainley, 2006). Interest can directly affect students’ motivation, learning strategies, and ability to process information (Schiefele, 1991). In addition, interest mediates the relationship between students’ overarching goal orientation, their task-related goals, and their motivation to learn (Ainley, 2006). Interest is typically described as content-specific for an individual, e.g., a learner may be interested in history but not in math. However, we find it useful to also consider situational interest, which is brought about by different circumstances or events. For example, a person may not be interested in math (low individual interest), but might find a math game with badges situationally interesting. This is important since situational interest can lead to individual interest in the topic that persists over time (Hidi & Renninger, 2006; Krapp, 2007; Schiefele, 1991).

Achievement Motivation

Achievement is inextricably linked to goals, since in order to accomplish something; it is useful for individuals to have a clear idea of what they are working towards (Beck, 1983). Depending on what one believes to be important, the motivation for behaving in a particular manner in that situation will change (Dweck & Leggett, 1988), and we plan to explore this link between users’ patterns of behavior and their psychological processes in the context of badges.

The study of achievement motivation is centered around two types of goal-oriented activity: (1) mastery goals, where the focus is on learning and understanding in order to achieve competence, and (2) performance goals, where the focus is on achievement relative to peers or avoiding negative views of their competency (Dweck, 1986). Depending on the type of goals the students are aiming towards, they will respond differently to changes in a situation, especially failure (Ames, 1992; Urdan, 1997). Research into these response patterns has been valuable in explaining adaptive or maladaptive thought-processes, emotions, and behavior in different types of achievement situations (Spinath & Stiensmeier-Pelster, 2003).

These response patterns translate into two types of goal orientation: mastery goal orientation and performance goal orientation. Mastery goals have been shown to have a slight advantage over performance goals, however, the range of outcomes in past research is likely due to influence of other factors (Utman, 1997). For example, goal orientations, when combined with students’ interest in the situation and the topic, can affect their motivation to learn in a given setting (Schiefele, 1991; Krapp, 2005).

With this theoretical framework as foundation of our empirical investigation of the mechanisms underlying the effects of specific types of badges on goal orientation and learning outcomes, we propose the following three research threads:

(1) Observation of Game Play with or without Badges

We propose to take a broader standpoint by observing the differences in behavior and perceptions within a range of publicly available games depending on the presence or absence of badges with different types of functions. Badge functions will be divided according to whether they recognize, assess, motivate, or evaluate learning. Video recordings of gameplay will be analyzed for trends and insight into participants’ perception and valuation of badges as part of their gameplay experience, and for changes in gameplay patterns due to badges. This observation could alternatively be conducted with a DML awardee if their system implements these types of badges.

(2) Development of Badges Impact Scale (BIS)

The results of the observation study will be presented in a workshop with experts in the field to discuss findings and start designing the BIS, which would be completed as post-workshop activity, reviewed by experts, and later piloted.

(3) Experimental Study

In this final thread, we will conduct an experimental study on the effects of different types of badges. The following is a description of the design we currently plan for this study, however, results of the previous two research threads may lead to modifications. Unlike in the observation research, which considered badge types such as recognition, assessment, motivation, or evaluation of learning (Hickey, 2012), we will use the orthogonal construct of goal orientation. Each of the four badge types described by Hickey could be designed as either encouraging a mastery orientation (related to personal progress) or as encouraging an achievement orientation (relating own performance to that of the group). Therefore, we will mod our puzzle game, Noobs vs. Leets, designed to teach geometry concepts, to systematically implement various badge designs and badge presentation methods. This will allow us to assess the effects of different designs based on achievement goal theory. The study will use a 3 x 2 design (3 levels of the badge design: No badges, Mastery badges, and Performance badges; 2 levels of achievement goal: Mastery and Performance).

Mastery badges will be designed to encourage the learner based on their own ability, helping the player assess their own aptitude and progress. Performance badges will be designed to reward the player in comparison to their peers, giving feedback on progress relative to other players’ performances. The different levels of achievement goal manipulation will be achieved by asking participants to read a short paragraph instructing them on the purpose of the game, how they should approach gameplay, and how to measure achievement within the game. We expect to see variations in our results depending on whether or not the goal manipulation condition coincides with the type of badge condition received.

Players will complete a variety of surveys that assess their individual interest in games, individual interest in geometry, and situational interest. Pre- and post-tests of geometry items will evaluate their conceptual knowledge. Players’ goal orientation will be assessed before gameplay using items from the Patterns of Adaptive Learning Scale (PALS, Midgeley et. al, 2000). Measures of game performance and logfiles of in-game actions will be collected as well as videotapes of players’ reactions. These data and assessments will allow us to study the effects of the different badge designs on key indicators of players’ attitudes, such as individual interest in the subject matter, situational interest, goal orientation (mastery v. achievement), as well as learning outcomes.

Outcomes & Deliverables

Outcomes of the proposed work will be:

–A workshop to present findings from our observations and design the BIS with leading experts in the field.

–A presentation at a national or international research conference.

–A white paper or MacArthur Foundation Digital Media and Learning report of our findings that will also be submitted to a peer reviewed journal.

Public Forum Narrative

There has been a lively debate recently in the badges community about the impact of badges on motivation. Some are concerned that badges might stifle intrinsic motivation and cause users to be more focused on winning new badges than on the work they are doing. Others support the use of badges, considering them superior to grades for evaluating performance (Openbadges List, 2012). Yet there seems to be agreement that the way badges are designed, and how they are interpreted by the learner, will ultimately determine the effects they will have on users.

We propose to investigate this question empirically. We will first examine the issue from a broader standpoint, observing the differences in behavior and perceptions within a range of publicly available games and the presence or absence of badges of specific functions.

Next, we will develop a Badges Impact Survey (BIS) that can be used by badge issuers and researchers to survey badge users in order to measure the impact of badges on goal orientation and learning.

Lastly, we will mod our geometry puzzle game, Noobs vs. Leets, to systematically implement various badge design and presentation methods, allowing us to study the impact of these different designs on key indicators of players’ attitudes, such as interest in the subject, goal orientation (mastery v. achievement), and learning outcomes.

Results will be formulated into design patterns for the developers and issuers of badges that will help build a more nuanced understanding of the impact of specific badge designs.

Citations

Ainley, M. (2006). Connecting with learning: Motivation, affect and cognition in interest processes. Educational Psychology Review, 18(4), 391–405. doi:10.1007/s10648-006-9033-0

Ames, C. (1984). Achievement attributions and self-instructions under competitive and individualistic goal structures. Journal of Educational Psychology, 76, 478-487.

Ames, C., (1992). Classrooms: Goals, structures, and student motivation. Journal of Educational Psychology, 84, 261-271.

Bandura, A. (1986). Social foundations of thought and action. A social cognitive theory. Englewood Cliffs, NJ: Prentice Hall.

Beck, R. C. (1983). Motivation: Theories and principles. Englewood Cliffs, NJ: Prentice-Hall.

Dweck, C. S. (1986). Motivational processes affecting learning. American Psychologist, 41(10), 1040–1048. doi: 10.1037/0003-066X.41.10.1040

Dweck, C. S., & Leggett, E.L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95 (2), 256-273.

Henderson, R. W., & Landesman, E. M. (1993). The interactive videodisc system in the zone of proximal development: Academic motivation and learning outcomes in precalculus. Journal of Educational Computing Research, 9(1), 29-43.

Hickey, D. (2012). Intended Purposes Versus Actual Functions of Digital Badges. Post in Blog, Re-Mediating Assessment, on September 19, 2012. Available:http://remediatingassessment.blogspot.de/2012/09/intended-purposes-versus-actual.html

Hidi, S., & Renninger, K.A. (2006). The four-phase model of interest development. Educational Psychologist, 41(2), 111–127.

Krapp, A. (2005). Basic needs and the development of interest and intrinsic motivational orientations. Learning & Instruction, 15, 381-395.

Krapp, A. (2007). An education-psychological conceptualisation of interest. International Journal for Educational and Vocational Guidance, 7(1), 5-21. doi: 10.1007/s10775-007-9113-9

Lave, J., & Wenger, (1991). Situated Learning. Legitimate peripheral participation. Cambridge, UK: Cambridge University Press.

Middleton, J. A., & Spanias, P. A. (1999). Motivation for Achievement in Mathematics: Findings, Generalizations, and Criticisms of the Research. Journal for Research in Mathematics Education, 30(1), 65–88. doi: 10.2307/749630

Midgley, C., Kaplan, A., Middleton, M., Maehr, M. L., Urdan, X., Anderman, L. H., et al. (1998). The development and validation of scales assessing students' achievement goal orientations. Contemporary Educational Psychology, 23, 113-13.

Midgley, C., Maehr, M. L., Hnida, L. Z., Anderman, E., Anderman, L., Freeman, K. E., et al. (2000). Manual for the patterns of adaptive teaming scales (PALS). Ann Arbor: University of Michigan.

Miller, R. B., Behrens, J. T., Greene, B. A., & Newman, D. (1993). Goals and perceived ability: impact on student valuing, self-regulation, and persistence. Contemporary Educational Psychology, 18, 2–14.

Nicholls, J. G., Patashnick, M., Cheung, C. P., Thorkildsen, T. A., & Lauer, J. M. (1988). Can achievement motivation theory succeed with only one conception of success? In F. Halisch, & J. H. L. von den Bercken (Eds.), International perspectives on achievement and task motivation (pp. 187–208). Lisse: Swets und Zeitlinger.

Nolen, S. B. (1988). Reasons for studying motivational orientations and study strategies. Cognition and Instruction, 5, 269–287.

Norman, D. (1988). The Design of Everyday Things. New York: Basic Books.

Openbadges List (2012). Thread of the [openbadges] list: Badges and Motivation, initiated July 17, 2012 by a post from Chris Sloan. Most recent response: September 19, 2012 (11 posts total).

Pajares, F. (1996). Self-efficacy beliefs and mathematical problem solving of gifted students. Contemporary Educational Psychology, 21, 325-344.

Schiefele, U. (1991). Interest, Learning, and Motivation. Educational Psychologist, 26(3-4), 299–323.

Spinath, B., & Stiensmeier-Pelster, J. (2003). Goal orientation and achievement: The role of ability self-concept and failure perception. Learning and Instruction, 13(4), 403–422. doi:10.1016/S0959-4752(02)00014-2

Urdan, T. (1997). Achievement goal theory: Past results, future directions. In M. Maehr & P. Pintrich (Eds.), Advances in motivation and achievement (pp.99-141). Greenwich, CT: JAI.

Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. Pintrich, & M. Zeidner (Eds.), Self-regulation: Theory, research, and applications (pp.13-39). Orlando, FL: Academic Press.

*****

Jan Plass is the Paulette Goddard chair in Digital Media and Learning Sciences in the Steinhardt School of Culture, Education, and Human Development at New York University, where he co-directs the Games for Learning Institute, and a Digital Media and Learning Research Competition on Badging and Badge Systems Development grantee based on his winning proposal Good Badges, Evil Badges? An Empirical Inquiry into the Impact of Badge Design on Goal Orientation and Learning. The Research Competition was funded by the Bill and Melinda Gates Foundation and administered by HASTAC, in collaboration with Mozilla. 

Research Competition applicants were asked to submit proposals for empirical and theoretical research that support and inform the design, development, and deployment of digital badges and badge systems across a diverse range of learning content, institutions, and approaches, including the Gates Foundation supported Project Mastery Sites, as well as research focused on the efficacy of Teacher Mastery badging projects.
 
63

No comments