Project Q&A With: Design for America: A Badge Community for Innovation

Project Q&A With: Design for America: A Badge Community for Innovation

Design for America: A Badge Community for Innovation will develop a unique online community devised to foster innovation and enable the widespread sharing of knowledge required for a prosperous future. By utilizing the motivational, educational, and connective power of an online badge system, this online community platform will support the professional growth of young innovators and the collaboration of social change makers. Undergraduate designers will be able to track their design projects, share their stories, and learn from each other on the way to solving local problems affecting health, homelessness, the environment, and education.

What are the 3 most important things about building a badge system:

Find out what your community values! If your badges don’t represent what is valued by learners. This is especially the case for more informal environments.
Check your ego at the door, and test, test, test! It would be an incredible statistical freak if the best badge system you can build is the one that you first imagine. Put as many aspects of your design in harms way by testing them with real users (or the closest approximations you can find) and iterate upon what you learn from these tests. This becomes more important the more different you are from your users.
Get on the same page early about goals of your badge system. Be very explicit about these goals, and voice them often. Any large complex system is going to require a large number of people with different expertise and backgrounds - to make them work cohesively requires a shared understanding and set of goals. While your goals may change, your commitment to communicate these goals regularly should not. Have confidence in your changes, and explain your changes with a strong set of reasoning.

We have identified six key audiences for our badge system:

  1. DFA project members: DFA university students who are working on projects.
    Needs: Students desire support in the design process. They want to be aware of possible trajectories that other project teams have or are taking. They also want to be able to select from a menu item of design activities/badge challenges that can help them with their projects.
    Solutions: A community expert recommender-systems based upon who has what badges. A presentation system that shows the history of badge challenges undertaken on each project page. A community wide stream that shows what activities project teams are undertaking, including badge challenges undertaken and completed.
  2. DFA studio leads: Each DFA studio has 2-5 leads who are responsible for maintaining the studio, and supporting the project teams at the studio.
    Needs: DFA studio leads want to have quicker and easier oversight of their studios. Studio leads report that it is a tremendous amount of work keeping in touch with everything all the project teams are doing.
    Solutions: Functionality and UX design that makes it easy to browse over projects by studio. A central steam that shows what activities project teams are undertaking, including badge challenges undertaken and completed.
  3. Professional mentors: Design professionals and others with relevant expertise mentor DFA project teams, and give feedback at events. The Digital Loft hopes to offer another channel for mentorship and feedback along badge criteria.
    Needs: Professional Mentors are motivated to do meaningful work with DFA, especially when they have relevant expertise useful to project teams. However, they want to feel that the team is ‘pulling them along’, not that they are ‘pushing’ or motivating the team. They also want to control and cap their involvement - they don’t want to feel continually beholden to teams. To give quality feedback they need to be able to quickly gain relevant information about the project.
    Solutions: Functionality and UX design that makes it easy to browse over projects by studio. A manual recommender system, so that mentors can recommend badge challenges to their mentee project teams. Strong privacy functionality. Feedback criteria to help mentors give feedback (see fig 1).
  4. DFA Fellows and Director of Operations: Design for America HQ is staffed by 3 DFA Fellows - previous DFA students who work full time at DFA, mentor teams and studio leads, and attend to the DFA network. They are also responsible for the design of DFA instruction.
    Needs: They need to quickly browse to see the progress and struggles of those in the studios they are mentoring (similar to studio leads). They want to have measures of studio, project, and network health. They also need to easily edit instruction related to badges.
    Solutions: Functionality and UX design that makes it easy to browse over projects by studio. A manual recommender system, so that mentors can recommend badge challenges to their mentee project teams. Quick and easy editing functionality for badge challenges. Help functionality that allows project teams to signal there level of confidence regarding the badge challenge they are undertaking.
  5. Potential Employees: Employees who might give DFA members a job based upon their experiences and achievements at DFA.
    Needs: To be able to quickly assess evidence of the competence of DFA members.
    Solutions: Badges that show design artifacts that are strong evidence for competency.
  6. Design-Based Researcher/Educational R&D (Professors, grad students, research assistants): Researchers working on design-based research projects with DFA and the Loft.
    Needs: Design-Based Researchers need a lot of usability data, and feedback on the design of the Loft from users. Like the fellows, they also need to iterate upon the design of instruction.
    Solutions: Analytics. Quick and easy editing functionality for badge challenges.

There are a number of other important stakeholders involved in the Digital Loft Badges System, but given our current resources these are the ones we can seriously consider and investigate for the time being. Our initial goals were to highlight expertise in a geographically distributed community, scaffold a complex design process, and to encourage participation and contributions within the online community. Having reviewed all of the possible courses that different design teams can take, we have realized that we need to create many more badge challenges than DFA currently has. For this reason we have added another goal of drawing in a community of badge challenge designers. For this reason the design team has recently spent a lot of time and resources developing editing functionality so super-users can edit badge challenge pages by simply clicking onto the page.

Badge system design/models:

We currently have the functionality for skill badges and certification badges. The skill badges represent the ‘lower-level’ design skills (e.g user interviews, persona creation, affinity diagraming, brainstorming). The certification badges are for ‘higher-level’ accomplishments in the design process (e.g. synthesizing insights, creating a design argument, developing prototypes). Later we hope to develop participation badges for user contributing to the Digital Loft (e.g. giving feedback on the badge challenges of others).

We also hope to use badges as a record of what design activities DFA project teams have undertaken -- this will help project teams document their process, other students see which badge challenges successful teams have undertaken, and the design team to understand what are more successful routes through the process. Also the evidence for each badge can be used by DFA’s project teams as they learn the skills related to each badge challenge. On top of this they can be drawn upon by DFA’s instructional designers to curate and turn into quality instruction.

Methodology and Design Principles:

We are following a user centered instructional design methodology (Easterday et al. 2013) which involves: a) initial need finding; b) learner and expert interviews to determine the desired expert competencies and learner struggles (e.g. Clements & Battista, 2000); c) defining learning goals along the Understanding by Design framework (Wiggins & Mctighe, 2005); d) defining a comprehensive outline for each badge (framework created by Dr. Nicole Pinkard of the Digital Youth Network) and related rubrics (Wiggins & Mctighe, 2005); e) building out designs; f) testing designs (Tessmer, 1993).

The way in which we plan to use badges to scaffold the design process has been highly influenced by the Learning by Design framework (Kolodner et al., 1998), analogical encoding (learning from and comparing across cases - Gentner, Loewenstein, & Thompson, 2003), as well as Hatties model of feedback (Hattie & Timperley, 2007), and Ron Burger’s In-depth Critique Protocol. For a more detailed outline of the learning principles for the Digital Loft see Easterday et al. (2013).

What are three things you learned about badge system design?

  1. Less faith in user generated content: Testing has taught us to have less faith in the quality of user generated evidence that is tied to badges. We had originally assumed that such examples would be immediately useful instructional material. We now see that although badges do have the ability to help point the way to high quality work within an online community, the real value is in sourcing good examples for instructional designers to curate.
  2. Badges ≠ Process Steps. Badges need to map onto what the community finds valuable.
  3. Badges need to look like Badges! This seems really obvious on the face of it, but for DFA we wanted to underplay traditional graphic representations of badges as initial needs analysis suggested our users wouldn’t react well to badges. However, testing has unveiled two significant problems with this approach. Firstly, users don’t recognize the badges as badges. Secondly, we want to connect, but differentiate badges from the process icons - users got confused between what was a badge, and what was a process icon.

What is left to do? What is left unanswered?

There are a number of important questions left to answer. Our most immediate questions are:
What is the quality of community feedback? How can we design to improve the feedback that learners receive on badge challenges? Have we judged the amount of content in each badge challenge correctly? A badge system that supports students taking on real-world design projects will need a really modular badge system. Getting this level of modularity right is a big challenge.

Can we draw in a community of badge designers to design or co-design badge challenges with us? How can we ensure the quality of these badge challenges? We anticipate a very high demand for many different types of badge challenges given the variety of projects we have in the DFA network. In DFA we produce our own instruction, but we also draw on other expert instruction from IDEO, Frog, Jon Kolko, and design professors within our network. We anticipate that we will be able to get design experts to co-design badge challenges, as many of the experts we initially approached stated they were interested in designing Digital Loft badge challenges.

What are the 3 main challenges to widespread adoption of your badge system for your organization?

3 main challenges to widespread adoption include:
Getting the feedback system to work well. The design of the Digital Loft Badge System requires that the community are able to provide quality feedback on the badge challenges of project teams. Initial testing with design learners at Northwestern University suggests that this will be a significant design challenge.
Ensuring that the usability of the site is very very high. As DFA is a student-led informal space learners are not ‘obliged’ to use the site in order to participate in the learning environment.
Creating enough badge challenges to meet project-team needs.

What is your badge system testing strategy? How have you or will you be testing your badge system prior to deployment?

We have been testing the Loft for the last 4 months. Testing involves:

  • Regular feedback on paper prototypes from DFA members
  • Regular feedback on badge challenges from DFA member, instructional experts, and content experts
  • Testing process page and feedback functionality by using the Digital Loft for a 10 week design class at Northwestern University. The class was taught by Matthew Easterday. 15 proxy users were regularly interviewed on their experiences using the Loft.

What is the success of your badge system contingent upon?

  1. Feedback: Producing quality feedback that helps our learners produce better design projects that have more chance of making positive social impact.
  2. Ownership and Adoption: Creating a sense of ownership of the Loft amongst the DFA students so that they both use the Loft, and give suggestions to the Loft design team.
  3. Contributions: Project teams uploading design process artifacts associated with badge challenges for review, and the community spending time to give quality feedback on these artifacts.

Research:
Professors Easterday and Gerber will conduct design-based research on the Digital Loft. Evaluating the Digital Loft will also form the basis of two PhD dissertations. Both Daniel Rees Lewis (currently project manager - starting Learning Sciences program at Northwestern in Autumn 2013) and Colin Fitzpatrick (current 1st year Technology and Social Behavior PhD) will conduct design based research evaluating the design and effectiveness of the Digital Loft for the next four years.

Continual evaluations will form the basis of iterative design-and-test cycles. Methods will include (but not limited to): user interviews, think alouds, observations, and online usage analytics.

We have currently published 1 paper on the Digital Loft:

  • Easterday, M. W., Rees Lewis, D. G., Gerber, E. (2013). Formative feedback in Digital Lofts: Learning environments for real world innovation. Workshop presentation at the Artificial Intelligence in Education Conference 2013, Memphis, US.

We also have 1 paper in preparation:

  • Rees Lewis, D. G., Easterday, M. W., Gerber, E. (Manuscript in preparation). Understanding Civic Innovation Preparation: Implications for Support Tools. International Journal of Design.

The Digital Loft Badge System will first be implemented on the 29th of July at DFA Northwestern’s Summer Studio -- an intense 6 week program with 4 project teams (20 learners) and 4 team anchored professional mentors. The Loft will then be made available to DFA studios nationwide in September 2013.

DFA as an organization has had a history of spreading very effectively. As an organization it has spread effectively through word-of-mouth with its members. As such we do not plan to ‘mandate’ use of the Digital Loft in the first year (starting September 1013). Instead, we request a number of early adopter volunteer teams to use the Loft. Eight teams from 4 different DFA studios (Northwestern, Cornell, Virginia-Tech, and Barnard-Columbia) will be enlisted to use the Loft, and receive additional mentorship, and the opportunity to co-design the Loft. These early adopter teams and mentors will be interviewed every month to allow the design team to modify the design to fit the users needs.

What plans do you have to scale your badge system?

Our strategies for scaling include:

  • Seeding content from high-status teams so that the Loft has some content on roll-out.
  • Requesting high status members to be involved in the first 6 months after roll out - giving feedback, and maintaining a strong social presence. Four ‘high-status’ DFA members have committed to giving feedback and maintaining a strong social presence for this time-period (and beyond). These DFA members are on the DFA Council, and are currently running succesful start-ups,.
  • Designing functionality that promotes high status teams on the landing page (see fig. 2). As such the goal is to show ‘people like me’ doing exciting project work.
  • DFA HQ cultivating relationships with professional mentors to participate. In the past we have found that when asked by Design for America HQ, professional mentors are much more likely to contribute and help DFA project teams.
  • Providing very obvious opportunities for DFA project members and studio leads to give feedback and feel ownership over the Digital Loft. This will include 'crit' sessions at DFA’s leadership studio, and functionality on the Digital Loft for users to suggest design changes. We will also offer opportunities for more experienced DFA members to design Digital Loft badge challenges.

Ultimately adoption will only occur if the Loft solves the real needs of the users.

Contrary to more traditional educational research projects, we envisage a permanent collaboration between researchers and practitioners. We see there being much more benefit in narrowing the gap between educational research and practice, and that educational research should play the same role as R&D in the corporate world. That is, researchers should employ design-based research practices that are beholden to their effectiveness in solving problems for educational practitioners. For this to occur, long-term partnerships need to arise, with the product of such partnerships being: a) effective practice; and b) communications to the wider educational community on how to scale similarly effective learning environments to other contexts. For this reason we hope to establish and ‘educational R&D’ research group, led by Professors Gerber and Easterday to work with DFA. This group will be responsible for writing grants and other fundraising activities for DFA and the Digital Loft Badge System.

We are in talks about sharing the Digital Loft Badge System with Badges for Lifelong Learning Competition winner, Mouse, who have a very similar learning environment to DFA in their Mouse-Corps program. We plan to co-design human-centered design/co-branded badges with Mouse for the Mouse Corps program. This way we can have a badge-trajectory for human centered design learners from high-school through college.
 

No comments