Project Q&A With: The SA&FS Learner Driven Badges Project

Project Q&A With: The SA&FS Learner Driven Badges Project

UC Davis Agricultural Sustainability Institute (ASI) is developing a model platform for validating experiential learning within formal institutional contexts at the undergraduate level. In creating a learner-driven, content-rich badge system, ASI will establish a new model for bridging learning in and out of the classroom, and enable learners to better communicate their skills and competencies to a broad audience.

What are the 3 most important things about building a badge system you would share with another organization just getting started?

  1. Learn about design research and what that process entails. Think very carefully about your user experience. 
  2. It’s really important to understand what’s already out there and who’s building things that you can borrow from or collaborate on. While we don’t need one badge platform or system to rule them all, there are many great examples to draw from and by supporting and investing in further development of those core examples, we can actually move the whole enterprise forward in a coherent way and still provide design resources for the greater ecosystem.
  3. Know that this is a longer and more resource intensive process that other types of curriculum development efforts might be. Be realistic about your expectations for the process and don’t be afraid to be agile and do things incrementally and iteratively if possible, with an eye for extensibility into future phases once you have proof of concept, a working prototype or some first element of the system for your students and faculty to play with and get comfortable adopting.

Who were you addressing with your badge system design?

Our badge system is designed first with the learner in mind. While we are very interested in the potential uses and markets for earned badges, and have thought carefully about these issues in our design process, our main goal is to support a learning process that is student-driven and outcomes-based, in a way that helps faculty and other mentors and advisers interact effectively with students and create a learning community that can foster and dynamic and networked experiences for everyone in the system. 
 
We have designed this system to be used with the curriculum for the SA&FS major, but have also attempted to keep the structure open and adaptable, so that it can be easily implemented for any major that has identified a set of learning outcomes for its academic program. 
 
We envision this system to be useful, at a minimum, for all majors at UC Davis, as well as majors similar to SA&FS at higher ed institutions across the country. We collaborate on curriculum development with many programs and institutions like ours through our work in the Sustainable Agriculture Education Association (SAEA) and in our role at ASI as Chair of the Kellogg Interinstitutional Network on Food and Agricultural Sustainability (INFAS) and as a member of AGree, an organization engaged in efforts that include reforming agricultural research and education in higher ed institutions.  We've also been active in developing opportunities for collaboration with other colleges and universities—some are members of the Open Badges community—interested in adopting badging systems, as well as other DML grantees, such as the 4H badges project, where we're exploring the potential to develop learning pathways from K-12 into post-secondary institutions throughout the land grant university system. 
 

What were your initial goals for the badges? 

Our initial goals were as follows:
 
  • Develop a tool that helps students and faculty visualize and concretize the fairly abstract process of engaging in and reflecting upon competency development and integration of learning experiences across disciplines, and from inside and outside of the classroom
  • Help students and faculty organize their thinking around the nature of competency development, how it unfolds and where it takes place
  • Create a system that helps individuals track progress over time and beyond the traditional grading and evaluation system contained within the course structure; this will provide students with a more comprehensive view of their growth and development and support them in the creation of an individual portfolio of work and collections of badges
  • Create a system that provides data we can use to evaluate the strengths of our program and that will inform our adaptive curriculum management process. 
  • Design with extensibility in mind; serve as a model for other programs and future design efforts in the badges ecosystem
  • Propose a theory of learning, motivation and social change through our system design, as follows:
    • Learning is a socially mediated process and students need feedback from peers, mentors and experts to validate and solidify their learning. 
    • In order to integrate learning experiences happening across disciplines and in differing contexts and spaces, students need tools to support reflection and synthesis at a more holistic level. This system should facilitate that process. 
    • Students should be in control of how and to whom they present the products of their education, via portfolio and badging mechanisms. This allows for an interest-driven learning process that leads to the development of unique pathways for each student and unique presentations of their individual contributions before their wider community. 
    • Badges should be tied to rigorous standards for learning and achievement; those standards should be enforced not just by the administrative authorities within the major, but through participation and consensus mechanisms from within the learning community, which includes students, faculty, and other supporters and mentors of our students.  

Did those goals change at all throughout the design process?

These are still our goals, though we’re focusing on achieving some of them in this initial design phase and others in future iterations. Specifically, we are currently focusing on the quality of the individual user experience and its supporting data model. We have ongoing questions about how to best implement the peer-to-peer features of our system, as well as how best to invite non-UC Davis users to participate in the SA&FS learning community via badges. For now, we are focusing of the individual student’s portfolio development and badge building aspects of the system. We hope to show the importance of some of those more social/community aspects of the system in a future phase. 

What types of badges are you using (participation, skill, certification, etc.)? Are there levels or pathways represented in your badges? If so, please describe.

Our badges have a tiered design. As a program, we have identified seven competencies or student learning outcomes, each of which will map onto a specific competency badge. Those competencies include: Systems Thinking, Experimentation & Inquiry, Interpersonal Communication, Understanding Values, Strategic Management, Civic Engagement, and Personal Development. 
 
Additionally, we have created badge types that do not have pre-determined criteria and that will allow us to continually design and define new badges for our system. Students can earn Skill Badges, Knowledge Badges, Experience Badges, Honor Badges and Competency Badges. As a program, we might develop a Skill Badge in “Community Nutrition Education”. We also intend to encourage students to design badges. A student might propose a set of criteria for a knowledge badge in “Agricultural Pests of California”. They can earn that badge individually, or we can make it available to other students as well if we, as a program, determine the criteria to be sound. 

How were the criteria for the badges determined? What pedagogies (if any) informed the learning and badge system design?

The system is designed to be interest driven, socially mediated and academically relevant. In order to satisfy these design principles of a connected learning approach, we made a decision to focus on developing a process for criteria selection and subsequent acceptance of badge design that students could be a part of. Thus we don’t have defined criteria for all our badges, nor do we have a complete, finite set of badges identified. We have a platform for developing new badges and earning ones that are made available by the program. Eventually, we’ll have a community review process in place that helps guide the process of choosing which badges should be awarded within our system. 
 
We only have program-determined criteria for our 7 SA&FS competency badges. These badge criteria are determined out of the learning objectives from the core curricula of the major. The criteria consist of having completed the core courses of the major, including the senior capstone, having complete the internship requirement of 12 units, and then, depending on the competency, a mixture of other types of badges or work from the student’s portfolio. 
 
The badge system design is informed by connected learning theory, competency-based learning, experiential learning theory (Kolb, Dewey, etc.), and a social constructivist critical pedagogy that underpins the core learning experiences in the major. 
 

What are 3 things you learned about badge system design?

  1. User Experience. This is not always simple or intuitive from the curriculum designer’s perspective. Thinking through the user experience is a really critical part of the design process. Getting feedback and engagement from your users early in the design of the system is key. The system will likely fail if you don’t get this right. 
  2. Speaking different languages within the team. Communicating design ideas from content expert to design expert to programming expert is something that requires skill and experience. An interest in badges does not equal the capacity to think with agility and an innovative mindset about the potential of a badging system. Translating the complexities of a human-centered user experience into the logic of programming language takes time to get right and a great deal of refinement. This can be a longer and more resource-intensive process than what might be expected, especially for those content experts unfamiliar with web development in general and human-centered design in specific, or those web experts unfamiliar with progressive educational theory. 
  3. Designing to include faculty. A badge system can lead to some real efficiencies in the interactions between faculty and students, if designed with both parties’ needs in mind. If you want to be successful in realizing this potential, it’s really important to think completely through faculty user experience.  

What would you do differently if you were to start over?

We have an opportunity to redesign the system right now, which is allowing us to go back to the drawing board on some aspects of the system, while sticking with those features we’ve identified as fully functional and desirable. We've attached the current system overview and will provide complete documentation of the redesign in our final narrative report. In a sense, we are currently asking ourselves what we’d do differently, and most of it has to do with starting from a user experience perspective, and including the faculty user experience in that thinking. We made some assumptions that faculty would be willing to deal with a less polished interface and a clunkier, back-end experience. Given the demands on their time, that doesn’t turn out to be a fair assumption. We’re looking at how to get input from our own faculty as well as those professors at other institutions who’ve contacted us about our badge system. We anticipate that this will help us rethink how faculty engage in providing feedback to students and how the badge-earning and issuing process will include faculty, what number of them is effective and to what extent. 

What is left to do?

We have to finish building the system, including the aspects we’re in the process of redesigning, open it to user testing and then roll it out with students in the fall of 2013. Then, we need to figure out how the system might really be built upon and/or adopted by others. We need continue to engage in communication with faculty at other universities who are interested in our system, and look to a next phase of development that might source ideas from across institutions, and take a more collaborative, crowd-sourced design approach, with many of those interested faculty members participating. We’d be interested in taking the lead on that kind of project because we see a tremendous benefit for our own development and for the creation of a system or a code base that meets the needs of other programs, too. 

What is left unanswered?

We still need to figure out whether our theories around learning and motivation really work in this context. Will students be motivated to give and get feedback from peers, mentors and experts in their field? Will they want to earn and share badges without us requiring them to do so? Will they be motivated to design new badges that reflect their own interests and body of work?  We need to get to the next development phase so we can build out more features that will make the system hang together with a coherence that helps drive students into the pathways, spaces and practices that should encourage and motivate them to keep using the system. 

What might help you continue to succeed?

We really need continued dialogue with other higher ed institutions that are interested in building badging systems so that we can share ideas and possibly build off of the same code base. 
 
We also need continued exposure for our program and effective communication about what we’re trying to accomplish. There a lot that’s still unclear about badges and often our work is hard to explain in traditional terms, so we’ve seen it misrepresented on occasion. We need to figure out the right messages about our work, and where to communicate those messages for the most impact. 
 
Certainly, access to financial resources, both internal and external, would help us continue building our own system, supporting the development of other programs and communicating more broadly about the potential of badges. 

What are the 3 main challenges to widespread adoption of your badge system for your organization?

  1. We have to get to the point where what we have is ready to be made available to our students and to other programs interested in our work, and we aren’t there yet. Thus, development time has been a challenge because we want to engage users in the design process, but we don’t want to turn them off from using the end product because we didn’t give it enough time in development before we release for testing and they get frustrated by a bad user experience. It’s proving to be a more delicate balance than expected. 
  2. We need to become increasingly effective in our communication about the system and the design principles behind it. This is true within our program and to the academic and badge-curious community at large. We believe we are proposing a reimagining of learning within higher ed and it’s easy for people to misunderstand certain aspects of that proposition and the real practices behind it. 
  3. Resource questions. Will we be able to continue building to a full-scale implementation within our program? What internal and external resources will exist to help us move towards a wider-scale adoption at UC Davis? And for other institutions, what will be the funding model that will allow them to build on what we’ve done, or make a departure from it, in adopting badging practices within their programs? 

What is your badge system testing strategy? How have you or will you be testing your badge system prior to deployment?

In late summer/early fall, we’ll be testing our badging system with our own students and opening it up to people who are interested in checking it out using an OpenID login. We’re hoping to get a wide range of users outside our program engaged in the testing so other programs can learn for themselves and them we can also benefit from their feedback. We’ll be communicating about this testing phase with the badges community well in advance. 

What factors are the success of your badge system contingent upon?

  1. Determining the right balance point for the social networking aspect of the system. How much feedback is useful? What types are most effective? How can we most easily moderate these processes? 
  2. Continued support within our institution, from administration and faculty. 
  3. Getting badges to be seen by their consumers (grad schools, employers etc.) as a desirable and easily discoverable commodity. 

What have you done, or do you plan to do, to evaluate your badge earner community?

As a formal academic program, we have evaluation mechanisms that predate the badges, but we’re also very interested in looking at the analytics to see what kinds of learning experiences are most valuable to our students, which students are most motivated by and engaged in the system, and how they are using the badges to communicate about their work. These evaluations will help us iterate on badging design as well as curriculum design for the whole major. One of our goals as a program is to employ an adaptive curriculum management model, and the data from this system could help us take that work to a whole new level. 

Please describe any impact your badge system may have already had on your organization and your learners.

One major impact is that our institution hasn’t traditionally been that active in thinking about how digital environments can support offline, experiential learning in meaningful ways and this project has started a dialogue for us around how our students can and need to develop digital literacies. Our badging system may help students develop those literacies, an more importantly, we think it provides solutions for an ongoing reflective practice and community of learning that is core to an integrative, outcomes-based and experiential learning curriculum, and is frankly very challenging to implement successfully because it's highly complex, individualized and resource intensive . It’s also got us talking about how we can use education technology in a number of contexts, badging being only one aspect of a broader strategy of connected learning. 
 
For our learners, who aren’t yet earning badges, we currently have a rubric-based competency assessment framework, where students check boxes to mark their progress and submit those forms to their professors. They don’t get to organize or view that progress across courses or over time, and that makes competency assessment as series of one-off experiences that lack relevance and coherence. The students are hearing about the prospect of replacing that system with badges, and are excited that they might be able to have a tool that is easier to use and provides a more connected and enduring picture of their achievements and competency development. They are also excited to be a part of a program that is innovating on their behalf. Even a simple awareness that we’re developing this kind of system seems to have encouraged them to think differently about how they reflect and capture value from their learning experiences, and what they might want to show to others from that portfolio of work. There are, of course, some questions and concerns about what this development means for them, and we’ll need to address that as they start using the system and earning badges.

How would you characterize the impact your badge system will have on the badge ecosystem?

Because of the way we’ve structured our thinking around curriculum design as it relates to badges, and because of our position on the curve of innovation and adoption within the movement, we believe we have significant opportunity to impact at least the higher ed ecosystem, and likely the broader badging community with our work.  Additionally, our position as a Land Grant University and our exploration of future collaboration and alignment with what’s happening in the K-12 space through 4H and other organizations, means that we’re already engaging in dialogues with potential for impact and the ability to highlight learning pathways across educational institutions and sectors. 

What plans do you have to scale your badge system?

All plans are contingent upon further resources, but we’re really looking at how to scale within our institution, how to create a design process that enables us to spread the development of the platform to other interested institutions that we’ve been in dialogue with, and we also envision working within the national network of similar institutions (INFAS) many of which support majors like SA&FS. We’d like to see if we can not only spread the adoption to other schools and programs like ours, but interconnect them so that students and faculty can share and provide a community of support around portfolio building and badge earning across institutions. We imagine, students in our programs going on to graduate work within that network, or perhaps taking key positions in the regions where these schools are located, and if they could foster those connections through this system, the impact and synergistic effects could be very big, for those individuals and for our field. 

How successfully are you getting institutional buy-in, or adoption from your learners?

We’d like to get blog posts going on HASTAC, through the UC Davis Academic Technology Services EdTech blog, The Wheel, and on our ASI website. We also plan to continue communicating developments on our project through press releases and ongoing conversations with reporters at the Chronicle of Higher Ed and Inside Higher Ed. We have not yet decided to what extent we’ll be using Facebook and Twitter to communicate about our badges system. 

No comments