Project Q&A With: American Graduate Let's Make it Happen

Project Q&A With: American Graduate Let's Make it Happen
As part of the larger American Graduate Initiative, the Corporation for Public Broadcasting identified three programs through which to explore the potential of badges for learning during this grant year: PBS NewsHour Student Reporting Labs, Roadtrip Nation, and Storycorps U. Though geographically distributed and vastly different in their curricula and specific program strategies, each of the participating producers shares similar challenges and population characteristics. Each seeks to leverage the resources of public media to help students find relevance and confidence through storytelling, relevance that may help propel them to high school graduation and a successful post-secondary education or career pathway. For some, this may involve crafting a narrative for the purpose of reporting the news. For others it may be giving a personal story of struggle voice. All programs serve at-risk high school students from largely underserved populations and struggling schools.
As we began the grant year last summer we chose to first unravel the various systems in which we would need to deploy assessment strategies and badges, to establish a common vocabulary for discussing learning within the context of badge systems, to identify rubrics by which each producer's instructors define proficiency, and to clearly define the strategic goals of each community of practitioner and learners and how they might relate to expansion across the much larger American Graduate Initiative community. Since each group also utilizes their own technology system (and sometimes third party platforms including Voice Thread, Tumbr, and others) we simultaneously discussed how and when the platform-agnostic authentication, analysis and assessment functions of the Leverage technology platform might best and most effectively be integrated. 
Our first questions for all producers were the following and informed our technology strategies and the articulation of program goals:
1. How do you define and recognize proficiency? (which draws out the realities of what standards and rubrics are utilized, how data is collected and stored, what proficiency looks like within the context of each community of users, and what evidence remains to substantiate claims)
2. How and where do your learners demonstrate effort? (which draws out format for evidence, e.g. work products, technology resources available, e.g. websites with use accounts, technology and web availability)
The dialogue surrounding these questions helped team members distill the following large program goals and technology needs:
Goals identified by each producer include:
Roadtrip Nation: serve as an example and leader in the American Graduate community, maintain engagement
Storycorps U: facilitate students in tracking personal progress, create space for peer to peer feedback, map to shared standards
Student Reporting Labs: increase student engagement, improve usage of collaborative space, consolidate work products and feedback, facilitate students in tracking personal progress, help teachers articulate 21st century skills attainment and levels of proficiency demonstrated.
Technology analysis, needs and resource allocations include:
Roadtrip Nation: RTN has it's own user authentication system and community website through which engagement-type badges have been issued for a year. With the RTN team, we have worked to identify how "meta level badges" might be used to more clearly represent the life and 21st century skills learners explore during execution of the curricula. After considerable conversation and internal board discussion, we have arrived at the conclusion the program and alumni are best served through the award of an RTN "scholar" badge representing completion of a year's course and an RTN "alumni" badge provided to those who, post completion and graduation, return to the program, participate in community forums, and further the mission.
Storycorps U: Storycorps U does not have a website that supports user accounts. All students complete their work products in Voice Thread, a third party interactive media platform. The decision was made to work with VT to develop a strategy allowing them to be a "badge displayer," and using Storycorps as the first pilot of how an educational program incorporating portfolio elements might use badges for assessment and recognition. Storycorps also managed their own internal curriculum expansion and revision cycle this year while simultaneously Voice Thread revised their backend infrastructure and front end framework. All teams are working toward a Fall 2013 launch of badge display within VT, a php-based backend system using Twitter Bootstrap's frontend framework. In addition, Storycorps has a Wordpress public website in which we will automatically push badges for display in the aggregate at the school program, rather than individual, level. Leverage serves as the communication bridge and data and badge repository between both.
Student Reporting Labs: SRL had a Drupal-based Open Atrium community site with which they experienced little community uptake and much resistance. They also utilized ftp sites, emails, Tumblr, Facebook and other social media sites and technology devices to collect student work. Early on we recognized that not only did the wide range of systems make it exceedlingly difficult for program administrators to easily support existing programs, as well as plan for growth, but it also made it unreasonably difficult for teachers to progressively track student effort and organize work products. Students also found it challenging to track their own progress. We formulated as strategy for creating a badging and assessment layer on top of Open Atrium that can either be tied directly to the framework or sit independently (so that it can be deployed for other American Graduate programs not using Drupal). The new layer eliminates the need for email file transmission, allows teachers to manage an infinite number of groups and students, creates total data and progress transparency, includes tools for reporting, assessment, student reflection and recognition not limited to badges, incorporates a library of content to encourage community, information sharing, mentorship and more.

What are the 3 most important things about building a badge system you would share with another organization just getting started?

1. It’s not about the badges: Badges are an effective means by which to represent and organize accomplishment by an earner, but they do not, in and of themselves, act as an assessment. They are the culmination and distillation of a process that must involve instruction, demonstration of learning, assessment, and data collection. 
2. Badges do not a community make: Badges for Learning do not make learners become engaged if they are otherwise wholly disconnected. “Turning badges on” does not create an instant easy solution to learner engagement. They may create a meaningful bridge between content and learning, however, and help earners develop a sense of personal reward, confidence, and connection to the learning process. This doesn’t happen in a vacuum, though. Trainers, teachers, and peers can’t be separated from the process and must be incorporated into an overall strategy.
3. There is no number of badges that is the “right” number: A system with Badges for the recognition of authentic learning should be grounded in a fundamental understanding of how a population of individuals are instructed, scaffolded and assessed. There is no “number” or “type” of badges that can be standardized or generalized across communities of learners, and a decision on what to “badge” and when requires careful consideration and discussion. The right badges for learning are ones that articulate the instructional goals of an organization, represent a process, act, or skill that can be understood both inside and outside of the organization (respectively for community-level and externally-sharable badges), and that are meaningful to the personal growth of the earner, however that may be personally defined.

Who were you addressing with your badge system design?

Each producer works with opt-in student populations. Involvement is facilitated, curated, and not open to a general public community. Badges were designed specifically to address the aesthetic branding and task-level requirements of each program and with particular attention to engaging urban middle and high school students. 

What were your initial goals for the badges? Did those goals change at all throughout the design process?

Our goals have remained the same since the beginning due in no small portion to how rigorous our initial process was. To recap:
Goals identified by each producer included:
Roadtrip Nation: serve as an example and leader in the American Graduate community,  maintain engagement
Storycorps U: facilitate students in tracking personal progress, create space for peer to peer feedback, map to shared standards 
Student Reporting Labs: increase student engagement, improve usage of collaborative space, consolidate work products and feedback, facilitate students in tracking personal progress, help teachers articulate 21st century skills attainment and levels of proficiency demonstrated.
While our goals remained constant, our design decisions and methodology for how best to deliver on those goals did change mid-program year or at least one producer. We did learn that specific tools for managing groups and providing feedback were not entirely what Student Reporting Labs teachers wanted and needed. We had to rebuild their system, adjust pilot expectations, and start from scratch. They wanted greater control, more student reflection as a way to assess learning, more socialization opportunities as a way to further student outcomes, and badges to distinguish teacher effort.
What types of badges are you using (participation, skill, certification, etc.)? Are there levels or pathways represented in your badges? If so, please describe.
All producers are employing badges that represent concrete activity and the disaggregated task-level. Students have to complete an identified task that has a curricular tie in and observable result to earn a badge. However, some are very simple to complete and serve as low-barrier engagement tools. For Storycorps U and Student Reporting Labs, work products are heavily currated and teachers / administrators are commenting proficiency levels (and possibly domains explored) into the final culminating badges.

How were the criteria for the badges determined. What pedagogies (if any) informed the learning and badge system design?

A case can be made that each producer, though very unique in their mission and educational content and requirements, does employ a single framework (Common Core) on a single pathway (the production of one or more viewable, concrete media products). These were heavily taken into consideration when deciding what badges would be sharable outside of silo'd communities. We are still engaged in the discussion of how at the task and aggregate level badges earned across programs might bubble up into 21st century skills badges that might be endorsed at the American Graduate Initiative level and a common currency and value across the thousands of producers and education providers in that larger community. Because Leverage is an infinitely scalable assessment and analysis platform, we know that, if and when the time comes, we are well-positioned to support that effort. 

What are 3 things you learned about badge system design? What would you do differently if you were to start over?

Our lessons learned are not so much about "badge systems" but more about the realities of assessment, on-boarding of communities, and the readiness of institutions to take on the tough job of technology integration and dissemination.
1. Never create a new process or system without first determining the emotional readiness of your users to begin what will be a long process. (i.e. Are your end users capable -- from a time and knowledge standpoint -- to try something new? Do your administrators know it?) Learn where people are first and manage expectations and deliverables from there. 
2. Never design a system based on the beliefs of program administrators about how end-users navigate content and requirements. (i.e. Never ask a principal how a teacher teachers -- ask the teacher. Never as a program supervisor how a case work works with cases day to day.). Ask the person who will have fingers on a keyboard, hands on the shoulder of a learner before you code a single thing or design a single badge.
A distillation of these two is: Identify the PROBLEM, then create the SOLUTION. Don't presume or try to create a tool just to address a perceived need. Uncover the root of a challenge, a roadblock, a frustration and determine how assessment and engagent and feedback may be part of a real remedy and catalyzing change.
1. Recognize that in technology dissemination, you must train the trainers first, and well, with as much hand-holding and scaffolding as they need if you hope to build real engagement.
If we were to do this over again, I would do point #2 differently and, if grant execution allowed it, would begin work in the Spring so that system alphas could be ready for summer teacher training and Fall implementation. To begin on any other schedule is an uphill battle in education.

What is left to do? What is left unanswered? What might help you continue to succeed?

SRL is a fully-launched whole assessment, community, and badging system. RTN will implement their new badges for summer messaging and Storycorps, through Voice Thread, will launch in time for August teacher training. 

What are the 3 main challenges to widespread adoption of your badge system for your organization?

For each producer, our main challenge has not been interest in badges, a belief in the importance of assessment, or any technical impediment other than limited resources. Our challenges are more human in nature:
1. Mapping to the academic calendar of our adopters so that we can train and onboard them when they are willing, ready, and able.
2. Creating a relationship of trust with teachers so that they feel safe expressing concerns and articulating any problems or questions they have.
3. Finding, integrating, and managing content that may or may not already exist such that systems and badges that are, by design, curated so as to maintain the highest level of quality and integrity, have enough features and content that is learner-facing and self-paced to encourage adoption and engagement.

What is your badge system testing strategy? How have you or will you be testing your badge system prior to deployment?

The Leverage platform is currently and has been simultaneously used across business, education, and government educational learning systems, simulations, and games for over eight years. We are confident in stability of the system, its ability to manage highly granular data at very large scale in distributed populations and filtered through unlimited assessment rubrics and rules. The user interfaces for each program, however, are all customized as is the messaging and training strategy. Those we constantly tweak and adjust through each conversation and through communication with groups that have piloted features this academic year. The testing process is ever-evolving and iterative. The commitment to task-level, individualized data collection and feedback mapped to established rubrics, however, is a fixed foundation.

What are the top 3 factors that the success of your badge system are contingent upon?

1. Effective training.
2. Engaging content.
3. Open dialogue with learners, earner, and implementors.

What have you done, or do you plan to do, to evaluate your badge earner community? What will you do with the results of your evaluations? What research, if any, will be based on these evaluations, and do you plan to publish the outcomes?

Other than the process we have outlined, we have no grant resources available to allocate to formal research. Independently, however, the Corporation for Public Broadcasting as funded a 3-year development, scale up and research effort for Student Reporting Labs to kick off on July 22.

Please describe any impact your badge system may have already had on your organization and your learners.

Particularly within the context of the Student Reporting Labs community, just discussing badges and implementation has helped program administrators to understand more deeply how their curriculum is being used within school programs. This enhanced understanding led us to make fundamental changes to the way the end-user experience was structured. It has also provided a focal point for discussion among other American Graduate Initiative producers and within the Corporation for Public Broadcasting, as well. This may, in fact, be the greatest value this academic year: badges gave a reason for discussing assessment, data, technology, and synergies across educational programs and a tangible symbol around which to shape conversation.

How would you characterize the impact your badge system will have on the badge ecosystem?

Perhaps our greatest contribution to the larger community may be in the diversity of situations we’ve had to troubleshoot this year because of the varying training and technology needs across partners. We’ve traversed low and no technology situations, multiple standards frameworks, users falling in and outside of COPPA and FERPA definitions, and a vast array of topics around how to best and most seamlessly collect data during synchronous and asynchronous educational experiences and on and offline activity.

What plans do you have to scale your badge system?

The underlying systems for the badges (Leverage) is infinitely scalable and can support program growth in any ways producers feel exemplifies their mission. For Pragmatic as a technology partner, we have actively presented and discussed our experiences and the potential of the technology to facilitate learning pathways to other producers within the public media community and within other organizations including the Clinton Global Initiative America.

How successfully are you getting institutional buy-in, or adoption from your learners?

Support across producers and within CPB has been positive and encouraging. We expect it to grow.

For those who want to follow the development, implementation, and adoption of your badge system, what social media sites will you be posting updates to?

For Leverage, we are always happy to share outcomes and lessons learned personally. For outcomes and progress within each producer’s program, news and results may be found on their program sites when they feel it beneficial and appropriate. We hope that there may also be public-facing information on the American Graduate Initiative website in Fall 2013.

No comments