Who were you addressing with your design objective?
The Art and Science of Vision team design objective worked to address San Diego-area learners, ages 13-18. The team specifically sought average students--not failing, but not entirely sure of their future academic or career paths.
What are the three essential questions the field needs to answer to move learning playlist design and implementation forward?
Based on the team's experience, in order to move playlist design and implementation forward, the field needs to answer:
- How will young learners find out about online learning playlists and stay motivated to complete them?
- Will there be a universally agreed-upon worth of each digital badge?
- How will the creation and maintenance of learning playlist experiences be sustainably funded?
Have the goals for your project changed at all throughout the design process?
As in many projects with a deadline, certain lofty goals had to be revised as we neared the end of the grant period. While we hoped to speak more to employers and see what they might value as part of learning experiences and badges, we were unable to pull such a group together. We also developed ideas outside our initial proposal scope, such as creating collectable physical badges in the style of popular enamel pins and developing a "capstone" that connected all three of our playlists, that time restrictions prevented us from implementing. Overall though, we stayed true to and met all our initial goals.
What did you learn through the design process? What would you do differently if you were to start over?
We went into the project with existing ideas about content. When we began getting input from young learners, we asked them about the form that content could take. While these were important questions, we failed to take into account the types of information learners might find relevant and engaging. If we started the project again now, we would engage learners earlier in the process, giving them say in the content, and not just the delivery. We think this would help us find language to make the type of content we want to share more relevant to learners.
What are the 3 most important things about designing your system or solution that you would share with another organization just getting started?
The three most important things we would tell other organizations just starting are:
- Throughout the process, think about how you might build sustainability into the project;
- Get as much feedback as you can, as early as you can, from the learners that you want to use your platform;
- Get input from other organizations who are doing, or have done, similar projects; they have insights about aspects of the project you may not even know about.
What is left to do? What is left unanswered? What might help you continue to succeed?
We are happy to say we accomplished everything we set out to do, in terms of creating and testing learning experiences. We have not answered how to get learners familiar with and using the platform without adult prompting, and how, financially, to cover staff time as they continue creating new and monitoring existing content. We have some ideas about how this might be done, but must seek additional funding opportunities to continue this discovery process. Outside of funding, we have found a lack of understanding about digital badges, and believe that spreading the word about them would be one of the best ways to get widespread buy-in.
What are the three essential questions the emerging field of connected learning playlist design needs to answer or make happen in order to move your work forward or scale it?
In order to best move forward with our connected learning playlist work, the field would need to answer these essential questions:
- How will the creation and maintenance of online learning playlists be funded?
- How can online learning playlists better be incorporated into the work we already do and the needs we have already identified?
- How can we work towards making digital badges more universal and meaningful?
What parts of the playlist platform technology are working well? What limitations are you experiencing?
Throughout the process, we have seen the platform technology evolve; many items that started as limitations became strengths. We appreciate that we can now edit certain parts of published playlists, as they make iterative, user-centered design easier. Perhaps the biggest remaining limitation is the evaluation experience. Evaluators do not receive submission alert emails, which limits their ability to respond quickly. Learners only receive feedback via email, which limits their ability to receive feedback in real time. Streamlining this process could make a substantial improvement in ease of use.
While user-testing, we ran into an issue not with the platform, but with the ability of local schools to keep up with the technology. With eight learners attempting to use the platform on the high school's WiFi, loading times proved painfully long, causing the learners to lose focus. The WiFi was even more problematic when students went to play the videos that, based on their early feedback, made up the bulk of our content. During this testing session, we worked around the platform to make the XP content available to the students. While LRNG may have trouble directly addressing school WiFi, perhaps some sort of caching system could be explored. Internet speed concerns should also be considered when recommending content and contexts.
In our attempts to provide translations, we met a few limitations. In San Diego, a binational city with many Spanish speakers, bilingual content is vital. In our initial proposal, we noted our commitment to include all materials in English and Spanish. We did this, but found no easy way to incorporate our translated text into the platform. In the end, we created two sets of playlists, one in English and one in Spanish. This was a good solution, but the surrounding content generated by the platform remained English. In the future, we would love to see updates that incorporate multi-lingual functionality.
There were also a few much smaller limitations. One was the inability to sort and search in the content creator. After creating numerous XPs, it became difficult to find any particular one. Another was that badges did seem to register being saved. Throughout our use of the platform, the site always asked if we would like to proceed and lose badge content, when really the item had been saved. Again, these are small items, but taking note of and fixing these could go a long way towards increasing usability.
What have you done, or do you plan to do, to evaluate the efficacy of your learning playlists in your community/communities?
We undertook an iterative process with our project, testing with our audience as much as we could throughout the process. We began by hosting a series of focus groups to learn about teens' preferences and needs. The biggest takeaway from these groups was that many teens prefer to learn through watching video. Taking their input seriously, we decided to present as much of the playlist content as we could in video form.
We were lucky to have a high school intern on staff during our creation of playlist content. This meant that we had a teen voice to turn to even between focus groups. Our intern helped us on video shoots and in post-production, providing input on visuals, music, and sound effects. She also did pilot runs of all the XPs, helping us identify areas where instructions were unclear and giving honest feedback about the quality and fun of each different activity.
When we brought our final draft of content to students in their classroom, we ran into the technical issues with school WiFi detailed above. Despite the trouble with the videos and the platform, students still expressed that they enjoyed learning through the various activities. Their creativity shined as they created their own stop motion animation videos and optical illusion photos. It is certainly hard to beat moments when teens said, "Our math teacher is going to be so excited that we finally found a use for parabolas," while making a camera obscura, or, "If school was like this every day, I'd actually want to come."
That being said, our group of eight testers needed serious guidance to stay on task and on track. The classroom setting and WiFi problems contributed to a teacher-student feel, rather than encouraging self-directed learning. If possible, we want to test with youth in different environments, and are considering further tests with summer camps and interns.
What are the 3 main challenges to widespread adoption or scale of these learning playlists for your organization?
The main challenges to widespread adoption of learning playlists for our organization include:
- Convincing teens, parents, and educators that this new tool is worth incorporating into their already busy schedules;
- Developing a wider understanding of the model so that our educators can find natural connections with the work they already do;
- Convincing different stakeholders of the long-term value of digital badging.
What plans are in the works, or do you plan to put in the works in order to sustain your system?
Our plans for sustainability are based around incorporating playlist content and LRNG more thoroughly into our existing projects and programs. This includes considering the LRNG model when working on our own collections discovery playlist idea and recommending the platform as an option when museum partners come to us looking for ways to present educational programs or materials. We are particularly excited about the potential to fit the platform into our fledgling program of school telepresence museum tours. Teachers taking part in this program noted that students needed something to do during the tour. Answering questions through LRNG may be the answer.
How can we use the platform to improve field trips? Summer camps? After-school programs? We are still figuring out what this might look like, but now that we known the platform and seen what other organizations are doing, we get new ideas every day.
How are you getting institutional buy-in, or adoption from your learners or other stakeholders?
To get institutional buy-in on this project, we relied on existing partnerships that we have built over time. As a technology services collaborative, we rely on partnerships for everything we do. Together with our partners, we know we all have our audience's best interests in mind, and are willing to try new ideas and programs that others propose.
As for learners, we have found the best way to get them involved is through their schools. Food and financial incentives have also proved valuable to these audiences, but we are interested in new and better ways to get learners using the platform on their own.
What outreach strategies will you employ to communicate and support your playlists?
How are you going to, or did you, incorporate the feedback from the workshop into your plans going forward?
The workshop was extremely useful, allowing us to hear what others were doing with the platform and the challenges they faced. The workshop taught us that it helps to link videos from Google Drive, since many school firewalls block Youtube. We learned about the importance of a quick turnaround on feedback for learners. We asked other groups about if and how they approached translations. While nobody had a solid answer, we connected with another group who was also adding translations, and have continued to exchange ideas. These lessons and connections have helped us think broadly about how we might use the platform in the future.
Have you budgeted adequately to materialize the design work you imagined?
The work we proposed and completed fit well into our budget.
Is there anything else you would like to share with us about your progress?
We are honored to have been involved in this exploratory phase of online learning playlists and digital badges. This type of interdisciplinary learning fits well with the structures and resources of the museums in Balboa Park. We have seen so much progress in the short time of our involvement, and are excited to see what the future holds these promising new learning experiences.