Blog Post

Measuring outcomes in informal learning spaces


By Jessica Morley

Most people are familiar with the philosophical thought experiment that asks the question “If a tree falls in a forest, and nobody is there to hear it, does it make a sound?” Trying to answer this may seem trivial, a frivolous exercise that serves no purpose other than to stretch the reasoning abilities of the person answering the question. However, if the external layer of context, i.e. the tree falling, is removed, we are left with a far more fundamental question “if there is no evidence for something having happened, did it really happen?” In today’s data and outcome-obsessed world, the answer to this more fundamental question has significant ramifications for the ways in which the core elements of society operate.

One such core element that is considerably affected by attitudes towards evidence, is education. It is for this reason that I was incredibly excited to be given the opportunity to virtually attend an expert panel discussion, organised by HASTAC and the Futures Initiative, around the topic of. “Measuring what Counts: Credentials or Learning.” In essence, the speakers were examining the same philosophical argument but with a different context i.e. “If learning is not quantitatively measured has anything really been learned?”

The discussion was lively and highlighted a variety of opinions. However, two points, in particular, stood out for me. The first was that raised by the esteemed writer and education blogger, Anya Kamenetz, who argued that the more data we have access to, the less useful averages are. The second was made by Leah Gilliam, vice president of strategy and innovation at Girls Who Code, who highlighted the fact that learning is a social construct that happens in specific contexts and that, therefore, informal learning spaces can give those who have traditionally been overlooked in formal learning spaces, a chance to shine.

Both of these arguments resonated with me because of my own personal experiences with formal ‘measured’ learning and because they have important implications for AMMAl, the women’s personal development network that I manage on behalf of the social enterprise Chayn.

On teaching fish to swim, not climb trees

People often assume that because I attended a university that in many ways symbolises the traditional formal education system of the Western World, that I would be one of its main supporters; a spokesperson for standardised, exam-based education. They could not be more wrong.

School did not work for me.

I was almost completely deaf until the age of six and, as a result, some of the wires in my brain related to logical, mathematical thought disobeyed the rules and didn’t make the connections they were “supposed” to. Consequently I’m dyslexic and do pretty much everything back-to-front and upside-down. The teachers I encountered in the formal setting of school made no effort to work with me on this. Instead I was repeatedly told that I was not destined for any form of academic success and I believed them. Luckily for me, my parents didn’t. They knew I had more to give than the teachers assumed, but that it wasn’t being reflected in my grades because standard tests don’t account for different ways of learning and thinking. As the famous quote says: “Everybody is a genius. But if you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid.”

So, how did I counter this?

I spent my time outside of school in activities where the fact that I didn’t think like everyone else didn’t matter, in other words where swimming was more important than tree climbing. I went to performing arts school, played the piano (but never took exams) and joined a debating club.

Removed from the pressure of having to “conform,” I excelled, and my confidence grew. The irony of this is that it had a positive impact on my performance at school, so that by the time I was seventeen and starting to think about my life after school, I was no longer scared to put my hand up in class or stand-up and talk. My thought processes were still very different to those of my peers, but it seemed to matter less and I met an inspirational teacher who helped me to see that in reality, this could be an asset not a flaw. He encouraged me to apply to Oxford, arguing that because they put more emphasis on how you think, not just what you know, and your potential matters the most not your current abilities, I would be a good candidate. I thought he was insane but it turned out he was right.

When I got to Oxford, I had a serious case of imposter syndrome but having the one-on-one attention from tutors was a wonderful cure for this. I had one tutor in particularly, who was marvellously blunt and within a term of me being there told me “Your results on paper are never going to reflect your true capabilities because exams don’t suit you.” Knowing this from the offset was strangely liberating, it didn’t stop it upsetting me from time to time, but it meant that by the time it came to graduate I knew who I was and what did and did not work for me. I knew I’d never get a ‘grad’ job because I can’t pass standard psychometric tests but I was confident that there would be other opportunities for me.

This confidence I gained from being almost exempt from the constraints placed on individuals by the formulaic formal education system has stayed with me. This is why I am passionate about the ability of digital tech to transform the ways in which people learn. It improves access to learning outside of the school system and gives more people an opportunity to find confidence in their own abilities, when removed from a set structure.

AMMAL, is founded on this belief. The network provides women, particularly those that have not had much success with formal education, with face-to-face, or remotely-taught, training in tech skills and then provides them with access to a 24/7 peer-led support community designed to encourage them to use these new skills as a means of competing in a saturated job market. The learning is gamified, broken down into a mix of hands-on sessions and verbal ‘lessons’ but the atmosphere is relaxed, and all the emphasis is put on the qualitative impacts of the training day rather than quantitative measurements of performance. This is why we always ask how confident workshop attendees feel at the beginning of the day compared to the end of the day, how much they enjoyed the overall experience and whether they feel as though they could put the skills they have gained to practical use.

Our most recent workshop was on digital design. 100% of attendees left saying that they felt more confident in their ability to understand the principles behind design and to use digital design tools in ‘the real world’ and consequently 100% said that they felt as though they could put the skills into practice. To me this is invaluable, worth a thousand times more than a percentage on a piece of paper. It shows women that they can, and they should pick up new skills and consider careers in sectors that they previously thought were not suitable form them. In short informal education spaces, facilitated by digital tech, can breakdown barriers and give people the confidence needed to learn.

From this perspective, for me it is the learning itself, and the secondary effects of this learning, that counts. Yet the challenge is that employers haven’t caught up to this way of thinking yet, they still hire based on hard-evidence, wanting proof of skills-gained before offering someone an opportunity, focusing so much more on past performance than potential. This is enormously frustrating as, unless something changes, the full potential of the digital disruption of the education sector cannot be realised. To come back to the point that Anya made about averages being meaningless in a data-filled world, it seems madness to me that we can personalise medicine, diet, fitness regimes but we can’t personalise the way in which learning is measured. Why is it that we have reached a point of understanding that no two bodies are the same, but we treat minds as though they are?

This is why it was so encouraging to hear the panel talk about the development of digital badges and e-portfolios that reflect the fact that learning is often a process and cannot, therefore, be accurately measured in a one-off test. Ultimately there is a long way to go, but the fact that the conversations are happening is an important first step. I, for one, hope to be part of more and cannot wait to see what changes may be just around the corner.


If a tree falls in a forest but nobody is there to hear it, does it really fall?

Einstein quote about different animals getting up a tree

If we want to assess a student, don’t just assess what they can do today but what they can do tomorrow. Ongoing learning process is just as important

Anya Kamenetz

Looking for in the 21st century, people who are diverse in abilities and values, work well in teams, are creative. How do you get at those factors?

Testing reinforces notion in students that narrow type of achievement is important. Eliminates fun and creativity from learning.

Other qualities – social and emotional learning – are just as important for achievement

Anya: More data we have access to, the less useful averages are. We have ability to target strengths of different individuals. Automated platforms where you get video prompt and then you have to talk for 2 minutes. Purpose is that when you already have scores of exams, the individual answers and responses help to make sure students meet the standard (students that can’t speak English for example)

Bret Eynon

ePortfolio is not a technology but a practice that engages students, faculty, and staff. The technology is a part of it but is not the main thing.

Portfolios designed to help students develop a more personal identity as learners. Important for students to connect their learning with their learner identity over time, across semesters, across courses, across disciplines. Helps students share understanding of their learning with other students, faculty, advisers, etc. Help students develop a more personal sense of

themselves – personal self authorship.

Students assess their own learning and change through the portfolio. Faculty read and assess student work in portfolio as well as outside portfolio.

In programmatic assessment, the portfolio provides authentic artifacts of the learning process in each program – allows faculty to set goals and see how they were accomplished.

Leah Gilliam

Learning is something that is socially constructed – happens within a social context and can happen anywhere.

Do not love the idea of learning anytime anywhere, because it brings up issues of access. But learning is part of an ecosystem.

The idea that assessment can be build in is very important. You can use testing to measure particular outcomes, but idea of valuing what is really important to kids and having different contests to show what is important to them. Informal space also gives opportunity to see varying levels of experience amongst students. You want to be able to adjust based on what kids are interested in and have assessment able to reflect that.


How we learn in the 21st century is shifting from “issues of authoritativeness to issues of credibility” (Davidson & Goldberg, 2009, p. 27). Open digital badges will push us to define in greater detail what this means both in theory and practice. At the moment, badges present us with a design challenge to advance principles of credibility that we have yet to clearly outline. These principles are being embraced (if not exactly defined) in different fields like design and software engineering, where employers put less stock in schooled learning and traditional credentials, and reputation and evidence alone can be keys to advancement.

Cross-posted on Medium.

No comments