Blog Post

The difficulty in rating transformational experiences

The difficulty in rating transformational experiences

  "Five stars!"

“What a great MOOC!

Highly recommended!"

Having just searched for Android apps to support Coursera courses (for use in #FutureEd), I can understand the usefulness of a rating system for software. Most of us academics are also used to some form of teaching and course evaluations that are required by our institution to ensure that we are doing a satisfactory job (at least from the students' perspective). The results of these evaluations are often reduced to a single number, to enable the administrative system to deal with it — and with us. This is not unlike the numerical grade that is recorded as a result of the 'A', 'B', or 'C' that students are awarded, as a crude way of quantifying their performance on an assignment or course. Academic staff and students both quickly learn how to game the system.


It would be difficult to reduce an individual's experience in a MOOC to a single, quantifiable indicator or score. Unlike a traditional course, most MOOCs, like #FutuireEd, cater to all sorts of different people who participate in a course (or part of a course) for their own very different reasons. Judging from the Twitter stream (and from discussions with a few others on my campus who are doing #FutureEd), many are not completing the assignments and are picking and choosing what resources to read and what forums, if any, to participate in. There hasn’t been much research published yet that deals with MOOCs from the students’ perspective. A few researchers, like Jenny Mackness, are investigating how students navigate in, and make sense of, a MOOC. George Siemen’s is leading a team at Athabasca that is engaged in a meta research project, investigating who is researching MOOCs, what they are focussing on, their discipline, methodology, etc. 

It is understandable that highered institutions, employers, students, parents and others want to know know that a course has been “Quality Assured”. And they have good reason to want to know how well their work stands up in comparison with others. But we need to be wary of thinking about learning as an activity centred around a course that can be appraised, valued, judged, and exchanged (either for financial or social capital). I am reminded of a quote by Stephen Downes (who, with George Siemens and Dave Cormier, offered the first “MOOC” in 2008, an ongoing discussion of Connectivism and Connective Knowledge):

“The product of learning is not knowledge; the product of learning is a transformed learner”.

Personal transformation is a very hard thing to measure, value and rate. Yet it seems to be a strong personal motivator for many individuals who take free online courses. 

Many people who participate in MOOCs are also likely to read the newspaper and listen to public radio (although I have no research to back that up). One major attraction of synchronous, social MOOCs is the opportunity to engage with others around shared interests and develop a sense of community. Educational resources, and courses, offer fuel for the fire. But it’s not the wood, and it’s not the fire that enables personal transformation — it’s the choir; the out-of-body experience one gets when you feel you are taking part in creating and appreciating something beautiful with others. Such an experience is its own reward. 

We rate hotels, music, live performances, movies, etc., so that others are able to make an informed decision about how to best invest their time and money. Rating and reviewing MOOCs seems like a sensible thing to do for similar reasons, and it would not be surprising to see such a practice develop. However, unlike a hotel or restaurant franchise, a living, changing, organic learning experience cannot be packaged, replicated, and sold to consumers who are looking for a satisfying (and predictable) product or service. It can't offer the same experience to the same person twice, and one person's experience may not be a good indicator of the experience another person will have. A transformational learning experience is like a good pot luck dinner party — you might have had the time of your life, but it can never be repeated. Every conversation is different, and your experience depends in large measure on what you bring to the party. 



Mark, I really appreciate your moving this from our Twitter exchange where we were having acronym fun, to me recalling a post I had done, whimsically, back in 2012, about MOOCE (Massive Online Course Evaluations), a funny acronym that allowed me to show a photo of my fav animal, the moose: And it also helped me make a quip that has been picked up a lot in the media, that we need a "YELP! for MOOCs."    

Your comment was so thoughtful that I asked you to move it to its own blog and I would love to have the current MOOC students see it and also to ahve anyone who comes on to find it more easily.   We are in such an era of numerical mania, where "standards" equals a number.   How do you rate a transformational experience?  Why do you need to rate one?   Sometimes the testing is fine, but other times it is absurd, a reduction of the moment of transformation into a mark in a bubble.  Really?  What an odd way we humans sometimes behave!  if my baby suddenly stands up and starts walking, I don't need to give her a standardized test to see if she can walk.  She. Is. Walking.  

I love your post for so many reasons that all come together succinctly.   The amount of research on "educational transformation" that is actually frm the students' point of view is just pathetic.  One reason that I have "Who's Your Favorite Teacher and Why?" videos interwoven throughout the "History and Future of Higher Education" video is, without being didactic, I want people to think about what kinds of teaching was transformative in their own lives.   Before they pontificate about what is or is not good for others or for everyone, I wanted to bring the conversation down to that intimate level:  if a teacher helped transform the way you think about the world, what did they do that made that transformation?   what did you carry away from that transformational moment?   why?  what did it mean to you then, what does it mean to you now? 

Implicit in the question is:  could it be replicated in a MOOC?   If so, what part is important?   I've been joking that some of the amazing dialogues on line feel to me like "Intellectual"   That is, people are falling in love in the best possible way---finding others with whom they can talk through ideas on the most profound, energetic, deep, detailed level.   In some of the conversations online, there is the kind of electricity that comes from that loving teacher you remember decades later, that sens that finally there is someone in the world who "gets me," who understands who I am enough to engage me deeply and thoughtfully, whether or not we agree on the conclusions.  

It is amazing and humbling to watch and is not what I expected.  Can it be measured?  Not even close.   Is it happening for everyone?  Not even close.   Do I care?  

Hmm . ..    I"m not sure.   I've only interviewed one person who said,"I don't have a farvorite teacher.  I never had one person in school who understood me."  And then I said, "Well, some people are finding teachers that they hated and who they learned from negatively, saying 'I will never do x,' so, for them, that resistance is a favorite and definint experience."   This person answered, "No, not even negative.  There was not a single person in a teacher role who has had an impact on me."  This person is well educated.    That comment has stuck with me more than any other of the dozens of people I interviewed. 

And that's the follow up to your wonderful post.  Sometimes transformation does not happen.  You write, so aptly:  "A transformational learning experience is like a good pot luck dinner party — you might have had the time of your life, but it can never be repeated. Every conversation is different, and your experience depends in large measure on what you bring to the party."      

So now you have made me wonder:   what about the person who has never had a transformational learning experience?   Is it about the person, the teacher, the learning opportunities---or pot luck?    Thanks so much for making this your first post.  I hope it won't be your last.  I can't stop thinking about this deep, important contribution to all our learning together.    


First, let me take exception to Cathy's observation that "The amount of research on "educational transformation" that is actually frm the students' point of view is just pathetic." What we now called "student centered learning" was the primary focus of Socrates, and, for a very long time (like 1500 years) the primary focus on educational research: it was Montessori and Dewey, among many, many others. And it anticipated the wealth of media we now find so new and "unprecedented."

Were it not that these comments sections have indirect editing that limits links, I'd cite some more examples, but my advice is to "google" sources ranging from Project-Based Learning to correspondence courses, from autobiographies of "how I learned" to the Secretary's Commission on  Achieving Necessary Skills of the last century. The skills measured by tests are both real and, in large part, derivative - they ask how much of what the teacher directed actually stuck in the brain of the student. Some, admittedly more sophisticated (and thereby more expensive) tests build on essays for more subtle conclusions, but not many, and they are really unusual among the testing companies.

The kind of transformation you and Mark want to address gets to the real purpose of education itself: how do we grow up, and how does our grownup self reflect the culture, traditions, and the personalities of our inspiration? There really are many, many interesting ways to "measure" these transformations. Some are portfolios - and they can be either open or close ended (google AAEEBL for some examples and a surprisingly rich literature). Some are more structured, with open ended sections but a fixed set of categories. The ones I like reflect that SCANS and employer-career related framework with categories like reliability, creativity, collaboration, inquiry, etc. I used those in creating an ePortfolio project in Somerville, Mass, high school, and the result was teachers who were astounded at how much their kids actually learned, and kids who realized they didn't really understand many ideas until they taught others to use them. Those are the kinds of "transformative metrics" that can be made quite concrete - Dewey did it, as did Montessori and teachers from Greece through Horace Mann.

Those transformations can even be quantified, and don't all reflect a unique identifier. Smart students can learn from their overall environment - it's not all great teachers, since many times it's the atmosphere, the sense of time, the architecture and the walk between classes.

Now, applying these transformations to measuring MOOCs is like matching philosophy with number theory. Sure it can be done, but only sacrificing much from both sides. First it takes a clearer understanding of what that MOOC means to the student: for some it's a course, for others it's interaction (like this note), for still others it's the equivalent of a textbook, with wisdom mixed with "trollishness" and the challenge is to learn the best from those who may not always speak with wisdom. With that in mind, I would not suggest a Yelp for MOOCs, since your Yelp might look at the whole course, but mine might be inspired by a single class by a brilliant teacher. They'd both be full of stars, but...the relevance of those stars subjective at best.