Some Things about Assessment that Badge Developers Might Find Helpful

I recently met with Greg Wilson, the founder of Software Carpentry  to discuss how to assess the impact of teaching basic computer skills to other scientists to help them manage their data.  Greg is as passionate about education as he is about programming.  We discussedAudrey Watters’recent tweet regarding “things every techie should know about education.”

 As an assessment researcher scrutinizing the funded and unfunded Badges for Lifelong Learning proposals, I thought I would try something more specific.  In particular I want to explore whether distinctions that are widely held in the assessment community can help show how some of the concerns that people have raised about badges (nicely captured at David Theo Goldberg’s “Threading the Needle…” DML post).  I have posted a longer version at Remediating Assessment.

While most of the proposals have multiple goals for their badges, there seems to be three types of primary goals for using badges.

1.     Showing what somebody has done or might be able to do (e.g, Badgework for Vets).

2.     Motivate individuals to learn or do more (e.g., BuzzMath).

3.   Transform or create learning systems (e.g., MOUSE Wins!).

These three types of goals appear to correspond with the three primary assessment functions.

1.     Summative functions (assessment OF learning.)

2.     Formative functions for individuals (assessment FOR learning).

3.     Transformative functions for systems (assessment AS learning)

Different assessment functions generally follow from different theories of knowing and learning, but these assumptions are often taken for granted.

1.     Summative functions generally follow from conventional associationist views of learning as building  organized hierarchies of very specific associations.

2.     Formative functions follow from modern constructivist theories of learning as constructing conceptual schema by making sense of the world

3.     Transformative functions follow newer sociocultural theories of learning as participating in social and technological practices.

These three assessment functions often conflict with each other in complex ways

In particular, summative functions often undermine formative and transformative functions.  This is because ratcheting up the stakes associated with summative functions (i.e., the value of the badge) often requires assessments that are “indirect” and “objective” like an achievement test.  As John Frederiksen and Allan Collins pointed out back in 1989, such assessments have limited formative and transformative potential, compared to more direct and subjective performance and portfolio assessments.

My point here is that badge developers should consider the various goals for their badges, and the assumptions behind those goals.  Failing to do so can create “wicked” tensions that are impossible to resolve.  This can be toxic to educational systems because stakeholders ascribe those tensions to other things (politics, laziness, culture, faddism, etc.).

In response to my first draft of this post, Greg summarized my point more succinctly and more generally:

People have different philosophical models of education (whether they realized it or not) and that is why they talk across each other so often.

Greg also inspired me to suggest the following contribution to Audry Watters top ten list of questions you can ask to find out if somebody really knows education, if you want to know if the now about educational assessment:

Do you understand the difference between summative, formative, and transformative functions of assessment and how they interact?

(A longer version of this post is available at Re-mediating Assessment)

Cathy Davidson

Tweets

Hi, Dan, Are you on twitter?  Your post is just wonderful.  I tweeted it and it's now been retweeted several times.  I'll add your twitter handle if you have one.  Thanks so much for this, so helpful!  Best, Cathy

 

Here's a sample, just off my HASTAC dashboard, not the the full list;

 

boothmelanie RT @HASTAC: New on HASTAC: Some Things about Assessment that Badge Developers Might Find Helpful ow.ly/1i8Bul 20 minutes ago · reply · retweet · favorite

HASTAC profile

HASTAC New on HASTAC: Some Things about Assessment that Badge Developers Might Find Helpful ow.ly/1i8Bul 22 minutes ago · reply · retweet · favorite

vpiercy profile

vpiercy Some Things about Assessment that Badge Developers Might Find Helpful | HASTAC: bit.ly/AyxbUL @alansgandy @bouligny 33 minutes ago · reply · retweet · favorite

dajbelshaw profile

dajbelshaw RT @CathyNDavidson: Some Things about Assessment that Badge Developers Might Find Helpful | HASTAC: bit.ly/FPmlso by Dan Hickey 42 minutes ago · reply · retweet · favorite

machtm profile

machtm RT @CathyNDavidson: Some Things about Assessment that Badge Developers Might Find Helpful | HASTAC: bit.ly/FPmlso by Dan Hickey about 1 hour ago · reply · retweet · favorite

nancyrubin profile

nancyrubin RT @cathyndavidson: Some Things about Assessment that Badge Developers Might Find Helpful | HASTAC: bit.ly/FPmlso by Dan Hickey about 1 hour ago · reply · retweet · favorite

CathyNDavidson profile

CathyNDavidson Some Things about Assessment that Badge Developers Might Find Helpful | HASTAC: bit.ly/FPmlso by Dan Hickey about 1 hour ago · reply · retweet · favorite

dmlComp profile

dmlComp Some Things about Assessment that Badge Developers Might Find Helpful | #dmlbadges #openbadges bit.ly/A4qrY4

kbjeffrey

A response from a techie

I had a bit more to say than would fit in a comment, so I created a blog post:  Some Things About Assessment that Technology is Changing

dthickey

Can data every be theoretically neutral?

Karen--

Thanks for you thoughtful post.  I read it carefully and look forward to more exchanges.  Here is a question I have for you?  You stated

Which theory of knowing and learning an assessment is associated with has a lot more to do with the Activity and the Reporting than it does with the Data.  Badges are a mostly just a way to collect the Data.  The activities performed and the information shared will determine the effect on the learner.

 

I completely agree with you that designers tend to be more explicity about their assumptions about learning when designing activities and creating systems for reporting how users engage in those activities than when they create assessments that attempt to capture evidence that this learning transfers to some other context.

But that is the problem isn't it?.  People often use multiple choice assessments to assess open-ended constructivist learning, and are then surprized when the learning does not transfer.   Or they give open ended problems that are so similar to the learning activities that they only capture (and therefore encourage) very shallow learning. 

Just because people don't articulate the assumptions behind assessment practices, different practices still embody different assumptions about learning.  These assumptions in turn support some evidence of learning transfer over others.   And this is only talking about assumptions regarding summative assessment functions.  When we bring in formative and transformative functions, it gets even more complicated.  This is because badges are no  longer just a way to collect the data.  They become central elements of formative functions for motivating engagement within systems, or transformative functions for changing or creating learning ecosystems.  In this case, the assumptions that are embodied in whatever data or evidence behind that badges will be a major factor in whether those formative or transformative goals are accomplished.  If those assumptions are tacit and conflict with those goals, it is more likely to create wicked problems that will undermine the entire effort, IMO.

Thanks, and looking forward to more exchanges, and hearing more about your work.

Dan Hickey

kbjeffrey

Agreed, Data is never neutral

Dan -

I agree with everything you say in your comment.  The point I was trying to make is a bit different.  I think what I was trying to say is that badges can provide a way to structure discussions between assessment researchers and developers in a useful way.  It just isn't practical to expect that everyone who will be developing badge systems will be able to have deep understanding of all the issues that effect assessment.

To try to show you what I mean, I've written another blog post:  ePortfolios as Badges - A Badge System Design for Learning by Creating

Thanks for your response.  I'm look forward to hearing what you think of my presentation of your ideas about assessing portfolios.  I hope I at least got some of it right.

Karen