Grading 2.0: Evaluation in the Digital Age

Grading 2.0: Evaluation in the Digital Age

In this HASTAC forum, three Scholars invite you to consider evaluation and assessment in the face of new forms of digital media, new kinds of skills and technologies, and the ever-changing landscape of education and academia.

Grading 2.0: Evaluation in the Digital Age
 
As the educational and cultural climate changes in response to new technologies for creating and sharing information, educators have begun to ask if the current framework for assessing student work, standardized testing, and grading is incompatible with the way these students should be learning and the skills they need to acquire to compete in the information age. Many would agree that its time to expand the current notion of assessment and create new metrics, rubrics, and methods of measurement in order to ensure that all elements of the learning process are keeping pace with the ever-evolving world in which we live. This new framework for assessment might build off of currently accepted strategies and pedagogy, but also take into account new ideas about what learners should know to be successful and confident in all of their endeavors. 

How do we better align grading and assessment techniques so that they are more in line with how students learn today? The traditional 'teach to the test' evaluation paradigm continues to produce a classroom experience that focuses on specifically 'testable' results. That testing paradigm is also disconnected from all of the creative, production, remixing, and networking skills that students are developing through their everyday engagement with new media. Another issue is that the traditional assessment system tends to measure students individually and via multiple-choice and written-response questions. As teaching practices evolve to include more team-based projects that involve the use of smart tools to solve problems or communicate ideas, it will become increasingly difficult to assess students in the traditional ways. Furthermore, current widely-used tests are not designed to gauge how well students apply their knowledge to new situations.
 
In addition, how can digital media be used to develop new grading and assessment strategies? For example, last year, Cathy Davidson posted a blog entry here on HASTAC, called How to Crowdsource Grading. It outlined her course and the development of a new grading rubric, and student assessment techniques, to better work with her students in the classroom and in their course blog. The course was built around the idea of self-motivation, peer-to-peer review and a very clearly outlined contract for students on what each grade represented. This post attracted lots of commentary -- prompting her to write a response, called Crowdsourcing Authority in the Classroom. There are hundreds of other experiments such as these around the country, in elementary schools, high schools, college classrooms, corporate environments, and even feedback systems for nonprofits and municipalities. And there are more and more researchers thinking through these issues. There is clearly a great amount of interest in developing new technologies, and new forms of pedagogy, to better reflect grading, peer interaction and learning in the digital age -- help us think through these questions, experiments and strategies!

How to grade, assess, teach, learn and structure the learning experience for students in the digital age?
Many interesting projects are working on this question, and we invite you to share others with us below. For example:

- The Learning Record, a portfolio-based evaluation system designed to emphasize student learning, not product-based outcomes
- Nils Peterson and his colleagues at the Center for Teaching, Learning, & Technology (at Washington State University) have been working on developing new assessment strategies and forms of classroom engagement
- Pecha Kucha in the classroom - reframing the presentation from the unstructured long-form speech to the conversation-starting breakdown
- Digital Youth Research was a 3 year project to investigate how kids use technology and media in their everyday learning. They have reports available on their site, and the group recently published a book, Hanging Out, Messing Around, and Geeking Out
- Re-mediating assessment, a blog considering participatory assessment models in education, authored by Daniel T. Hickey, Michelle Honeyford, and Jenna McWilliams (Indiana University).
 - The DML Research Hub, funded by a MacArthur grant, is supporting two projects. One, lead by Mimi Ito, is called Distributed Learning Research Network, and works on distributed learning that happens in social environments. The other, lead by Joseph Kahne, is called Youth, New Media, and Public Participation Research Network, and investigates the ways that youth, through social and political participation in online communities, affects their capacity and motivation to engage in social and political issues.
 - Cathy Davidson and David Theo Goldberg's report, The Future of Learning Institutions in a Digital Age (also available as a free PDF). The report found that students are learning in deeply collective and innovative ways, and that learning institutions - schools - have to keep up or risk obsolescence. They offer ten principles for redesigning learning institutions and pedagogical systems to better reflect the way students learn today. The book-length version of the project, The Future of Thinking: Learning Institutions in a Digital Age will be coming out in 2010.

We invite you to join us as we discuss:

1. Technology & Assessment:
- How can educators leverage the affordances of digital media to create more time-efficient, intelligent, and effective assessment models?
 - How can we use new technologies and new understandings of mind and cognition to help us build better metrics, rubrics, etc.?
 - What can emerging technologies teach us about evaluation methods? What would the perfect evaluation tool for user-generated content look like?

2. Assignments & Pedagogy:
 - How can we develop assignments, projects, classroom experiences, and syllabi that reflect these changes in technology and skills?
 - What does it mean to design a course that takes seriously the idea that learning can happen through these digital technologies, new models for grading & evaluation, and new media skills?
 - How can we prepare our students for the kind of social, global, collaborative work in many of todays professional work environments?
 - Of course, developing and implementing these strategies takes a great deal of time and effort on the teacher's part. With ever-increasing stresses on teachers, professors and departments, how can we support innovators who are already crunched for time and energy?

3. Can everything be graded?
 - How important is creativity, and how do we deal with subjective concepts in an objective way, in evaluation?

4. Assessing the assessment strategies:
 - How do we evaluate the new assessment models that we create?
 - What is consistent about all of these forms of evaluation? What are the constants in evaluation and grading?
 - The huge problem of unequal access to technology and digital literacy needs to be considered - how do we account for these differences within our classrooms, schools, and countries?

What are your strategies and experiments? 

We're thrilled that HASTAC and the Scholars program are supporting these types of peer-to-peer conversations and are providing the online space to collectively share our strategies, ideas and experiences so that we might learn from each other.

A special welcome to the members of the Office of Assessment and Innovation at WSU who will be joining us in the comments!

HASTAC Scholar Discussion Hosts:

- John Jones, Ph.D. student, University of Texas - Austin
- Dixie Ching, Ph.D. student, New York University
- Matt Straus, Math/Statistics major, Duke University

118 comments

Hi Jayme,

Sorry for the late reply... just now catching up on all the great conversations that went on today. Loved the "creepy treehouse" concept! Everyone needs to read the original post, but the gist is this:

<snip>

creepy treehouse

see also creepy treehouse effect

n. A place, physical or virtual (e.g. online), built by adults with the intention of luring in kids.

Example: “Kids … can see a [creepy treehouse] a mile away and generally do a good job in avoiding them.” John Krutsch in Are You Building a Creepy Treehouse?”

n. Any institutionally-created, operated, or controlled environment in which participants are lured in either by mimicking pre-existing open or naturally formed environments, or by force, through a system of punishments or rewards.

Such institutional environments are often seen as more artificial in their construction and usage, and typically compete with pre-existing systems, environments, or applications. creepy treehouses also have an aspect of closed-ness, where activity within is hidden from the outside world, and may not be easily transferred from the environment by the participants.

n. Any system or environment that repulses a target user due to it’s closeness to or representation of an oppressive or overbearing institution.

n. A situation in which an authority figure or an institutional power forces those below him/her into social or quasi-social situations.

With respect to education, Utah Valley University student Tyrel Kelsey describes, “creepy treehouse is what a professor can create by requiring his students to interact with him on a medium other than the class room tools. [E.g.] requiring students to follow him/her on peer networking sites such as Twitter or Facebook.”

adj. Repulsiveness arising from institutional mimicry or emulation of pre-existing community-driven environments or systems.

Example: “Blackboard Sync is soooo creepy treehouse.” Marc Hugentobler

<snip>

-So, closed-system knock-offs are clearly not the answer. But it seems as if the well-meaning professor who tries to "go along with what the cool kids are doing" can fare just as poorly (unless you're KSU's Michael Wesch). I suppose it comes down to whether you really understand the medium (and the message!)

Anyway, thanks for your insightful post. What a great way to kick off the forum! :)

 

 

 


142

I am totally not going to defend the creepy treehouse. But I closed the online component of my course to the world (still allowing guests to log in w/ a guest username and pw) this semester amid concerns about FERPA and some student's objection to writing in public. Some students seemed to want to construct the same artificial boundaries in the online space that they were used to in more traditional classrooms. In other words, they wanted a sandbox, not a live site. As one student put it "I don't want people to google me and find dorky things like rhetoric." Has anyone else experienced resistance to using open Web 2.0 tools?

I'm torn about what to do next semester. Either I keep it closed, or I make students sign something that says they must be willing to write online, or I let them build their own treehouses. I like the latter idea best, but I'm wondering, then, how to foster any sense of collective classroom cohesion. If they are all playing in their own treehouses, doesn't that create a whole other set of problems? 

Strangely, students also decided, as a class, that they wanted to do a facebook fan page for their collaborative class project.

126

Great point Will. I feel like you're doing the absolutely right thing by negotiating this issue *with your students*. As you can see, creepy treehouse is very much a perception thing and there are no hard and fast rules in terms of what makes sense to students, gives them a "safe space" to experiment with their writing and thinking. Sometime a walled garden is best, sometimes they want to broadcast to the world using Facebook. Anyway, I guess I'm not really offering any answers here; please let us know what happens next semester!

111

Yeah. Next semester, I will let the whole HASTAC know. As I suggested in my other comment (the one about teaching FAIL), I am learning a lot in those moments where I am forced to stammer "Uh...good question...I'm not sure of the answer." (Those moments are frequent this semester.)

I am leaning toward a hybrid approach for next semester: make students create their own online workspace and then use the course site as a place where we come together. I'm not sure about permissions...if the students want to only invite classmates and the instructor (required) but not the world (ok) then it could get complicated with a score of different blogs/wikis/CMSs. Has anyone tried the hybrid approach? Is there a way to keep it simple?

91

Will,

We have used the term Hub and Spoke for the model you just labled "hybrid." The thought is that students should be working in their own space(s) as appropriate to their community of practice. The course hub is a place that the students all pull together, and post a link to where they work, have meta-discussion of their work, seek peer assessment, etc. The work takes place outside. The access to the Hub would be controlled by the University, the access to the spokes would be controlled by the student (and is often public). The post explores the rationale of using SharePoint as a platform rather than Blackboard. Today I might have less focus on SharePoint than when it was written.

Now the challenge. If students are working in spokes all over the Internet, how does the assessment get done and brought back to the university. Hence our invention of the Harvesting Gradebook concept.

116

Thanks for the hub-and-spoke link. That is a super-helpful way to think about it. Glad to know such a model has been vetted.

I'll probably use drupal for the hub--that's what we've been using for everything this semester--and point them to a handful of tools for the spokes. Do y'all train them in the spoke tools? Or just let them use/figure out whatever tools they want to use?

 

I suspect for assesment, the Learning Record that has been mentioned around here will be really useful. But I'm interested in the Harversting Gradebook, too.

 

119

Reading your post, Will, made think about the creepy treehouse from another perspective, and from that perspective how much the classroom is an eerie fort

But yes, we've also discovered a similar phenomon in our work where we have invited professionals to give feedback to students (in a professional program).  We have come to understand that many of students have not been adequately prepared to appreciate that kind of opportunity (even when the outcome of the activity might turn into a job interview).  That experience helped us understand how the challenges an individual instructor faces might be ameliorated by working with colleagues to scaffold a more coherent curriculum in which such opportunities are not unusual.  As long as you're the only one in your program moving outside of the classroom fort, I would guess you will have students who continue to believe that's the only "dorky" place where rhetoric matters.

124

Yeah, Gary, when I read "working with colleagues to scaffold a more coherent curriculum in which such opportunities are not unusual" I was like "That's what I'm talking about, people!" And that's what we are trying to do in the DWRL. When my students ask "Why aren't we just using Blackboard?" (i.e. why do we have to learn another set of interfaces) I tell them we are trying to learn, as a group, what other opportunities exist.

130

Read FERPA. It prohibits the University from disclosing certain information about students. It does not preclude a theater class that does a public show. Or Marching Band class. Or service learning in a class.

Being forced to act in a public creepy treehouse could be objectionable to students, but that should also reflect poorly on the instructor and institution.

123

Yeah. I think my FERPA fears were largely unfounded and uninformed. They came out of a THATCamp discussion we had a while back. You know, I am teaching with all these Web 2.0 tools for the first time (although I've been using them for a decade) and someone threw up a red flag and another person threw up another red flag and i was just like "fine, I'll close the door like a real classroom." Plus, my students are, in some cases, using copyrighted content. So there's the whole fair use thread we could open up too. Personally, I just had to narrow the number of variables I was dealing with. But I'm really excited about the hub-and-spoke system for next semester. I think that's going to solve a ton of problems.

Has HASTAC done forums on the legal and legislative issues of Web 2.0? (Sorry, I'm also a HASTAC n00b this semester.) It seems like that would be useful. Off the top of my head, i can imagine threads on FERPA, Fair Use, Copyright/Copyleft, Campaign Finance Reform, etc.

Are you saying students' objections should reflect poorly on the instructor? I'm not sure I understand.

130

I was trying to say that having students working in public in a creepy treehouse is creepy and reflects poorly on instructor and institution. Having students working on real authentic stuff in public reflects well.

Students may object to working in public, but the challenge is to help them appreciate that real performance before a real audience is what its really about. Imagine the member of the Marching Band class that didn't want to march at the Homecoming football game. Students get that Marching Band is about performing in public. In your class it should be as obvious why they are working in pubic.

140

Not exactly a solution to your situation, Will; however, I've found that asking students to adopt pen names (and explain their choices) is a productive excercise for multiauthored blogging throughout the quarter/semester.  It gives them a space to work through the notion of writing for a public and how, of course, identify formation is involved in that process.  That, and it's fun to create a pen name! I generally use WordPress as the platform (one class blog with many authors).

(BTW: We should chat at some point about your work on audio recording.  I'm doing similar stuff in my dissertation, which is a cultural history of magnetic recording, especially audio recording.)

143

Hi Will,

Concerning "legal and legislative issues of Web 2.0," a few resources come to mind:

  • Howard Gardner's Good Play Project (part of the larger Good Work Project)? One of the goals of Good Play is to examine the ethical dimension of young people's engagement with web 2.0 activities. Here's a snippet of a recent white paper:

"The new digital media are a frontier rich with opportunities and risks, particularly for young people. Through digital technologies, young people are participating in a range of activities, including social networking, blogging, vlogging, gaming, instant messaging, downloading music and other content, uploading and sharing their own creations, and collaborating with others in various ways. In this paper, we explore the ethical fault lines that are raised by such digital pursuits. We argue that five key issues are at stake in the new media, including identity, privacy, ownership and authorship, credibility, and participation. Drawing on evidence from informant interviews, emerging scholarship on new media, and theoretical insights from psychology, sociology, political science, and cultural studies, we explore the ways in which youth may be redefining identity, privacy, ownership, credibility, & participation as they engage with the new digital media. For each issue, we describe and compare offline and online understandings and then explore the particular ethical promises and perils that surface online."

  • I'd also check out the "Ethics Casebook" section on the Project New Media Literacies site. Project NML (the brainchild of Henry Jenkins) provides guidance and resources for educators interested in learning more about the concept of "new media literacies" and how to incorporate them into their practice.
  • "Bound by Law" a primer on fair use, copyright law, IP, etc., presented in graphic novel form!
123

Though the book is about program evaluation, I recommend Evaluation Ethics for Best Practice: Cases and Commentaries, edited by Michael Morris.  The principles pertain.

107

What I am doing is having a public weblog where I can do normal blog posts (notes to class, interesting links, etc) and copies of handouts/schedule/syllabus/etc. and then using Blackboard (ugh) for more private writing.

111

The word "assessor" historically meant someone who assisted the judge, which reminds me to think about who is the judge? If it is the students and we are assisting them in making a judgement about their work, then the assessment is "assessment FOR learning" (as opposed to "assessment OF learning," which I call "feeding the beast"). While in the FOR mode, it would be good to have educational use of a timeline replay button, as in the new google WAVE; and also to enlist increasingly powerful automated text analyses, and advanced visualizations as seen in the National Center for Supercomputing Applications Automated Learning Group.

Feeding the beast with "assessment OF learning" means shaking out a grade, putting a summary mark of some kind that aggregates all the process and product into a normative and comparative symbol. The beast, in various cases, is the certification review board, state board of education, college, department, school, major professor, teacher...any other entity outside of the learner. I think of those outside audiences in two clusters: trusted others such as mentors and advisors selected by and honored by the learner, and all others, including the institutional "community of practice" of a profession, culture, etc. These beasts must be fed by someone, usually not the learner except in the case of the immediate teacher, but sometimes on their behalf. Unfortunately, many times feeding the beast causes the one doing the feeding to ignore or forget the learner (perhaps unintentionally) through aggregation and summarization. Learners like rich details in feedback about singular items; beasts like summaries and large "n" sizes. We should try to avoid calling both processes by one name, since they are so very different in purposes, appropriate methods, and aesthetics.

One concern I have about "assessing process" is its impossibility in many important contexts. For example, the process that unfolds a piece of improvizational music has to be directly experienced by both the creator and listener (could be both roles in one person or different people) in order to be taken in and understood, and no amount of translation or summary can then provide other people with that same experience, if they missed the performance. "Assessing" the work, in this case, seems to be essentially expressing one's opinion about the totality of the lived experience at some later time, after short term memory begins to fade and the details are a shadowy reverberation. Thinking about both the FOR and OF learning stances, I'm not sure either one can effectively apply. Perhaps this dilemma also arises in "self-assessment" which is another story altogether (e.g. can one assess one's self FOR as well as OF learning? Are these two different metacognitive skills?).

125

Hi, Jayme,  Just FB'ing with a friend in the arts and we were exchanging stories about devastating Art Crits we'd witnessed, where the art "mentor" goes from painting to painting (or whatever the medium) and delivers a public, sometimes devastating evaluation.    Thought I would just get this in here to break up any implicit binary between creative/non-quantitative/soft/touchfeely/arts on the one hand and quantitative/hard/tough/science-technology on the other.   To face the art world, art reviewers, and those Crits, you have to be tough as nails.   I know that's not what you are referring to but I just wanted to make it clear that non-standardized does not necessarily equal "easy" or "freewheeling."

125

Jayme, I totally agree: as instructors we have to allow students autonomy to follow interesting questions wherever they go. In my experience, I've found that one way to facilitate this kind of freedom is to let students know that it's ok to fail. That is, if as an instructor I'm going to encourage my students to be adventurous, I have to let them feel free to follow a train of thought or project even if it ends up not working out. Otherwise, as you point out, the whole process seems like I'm forcing them to a pre-determined conclusion.

93

Hi John.  I made the mistake of not writing directly in the comments box so I've been busy correcting all the dropped punctuation in my posts.  Lesson learned. 

I agree about telling students it's ok to fail but I find sometimes that students don't really believe that the instructor is serious about only evaluating them on process.  They are understandably suspicious.  Sometimes it can take the better part of the semester for them to stop looking for the instructor's approval of their products.

116

Hey John,

Your comment made me go find this Dean Kamen video I watched as part of preparing/thinking about the forum prompt. Wise words, no? :)

 

130

I love this thread. I was thinking about suggesting a whole forum called "Teaching FAIL." I've been compiling some material to that end. Here's an excerpt from the audio version of Jonah Lehrer's book How We Decide. In it, he mentions the work of Carol Dweck. Dweck's work in developmental psychology focuses on <gross reduction> what happens when you tell kids they are smart versus what happens when you tell them they tried hard </gross reduction>. Paraphrasing Niels Bohr, Lehrer suggests that "expertise is the wisdom that emerges from cellular error." I initially wanted to bracket out the "cellular" part. But that might be the whole point. Error is felt in/by the body, which is what makes it so powerful. In some ways, it brings pathos back into a longtime logocentric discussion.

134

"expertise is the wisdom that emerges from cellular error"

I like that.

115

Hi again Will,

Yes, love that quote as well! Also, thanks for mentioning the importance of Dweck's work as it relates to the whole enterprise of learning. I'm paraphrasing as well here, but her research has shown the importance of adopting an “incrementalist” view of intelligence, in which aptitude is viewed as something malleable (as opposed to “fixed” or a “born trait) and dependent on variables like time-on-task. Students who are shown to hold an incrementalist viewpoint tend to put greater effort towards learning because they believe their efforts can change their intellectual abilities. So as you imply above, it's more beneficial for students if they are praised for persistence and effort, an evaluation paradigm that allows for plenty of "cellular error."

114

"...incrementalist view of intelligence, in which aptitude is viewed as something malleable (as opposed to fixed or a born trait) and dependent on variables like time-on-task."

Not to flood the place with the greatest hits of pop writers, but the time-on-task thing is like Malcolm Gladwell's 10,000 hours to expertise idea, right? (I think I've almost racked up the 10,000 hours needed for the dilettante merit badge.) Like Tiger Woods isn't Tiger Woods just because he was born athletic. He's Tiger Woods because his dad made him practice all the time and would do things like drop his golf bad right when Tiger was in the middle of his swing. Every incremental stroke is part of a larger career that is shaped by those individual strokes.

101

What a funny coincidence. I write a quick post about Dweck's work and the cut-and-dry obvious superiority of supporting/inculcating an incrementalist theory of intelligence in students, then go to a brown bag at NYU given by a post-doc in my lab named Paul O'Keefe, where Paul proceeds to convince us how it's really not that simple. Possessing an "entity theory of intelligence" (the belief that intelligence is fixed) may actually be more appropriate depending on the activity goal... well, at least that's his early inklings. I'll try to get him to post something... maybe as a separate blog post, since we're getting a bit off topic... stay tuned!

118

I wonder what "depending on the activity goal" means...do tell if you have more information.

121

Good stuff, Will.  I've also been thinking through ways to embrace error (as an opportunity for new learning) in humanities courses.  Have you tried this at all?  Like, say, through some particular exercises or outcomes?  My courses tend to be process-obsessed; however, I haven't found an effective way to have everyone involved isolate what was at one point an "error" and articulate how that error became an opportunity.  For instance, rhetorics of "revision" tend to focus on the fix, not the opportunity.   

Regardless, thanks for all of your comments here!

115

Thanks Dixie for posting the link to Kamen's video. 

It brings together for me some of the central issues on this forum: creativity, learning, and assessment. Kamen discusses how often people begin without a map, make many mistakes, but then when they finally get a route, they just pass along the map. The process (the recap of  mistakes) are either erased or else presented as watered down anecdotes along the way. Many of us in writing studies see this same phenomenon. Teachers want their students to write well (not unreasonable) but often this want turns into a focus on what the student considers to be his or her final map. There’s no discussion of the process along the way and little value in taking chances or failing.

 

 Another video clip on creativity is this one by Ken Robinson on TED. He makes the argument that schools kill creativity.

Also, he talks about how the unpredictability of the future makes it extremely difficult to figure out what to teach let alone what to access. While he stresses that being wrong is of course not the same as being creative, Robinson is convinced that fear of failure prevents creativity. I tend to agree.

 

124

Thanks, Patrick.  I like your emphasis (via Robinson) on the fear of failure. To add to this discussion, what role do you think (for a lack of a better term) "ownership" of the course material plays in a fear of failure?  While I can't generalize, I would say that one thing I like very much about project-based curricula is how students tend to become the experts in their project development moreso than the instructor, especially when the projects mobilize competences that the students already have (likely unbeknownst to the instructor). 

Along these lines and re: assessment, projects lend themselves to persuading someone's instructor, peers, and target audience(s), rather than being accountable for content produced by authorities on the topic (e.g., a scholar of computers and composition).   

In other words, I'm wondering if it's a fear of failure that prevents (or curbs) creativity, or if it's the assumptions about expertise and authority that subtend content-based curricula.  Could be both, perhaps.  What do you think?  I ask because I'm looking for models that are less focused on the student and more focused on the institutionalization of learning and the culture of instruction in education. 

Appreciate your time!

124

Interesting question Jentery . . . I think fear of failure and assumptions about expertise and authority do play a role in curbing creativity. I really think both do.

You got me thinking about the relationships between failure and ownership and failure and expertise. Can one exist without the other? And in what ways can both curb creativity? And perhaps more pointedly, are their productive ways where these notions can work together?

It strikes me that notions of failure and ownership are closely connected, maybe even inseparable. On the one hand, I can think about a (bad) model of learning where ownership is exclusively in the hand of the teacher (or the institution). In such cases we might see the classic case of the disengaged student attempting to give the teacher what he or she wants—a reproduction of the teacher's experise. I realize that this is a crude oversimplification. On the other hand, I really like the model when ownership is shared, something like what Cathy Davidson describes in How to Crowdsource Grading. In this model, students do indeed become experts (and maybe owners) of the curriculum.

But what role should teachers' expertise play in the enactment of a course?  Or in the "institutionalization" of content knowledge? I'm considering trying Davidson's idea of crowdsource grading next semester when I teach a class on writing technologies. When I look at her wonderful inquiry-based model, I find myself asking where is her expertise, her “ownership,” if we want to use that term? My sense is that both are there in, for example, the course design and the suggested readings. But it is also open to revision, radical revision in fact.

So, as I plan my future course, I wonder why am I still a little nervous about adding a line like this one from Davidson's syllabus?

"all students will receive the grade of A if they do all the work and their peers certify that they have done so in a satisfactory fashion.  If you choose not to do some of the assignments and receive a lower grade…”

Maybe the answer is that many of us have been socialized to display our ownership and expertise through the distribution of grades...and maybe this model is what limits creativity.

Sorry for the ramble. Let me know if you have any thoughts here.  

 

 

115

Billy Collins is reading his poetry at the National Writing Project conference in Philadelphia. Here are a few lines from "School," a poem in which the teacher imagines all of the students he every taught coming together in a small town. Collins speaks to the ways in which grades can become self-defining (or a least the ways in which educators can imagine their students):

Their grades are sewn into their clothes 
like references to Hawthorne. 
The A's stroll along with other A's. 
The D's honk whenever they pass another D.


You can see the full poem here.

 

119

While the new media side of this question is interesting, I think the biggest change we need to work toward is a shift away from assesment on outcomes to something which primarly focuses on process. There is a growing body of literature in Educational Psychology that suggests that at all stages, but particularly at the basic stages of developing profficiencies, that a focus on process is far more important than outcomes.

In the big picture, I think we need to be focusing on fostering the kinds of habits which these portfolio assesments support. Namely, metacognition and self-regulation.

 

138

I don't know that outcomes and process need to be understood as being at odds, but I completely agree that we do little in currciula and grading to put the appropriate emphasis on learning processes.  As Jayme notes, promoting self-assessment and student agency is essential.  But the other reason is simply because learning takes time, which an ePortfolio imports.  To that end and as prelude to our harvesting work, and to underpin some of Trevor's points and pointing to research on learning, I point here

129

Trevor, I agree that the main benefit of portfolio-based assessment is metacognition and self-assessment (as well as self-regulation.)  One of the things that we would like to see happen with our "Harvesting Gradebook" project is that while institutional assessment might get embedded next to the students' work, wherever it resides, that students would be encouraged to craft their own embedded feedback systems so that they could say, "I was trying to do this with this particular project.  Is that the reading you're getting? Have I succeeded? How or how not?"  That allows the students some autonomy about the kind of feedback they are soliciting, for them to become more intentional about what it is they are trying to achieve and savvy about how to ask for the feedback they need.

124

Jayme could have pointed back to this piece from a year ago where a design was proposed for a system to allow learners to request specific feedback in a more general campus-supported harvesting feedback tool. All told, the technology for these ideas is readily available, but as Jon Mott notes in this post, its not really a technology problem. Mott is reflecting on a John Seely Brown talk that explored the role of play and the different kinds of ourselves that we are educating.

 

 

133

Is anybody else but me weary of this line of argument?  Yes, Nils, I could have pointed back to the blog post you mention but I would also like to point out that those are pictures, PICTURES of a dashboard.  I know because I created those pictures.  You say "All told, the technology for these ideas is readily available, but as Jon Mott notes in this post, its not really a technology problem."  Actually the technology isn't readily available and that is part of the problem.  We have had a lot of great ideas about how to streamline this process; how to make it easy to create and embed feedback mechanisms; how to visually output the data in ways that capture process - things that grades can't even begin to do; but the truth is that we have very little humanpower to put behind making it a reality.  The technology does matter when the implementation for really great ideas - such as  crowdsourcing formative feedback in meaningful and scalable ways - are being blocked for lack of human resources to focus on the implementation. 

 

I guess my post is really about asking if there is anyone in the HASTAC community that is interested in collaboratively working on some of these tools to make them a reality?  Sure, great tools don't make great teachers but they can make a difference in doing things that were never possible before - to get people thinking about assessment as truly formative without increasing the time commitment.

 

 

117

Hi Jayme,

I would definitely be interested in working on assessment tools with you -- and I'm sure many others will want to join before the forum's over! One straightforward way to think of improving the "effectiveness" of formative assessment in the classroom is "simply" by designing a tool that gets the information from the "reviewer" (teacher or student) to the "reviewee" in a more timely manner. One can easily dream up ways that technology can make that a reality, today.

Also, I totally agree with you that great tools don't necessarily make great teachers, but I did want to mention the possibility of designing assessment tools that can (almost as a welcome "side effect") illuminate or broaden the user's perspectives on assessment. In a way, one of the many neat things harvesting gradebook does is make the student more aware of this "business of assessment": what tools & metrics are commonly involved, how to construct a rubric, how to select an appropriate scoring method, etc. Another example is Herb Ginsburg/Wireless Gen's mClass:math pda product, which employs a diagnostic interview protocol based on Prof Ginsburg's research on how young children come to understand (or not understand) mathematical thinking. It's conceivable that a teacher who uses this diagnostic tool enough times will gain a better understanding of children's developmental trajectory towards math proficiency.

145

In my Interdisciplinary Computing and the Arts classes, I've been using a wiki, which could be said to be a very simple tool, but its been interesting to see how its been used.

For example, in my Intro to Electronics for Art class (wiki here: http://bang.calit2.net/wiki/VIS147A), I asked the students to post their first assignment in the wiki, but I didn't specify a format. The first poster created their own page, included multiple photos and a text description of their project and their process. (the projects page is here: http://bang.calit2.net/wiki/147a_Throwies_Projects). What happened next was that all the other projects followed that format, which was a format I was very happy with. The student actually told me later that they wanted to post first to encourage the other students to do better, setting a high bar. I felt that by simply making the project documentation something public, it improved the quality of the projects. Students also know that the projects are on the web so that I can share them easily with other instructors, which also may motivate them.

I'm very much an advocate of using open souce tools, and I think that a wiki is an easy way to allow students to share their work online, and can also be used in a wide variety of ways. I've also invited students to add relevant examples to the sections for each week, but they haven't gone that far yet! I did have to take some class time to explain how to edit a wiki, though, even in an upper division art and technology class! So don't assume that students all know how to edit wikipedia just because they use it daily.

Initially, I was motivated to use a wiki because it related to my own pedagogical approach, trying to learn from Paolo Friere and bell hooks how to encourage a horizontal learning environment where every student is empowered as an educator and a learner. It also is great because it allows me to easily compile examples and links to readings for class, which can then be used for lecture notes since they're already online. I usually use my wiki instead of a slideshow now, because there doesn't seem to be much benefit to slides.

129

Jayme, I hear you, the Harvesting Gradebook is documented, pilot tested, and we are using it for WSU accreditation, but we have not gotten the last mile -- its not 1-2-3 simple the way it needs to be (no matter how hard I imagine what it could be). So, in a way, it is a technology problem, we can't have the learning and assessment conversations that are needed, because the tool's clumsy aspects get in the way.

[ For those who have not figured it out, Jayme and I are colleagues. We blog here, where we have agreed to be able to disagree in public. We agreed this because we recognize a community of practice, in Lave & Wenger's view, is not a "warmly persuasive term for an existing set of relations." (Williams, 1977) ]

And I understand your post as a version of Margo Tamez' "Urgent call for help" a post aimed at rallying help to work on a problem. Tamez has created an interesting portfolio of her work in a community of practice. Her work has been very formative to shaping my understanding of ePortfolios and collaboration in community.

So, to add my voice to yours, we need to get the last mile. Recently Theron found a teacher using a Google Docs spreadsheet to make an online rubric for grading -- another attempt at ideas parallel to his demonstration of mashing up a portfolio with a rubric in Google Docs.

Each of these demonstrations are tantalizing, but each has a few too many wrinkles to meet the criteria of "Blogger simple." Strapped for resources to forge ahead as fast as our imaginations run, I agree we could use some help getting the last mile.

117

I'd be interested in collaborative work on this problem. I developed an "eFolio" also called the "Personal Learning Plan" (Gibson Personal Learning Plan on Google) and it has seen a lot of use over the years. I recently put it into a LAMP framework and I'm working now to bring that new version more into a WEB 2.0 style, but as someone mentioned here, its tough to make the problem of assessment easy and straightforward.  I think the problem of increasing the time commitment has to be addressed by saying that we actually need many more time-based measures, which I think is the only way to capture the dynamics of change over time. So this leads me to believe that we HAVE to invent more automated feedback mechanisms of very high quality, which can provide useful feedback FOR learning. (For the assessment OF learning, we can stay with the broken, few measures per unit of time, system that we have now, which is not going anywhere soon anyway).

Games and simulations show the way to do this in many respects, because the software stays in immediate, real-time connection with the actions of the user and gives rapid, ample hints for improved performance. A layer that can be developed much further is the post hoc analysis of the user path of a particular performance and a comparison with the crowd-sourced paths of other problem-solvers who utilized the same digital space. Ron Stevens' work at UCLA is a great model for some of this.

While working on an online case-based reasoning application with Sara Dexter, now of the University of Virginia, I was part of a team that built an eassy-scoring tool using a simple Bayes network that got trained from well-scored pieces. The engine could, almost out-of-the-box, do better than unmoderated humans using a rubric to score student work. I think this area holds some promise for a 2.0-type performance assessment system created and owned by some CoP.

Finally, for now, Robert Mislevy and his colleagues and collaborators have laid out the theoretical framework for this new kind of performance assessment system. So the outlines and hints of how to move forward are out on the table. There is a need for a collaborative research project to pull these strands together in new ways and make additional advances that might potentially address the subject of this great dialog at HASTAC.

120

I use the Learning Record portfolio system for evaluation in my courses, and one of my colleagues, Will Martin, has created an excellent online tool for storing and commenting on student work. You can view his walkthrough of the system (pdf), but if you're interested in trying it out, it is currently live at https://lro.cwrl.utexas.edu/ for anyone to use.

The LR doesn't currently support the multi-layered feedback that Jayme has described as part of the Harvesting Gradebook, but Will created the LRO as an open source project, which I believe he will soon be releasing so that others can build on the framework he has created.

107

Also, the conceptual framework of the learning record, which the DWRL will be addressing this Friday in the LRO workshop, is adaptable to just about any platform. It can be delivered in Word, Drupal, or even (gasp) on paper.

We'll live tweet it. Follow DWRL (http://twitter.com/dwrl) if you are interested. #LRO 

123

Thank you, John, Dixie, and Matt, for organizing this wonderful forum.  Great topic! 

Per the comments above about outcomes, as an instructor who relies heavily on project-based curricula, I've found that asking students to compose their own outcomes is an incredibly productive exercise, especially when those outcomes go through a peer review process and are mobilized in the final evaluation of a student's project.  Of course, these student-written outcomes can function in tandem with outcomes written by the instructor, too.  That way, you find a balance between institution- and community-based learning. 

Echoing Jayme's comment about metacognition and self-assessment, I would add that curricula with flexible and modifiable infrastructures that privilege modeling critical thought (over, say, being responsible for specific content or canons) really lend themselves to the crucial moment when a student asks a peer, "Have I succeeded?"  If students are responsible not for the content of a certain text, but rather for articulating why, how, and for whom they made particular moves in a project, then they can also ultimately address how to replicate those moves, revise them, or teach them to someone else. The project, as a model, has a work-life beyond a course. And again, outcomes are crucial to this kind of process-oriented learning climate. 

My only concern is naturalizing the word "play" as the antithesis of science and such.  After all, play is serious and culturally embedded, and it has consequences.  I concur with the impulse to encourage "low stakes" (read: will not be evaluated) exercises in a given class; however, what are the effects of rendering "low stakes" synonymous with "play," particularly in the context of the classroom or the course?  True, no one's going to quibble with the notion that students should be given opportunities to foster their imaginations and take risks, yet what I'm wondering is how to foster adaptability and self-awareness.  Both can be quite playful (in the sense of taking a chance and embracing error), indeed.  But, as Jayme points out, perhaps the primary issue here is how to sustain a space where chance and error are not just ok; they are exactly how you learn and document learning (sort of like a change log in code).  Right now, the only thing I found that works consistently is resisting the urge to grade students on their work until the quarter's/semester's end and relying primarily on peer review and workshops for feedback. 

What works for others?  What chancy spaces for abductive reasoning, modeling, or the like have you found productive? 

 

121

Perhaps this is because I'm in a visual arts department, but almost every quarter I find a way to tie in some physical theater exercises from Theater of the Oppressed. Augusto Boal has a wonderful book entitled "Games for Actors and Non-Actors", some of which are more verbal than physical, but all allow a space of play and experimentation with ideas. What I love about these games is not just their ability to spur conversation and engagement, but the way that students every quarter tie the material into the games in novel ways. Some ones that I stick to are Colombian Hypnosis, the Sculpture/Image game and the Mirror/Image game. Clearly these work best in a performance art class, but even in my interdisciplinary technology and arts classes, they serve as ways to allow students to think totally differently about interactivity, hierarchy and learning, outside of just a discussion sitting in a classroom. You'd be surprised how easy it is to find an empty, unused outdoor space inbetween classes on a university campus. Herre are two pictures of a game called Grandmother's Footsteps, an excellent exercise in rule bending, hierarchy and social inclusion/exclusion 1, 2. Often, students will relate the class discussion back to embodied experiences they had in the game.

Another technique i try to use to make my classes more horizontal (and neither of these two ideas really have to do with new media but i wanted to repond to your question), is to make sure that students have time for small group discussion, groups of 4 or 5 at the most, to discuss readings before we talk about them as a group and then i add my personal analysis later in case something was missed. That way, they can emote to their peers their real thoughts and get the thinking going, rather than just sit silently in a large group not feeling like they could say what they really think about the reading. It usually helps getting more people talking, but I still end up with the problem of the same few people raising their hands to speak even after those small groups, so I usually have to intervene a bit to hear from more people. I think the important point here is that students need spaces to talk to their peers without us watching or being present, apart from the comment I made earlier about wikis, and they don't nevessarily have the time to do this on their own. Electronic forums can facilitate this, but I think things like webct are usually open for instructors to see, and everyone's busy on facebook talking about silly photos and people's outfits, not the latest reading about the philosophy of electronics.

 

 

126

Washington State University has been exploring assessment in a community as an alternative way to think about grading.

To experience a simple method for harvesting feedback from work anywhere on the Internet, as well as create more fodder for this discussion, please assess Fiona's opening prompt for this discussion (above) and then click HERE to open the feedback tool. After some data has been collected we'll post a link to view the results.

 

127

So far we have just one response to the survey above. It is generating conversation here at WSU, which I'll update here when it settles. This reviewer gave an incomplete set of numeric scores using the rubric, but has very interesting comments which are below.

Identification of the Problem:
All of these things are "true"; none of them is mutually exclusive; many of them are difficult to do; many of them need contextualizing knowledge such as student group members should first understand group self-assessment and group processes); many of these have been discussed in relation to the very first instructional technologies (e.g., See Roxanne Hiltz's work on virtual classrooms and asynchronous learning networks, where she emphasizes how to use systems and software to promote participation and collaboration); each sentence could be a discussion thread.  

Contextual Factors:
 No one, without clear prompting, training, and encouragement from other group members, is likely to engage in the higher levels of contextual factors in blog posts.  So, using these kinds of ratings will end up seemingly very critical and I in no way want to be critical of these intentionally very preliminary thoughts.  Using these criteria might be a good goal of a very rigorous discussion session, once major positions and issues and background are made available.  That is, these kinds of criteria are likely to be met only in quite sophisticated venues such as school debating clubs or courses on logic and argumentation.  

Own Perspective:
For many such questions, I prefer:
 
 "Clearly presents and justifies own view or hypothesis while qualifying or integrating contrary views or interpretations."
 
 Forcing the implied critique of "May remain within “safe” or predictable parameters," depending on the issue, could seem condescending.
 

Data:
  
No sources except an occasional link to another blog post are presented throughout the discussions -- for good reason, and because these kinds of assessments were not explicitly made part of the goals of the discussion.  So, post-hoc application of such criteria I think could be quite harmful.  

Feedback on Criteria:
Trying to accomplish even the middle of all of these dimensions would require quite a long time, multiple revisions, and much longer text than most any kind of blog implies or could tolerate.  
 
Now, if these dimensions were the point of a course, and separate components of a final paper were each developed to work on each one of these dimensions separately, with good feedback, leading up to a final integrated argument and project, with group evaluation (following good group process understanding), that would be a great course and learning process.
  

Respond to these two perspectives on assessment:

1. Only faculty are qualified to create and ratify assessment criteria.
2. Expert consensus from the community of practice should validate the assessment instrument and criteria.

I think all potential stakeholders should be involved in validating assessment criteria.  

121

Thanks to the organizer of the forum questions -- these are definitely concerns and ideas to gnash thoroughly.

I'd like to offer a few tensions that I'm seeing from both the forum prompt and from my own experiences (teaching my own classes and mentoring other instructors):

1) What do we do with the desire for "effectiveness" and "time-efficiency" (which is a problematic and very real constraint) and the desire to be good teachers, mentors, and learners?  Unfortunately, these things are often at odds.  And is technology the best practice to find ways to be effective and time-efficient?

2) I completely agree with Jayme and John's points about making sure that students do have the opportunity to risk and fail, to discover what it means to be "creative" and to be "productive" without the threat of "assessment."  But I also think that we can tease out the differences between "evaluation" (which can be different than assigning grades or numbers) and "assessment" (which, given the language of testing, is about grades and numbers).  So, how might a paper or a project or a presentation be evaluated (e.g. workshop or peer review feedback) but not assessed? 

Moreover, how might we need to better negotiate the two poles?  After all, as much as we want to be student-centered and to have student-generative learning, there is still a time and place for direction, framing, and lecture.  I agree that process is important, and things like self-evaluation and metacognition are important -- but we still have to teach students and ourselves what that looks like, what that means, and how to evaluate/assess those things.  Given that one student's metacognitive process is not the same as another's, the danger here is ceding too much to the student.  And how do we balance one student's desire to learn the "old-fashioned way" against another student's desire to "mix it up?"  It might be also helpful to think about how to work within prescribed/proscribed frameworks (given that teachers are often constrained by larger structures, e.g. institutional demands) through oppositional, autoethnographic, and resistant practices.  (Obviously, we're all doing this to some degree already.)

3) Finally, I have a specific concern about collaborative projects -- mainly because I find them a pedagogical minefield -- given that asymmetries in power, in interest, in willingness, in ability, and in personality (even personal politics and values) often produce challenges in assessment and evaluation.  Having survived a number of group projects at almost every level of my education and having tried out multiple configurations of collaboration, I sometimes find the drive toward collaboration for collaboration's sake to be counterproductive.  I do recognize the value in group work, in group assessment, and in the nitty-gritty of just trying to figure out how to work, live, and socialize in groups.  it's just a thorny process. 

Simultaneously, I also find collaborative work to be used by some instructors as "time-savers" and as "time-fillers."  How might we metacognitively approach our own desires for collaboration?  Again, what happens when our best intentions and best practices simply do not yield what we want out of these projects, processes, and technologies?  Do we simply keep trying to apply the collaborative (or technological) fix? 

 

74

Thanks for pointing out these tensions.  I agree that collaboration for the sake of collaboration is troublesome.  It's one of the reasons we were drawn to Hastac from Cathy Davidson's interview about participatory learning.  Certainly, putting students into arbitrary groups feels phony -- though there are many people who would argue that it happens all the time in the working world when we are forced to be on committees that hold no interest for us -- but I think that others of us would agree that "surviving" these kind of artificial group setups is not really the most productive learning experience.  What really drew us to the participatory learning article was this notion that people could "join" at all levels.  That there could be lurkers.  That people could come and go as they became interested and saw ways that they could contribute.  So much of what we have been thinking about at WSU is how to provide these participatory spaces and make them conducive to contributing.

But this seems to require a really different setup for assessment.  I agree with Trevor that the way out would seem to be some sort of portfolio assessment where the student could make sense of his or her contributions and interactions in various participatory spaces but it quickly becomes so different from the traditional classroom situation with a grade at the end that we'd be kidding ourselves if we didn't admit it's a little scary.  Suddenly we're not only asking about the value of classroom assessment but the value of the university as a credentialing body.  But I agree wholeheartedly that there is a critical need for instructors to act as guides.  And universities are unique in the kind of diverse intellectual power that they house in one place.  It really seems like it's the assessment piece that is the missing part of the puzzle.

102

I agree with many of your concerns. I'd like to focus on your second point, though, since I've blogged about some of the implications of collaboration already. This semester, I've been doing a mix of group discussion and more straight-forward lecturing in my literature course. I tend to start each new work by doing something more along the lines of a formal lecture then, as the students seem to get a handhold on the work, shifting more and more toward student-driven discussions. This technique is, obviously, not revolutionary in any way. Still, I'm increasingly convinced that very simple things like arranging the desks differently or choosing to be silent for long periods of time during discussion have dramatic effects. My goal, though, is always to model thinking and guide their own thoughts, not dispense knowledge.

But neither approach is as easy as it seems. I've seen courses that were all lecture that were wonderful and courses with too much student-led discussion, or discussion that was ineffectual and rambling. So much comes down not just to methods, but to the ability and experience of the instructor. Not technology, not methodology, not assessment style... nothing but good or bad teaching.

This is all to say, these are important questions and ideas (and I'm already greatly enjoying this forum), but it's all so incredibly complex and individual already, can we ever hope to do more than determine some very general guidelines or suggestions? Even the "best" assessment style fails if the instructor is lousy.

126

Michael

You said "Even the "best" assessment style fails if the instructor is lousy."

I don't think so. We have evidence of powerful assessment coming to student projects from industry while at the same time the faculty in the program were saying they were not even sure how the assessment criteria were relevant. We have other evidence from an online discussion coure where the instructor went AWOL and the students carried on without the instructor using the course rubric to give one another great feedback and concluding this was one of the best courses in the program.

I think that a theme in this conversation is that the assessment is being done by the teacher -- but that undercurrent misses many of the ideas in Fiona's introduction -- I think assessment 2.0 is crowd-sourced (for example see Cathy Davidson). 

I think this conversation needs to be about a shift from institution-centric to community-centric learning (and assessment). See this spectrum.

 

 

111

Fascinating links. Thanks. While I'm excited by the idea of crowd-sourced grading, Cathy noted that this model is one that has high barriers to entry for those not in secure positions: graduate students, adjuncts, lecturers, etc. What I'm most interested in is finding ways to improve the assessment practices today, within institutional limits, even while we work for changing the models themselves. Question: if assessment and learning are best (determined to be most effective through quantitative analysis) when community-based, does the instructor have a role beyond setting the initial parameters for the course and providing gentle guidance throughout the semester as necessary? Beyond assessment, we're also talking not just about grading, but the role of the teacher in the classroom and what types of knowledge and learning we should privilege. Further, we need to ask if different methods are better suited to some types of work than others. Not every course has the same goals or subject matter. Will the same methods of assessment work for them all? I found the spectrum PDF you linked interesting in this regard because it suggests that unless you're all the way to the right (fully communal), then you're old-fashioned and doing it wrong. Is the current model of professor-as-expert so broken that there's nothing salvageable? Is metacognition, communal assessment, consensus, and process more important than detailed knowledge of a subject? How do we negotiate the poles? My fear is that, as is so often the case, we'll be so excited by new methods that we'll abandon what does work in the old methods.

121

Good points, Michael. I think one problem is that we're so ensconced in the traditional model of teacher-as-authority that as we bring in new technologies, even with the very best intentions, we end up with a digital version of the exact same participation structures:

 

 

That can't be right!

Yet we can't forget that the teacher continues to play an important role, even if that role is no longer as sage on the stage. Good teachers choose content, shape learning, guide conversations and collaborations. They don't need to be experts in every aspect of their content area anymore, so they are (ostensibly, at least) free to explore other strategies for supporting mastery of content and practices within their domain.

109

Hi Jenna and everyone,

I could not agree more. This is a great image to prove your point, Jenna:) Just recently in theSlactions conference, each presenter walked up to the podium and talked to a sitting audience while showing a Powerpoint. Very few speakers ventured into different scenarios of presentation. Nevertheless, some of us who organized the event acknowledged that we might have precipitated a certain type of interaction by using an auditorium ( a classic layout).

So I think that in the case of SL, its cartesian nature is definitely to be responsible for some of that reproduction of old spatial (hegemonic) class structure.

Where SL breaks the traditional hierarchical class system is in the way we present ourselves - clothes, hair.... I remember when I had a presentation to some top executives at the National University - the day before I went around my usual SL sites in SL complaining to my fellow female SL residents: "I have nothing to wear!" Eventually someone gave me a decent black suit (my wardrobe in SL being mostly black goth....)

But going back to the point: it is indeed interesting how the moment we have a space (CMS, toolkit, platform...) where we can dare to try new approaches, the first thing we seem to do is reproduce old structures (think SLOODLE - SL + Moodle, which with honorable exceptions seems like a great way to spoil two great environments SL and Moodle and turn into an environment that while embodied, is underlined by some sort of imposed top-down structure....

 

119

Although there's a lot to chew on in the "Tensions" post above, I was particularly struck by this comment:

I agree that process is important, and things like self-evaluation and metacognition are important -- but we still have to teach students and ourselves what that looks like, what that means, and how to evaluate/assess those things.

I use the Learning Record to evaluate student work in my courses, and over the years I have found that the students who resist it the most are the ones who are accustomed to getting good grades in school. For these students, some of whom have so internalized outcome-based evaluation that they rarely think about it, changing the method of evaluation in the classroom is deeply unsettling, forcing them to reevaluate their own learning processes. So far, I have found that one of the most challenging tasks I face each semester is teaching students how do deal with this shift in perspective.

134

Hi, Nils---I'm trying not to be overly-involved in this really fabulous discussion because the Scholars are the key here--allthough I'd love to be replying to each and every comment and post.  This is all so interesting and important, especially in view of the reauthorization coming up of No Child Left Behind soon.    Quickly:  I find this feedback tool exceptionally useful in the way it guides thinking and is also open ended.  It is both directive and not-directive at once and that is very helpful.   Is it open source?  May I use it next semester in "This Is Your Brain on the Internet"?  

97

Cathy,

The survey above is implemented in WSU's online survey tool, Skylight. You could do much the same thing is some of the commercial survey tools, but we like ours because of the rubric question type in it.

We can help you use Skylight next semester. Right now it requires a WSU-based login, but an upgrade is coming. An important feature of the tool is its collaborative authoring and reporting capabilities. We move data from Skylight to Google spreadsheets and use them to provide dynamic graphs of results, such as those you have seen at the bottom of this post.

104

I'm meeting with my small army of TA and Teaching Apprentices next week and we'll talk about this.   Best, Cathy

122

Here's a fanvid by a young woman who goes by valerie. It's her creative retelling of the story of Dean Winchester, a key character in the television show Supernatural.

The above fanvid is considered a high quality work by the standards of the fanvidding community. This is to be expected from someone as committed to productive enagement within the fanvidding community as Valerie is. According to her YouTube channel, she has been producing videos for over two
years (she lists her current age as 21) and in that time has uploaded two dozen videos, all of which are available for public viewing. She has logged just under 15,000 visitors to her page, and viewers leave comments in multiple languages. Valerie lives in Belarus.

Visitors to Valerie's site don't just watch her fanvids; they also provide feedback on her work. This feedback might also be labeled "assessment." For the most part, the assessments offered by Valerie's community is positive and encouraging; this is due in large part to her talent as a fanvidder, and perhaps in equally large part to the assessment norms that have been established within this type of affinity space.

For fanvidders, negative feedback is typically characterized by no feedback at all. In other words, that Valerie receives so many comments, and that her site draws so much regular traffic, is really the evidence that she's doing something that her community values. If at some point in the future her viewership or rate of comments dropped, she would have indicators that her value to the community was decreasing. Another fanvidder could be producing better vids on the same thematic focus; or the source material she chose to work with could be decreasing in popularity; or she could be producing too few videos, with too much space between, to keep her audience returning. Or--and this is entirely possible--she could be producing lower quality videos. Jim Gee writes that in affinity spaces, members actually want feedback--both mentoring and policing. Valerie, presumably, wants a continued audience for her work, so she would do well to pay particular attention to the types of videos that receive the most positive feedback, since that's the work that resonates with her intended audience.

Novice fanvidders also stand to gain by paying attention to Valerie's feedback, as well as by the trajectory of her work. They can see what kind of work she produced as a novice herself, and they can see how she developed as a vidder and what sorts of projects are valuable to this community. As fanvidders navigate across multiple fanvidding platforms (it must be noted that YouTube, while perhaps the best known, is not the only or even the primary source of fanvid materials), they develop their expertise about what sorts of work is of value to other members of this affinity space. In participatory spaces, assessment is build right in to the valued practices of those communities. Human activity, as Gee explains, is the natural home of assessment: You don't have to write an essay to prove you can craft fan fiction; you don't have to pass geometry to help build a video game; you don't have to ace the SAT to contribute to Wikipedia.

 

Okay, so human activity is the natural home of assessment (I stole that phrase from Jim Gee). But classroom practices are also human activity, aren't they? They're as authentic, if at times less relevant to individual and communal interests, as the practices embedded in affinity spaces. Where does assessment in its traditional form go wrong? In my view, it's lifted too far out of its natural home--human activity. Abstracting assessment, talking about it as separate from and independent of the everyday, meaningful practices of our social existence, is just...strange and confusing.

 

144

This is a nice and important exploration.  It is all too common to hear assessment (accreditation) referred to as an unfunded mandate.  When assessment is understood as you have described, then the lament heard from outside the institution is "what then are we paying them for."  And when it is understood in ways that are not understood as integral to human activity, we get more SATs, CLAs, and other standardized measure that have no utility for making students' learning experiences richer.

But I would venture it is not often understood either inside or outside our institutions in the way you describe.  That problem creates lots of problems.

111

I think this is an interesting case, and I do think that fan-based/community-based kinds of production and peer review are exciting and vivifying -- they provide an interesting model of how different kinds of knowledge gets produced and circulated (as well as proof positive that people create and support things they're invested in).  I do think though that even fan communities often replicate hierarchies, cults of personality, and rubrics of assessment that can be just as capricious, violent, and marginalizing as traditional, institutional, normative ones.  What gets defined as "good" or "bad" can be mandated by a coterie of popular users lending endorsements, driving up page views, or seeding the reviews with low scores.  The e/valuation of a fanvid can be incredibly uplifiting and it can be incredibly souring depending on who's watching, ranking, and commenting.  Even the most well-intentioned communities can collapse into flame war and fragmentation, even annihilation.  (I am reminded of all the BBS communities and storyboards I belonged to in the 90s.)  I can definitely see the appeal of these ostensibly emergent communities and determinations of success or worth, and it might even be interesting to talk about when the crowd goes south. 

116

"I do think though that even fan communities often replicate hierarchies, cults of personality, and rubrics of assessment that can be just as capricious, violent, and marginalizing as traditional, institutional, normative ones.…Even the most well-intentioned communities can collapse into flame war and fragmentation, even annihilation."

For example, see Wikipedia. :)

113

I've been looking forward to this forum since it was first announced. I spend a lot of time thinking about how to evaluate my students. I try to include more informal writing and class participation grades than I would otherwise as a result, but I still find myself basing the majority of my students' grades on their formal essays. Much of this, unfortunately, comes from departmental requirements. Since I'm currently teaching an English course and the department requires a certain percentage of student grades be based on formal writing assignments, my hands are somewhat tied. Still, I'm curious to hear what methods people have found that successfully navigate the need to have students produce formal writing while acknowledging and promoting the process of learning how to do such writing.

John, I know you've used the Learning Record before. I'd love to hear some details of your experience with it; how well does it work for a substantial writing component course?

Returning to a point I made in a recent blog post, much of humanities work (even in the "training" stages of undergraduate work) is so insisently solitary that I wonder how appropriate it is to focus on evaluating collaboration, as one of the questions in the first post in this forum suggests. That is, I remain skeptical of just how much collaboration can come into the work of qualitative literary analysis.

I don't really have a lot practical suggestions to add, but because I'm unhappy with the grade-the-final-product model I'm currently forced to follow, I'd love to hear of some successful different methods. I would love to be able to give an A to the student who begins the semester on unsure footing, but progresses greatly in ability by the end while giving, say, a B to the excellent student who doesn't seem to advance in abilities at all. The prevalent system, though, doesn't allow this type of grading, as far as I can tell.

115

Michael,

I'm interested in the notion of humanities-focused work being largely solitary--as a longtime adherent to humanities-type ways (with a B.A. and MFA in creative writing, followed by a quick coursechange into education), I've found that a lot of the ~real~ work of humanities scholars and practitioners has a fairly significant collaborative component. Indeed, it seems to me that the most successful practitioners across humanities-based disciplines are the ones who can, and who indeed want to, collaborate on projects, knowledge-building, and idea exchange.

Thoughts?

116

I don't see why you couldn't establish criteria that rewards growth during the term for some percentage of a grade.  The trick would be to deter students from intentionally doing poorly at the outset.  I've used self-assessment to this end, asking students to reflect on their own gains. When scaffolded well only a few will inflate their self assessments.  More often than not, those wallowing B students know it and report it.

116

You're right, of course. Even if you look only at the final products--articles, books, etc.--we could consider all the quotes and sources themselves evidence of collaboration. Still, the traditional model of scholarship is solitary research that leads to solitary writing. I'm currently working on two different large projects, one of which has me cloistered away (my dissertation) while another has me working very closely with a programmer (a digital humanities tool we're developing). And I have very little idea how I could change either one, even if I wanted to. While I could go to the dissertation group meetings (if my schedule permitted) and get more feedback on my writing, the process of reading, thinking, and writing would still be pretty much taking place all in my own head. Of course, dissertating is a bizarre, somewhat atypical process, too, isn't it?

Still, I'm curious to know what "real work" you've seen that is otherwise. I've worked as an editor for journals and books; that was far more collaborative, but it's also work that is, unfortunately, not considered as "real" as publishing one's own research. From what I've seen, too, most scholars tend to guard their ideas quite jealously until they can get them published. The sciences, on the other hand, seem far more open to collaborative work. To reiterate a point I made elsewhere, I'm not sure qualitative analysis, which is what most literary studies is these days, is at all amenable to collaboration... and I'm not sure that's a bad thing, either.

 

109

You're here, right? In this online forum? Where collectively, we hope to arrive at some conclusion, however tentative, about the role of assessment in the 'digital age'?

And while scholarship may be solitary, the rest of a humanities academic's life is often quite social. Teaching is a social, collaborative act (collaborative with students, with other faculty, or with a virtual or physical community). Publishing, speaking, attending conferences--all of that is, and always has been, a product of the social. I've been following an interesting conversation on twitter between Howard Rheingold and many others about the phenomenon of academics standing in front of an audience and reading papers--it's useful to hear their thoughts, Howard argued, but we're there for the face to face interaction. If we want the ideas in an encapsulated form, we'll read the article when it comes out. Or the book. Or what have you.

In fact, even the act of scholarship is increasingly social. My cognition extends far beyond my fingertips, into the collaborative network inside of my computer. My ability to tap into this distributed dognition will shape my ability to engage with a larger academic community. IMO, we would do well to prepare young scholars for this community.

And let's face it, most of our students won't be entering into academia some day. They'll be working in a range of jobs, many of which we can't even yet envision, but you can bet they'll require the ability to play well with others. Why should classroom engagement be any different?

106

I've been following that twitter discussion, too. My experience at conferences has been that almost everyone just reads their papers. Now, the Q&A afterwards is often productive of new ideas and new avenues of exploration.

I think what tripped me up about your comment is a basic assumption about what constitutes "real" work. There's the work that generates ideas, helps students, and creates community... then there's the work that departments and hiring committees seem to value. Even if that work is, in many senses, "unreal", it's so often the basis for one's sense of worth as a scholar, reputation in the field, and promotion. And that's the model we graduate students are being trained to follow.

The institutional pressures, though, seem unlikely to change until we HASTACers and our like-minded peers have taken over the positions of power.

By the way, I have to defend myself some (after your thorough take-down :) ) by pointing out that I'm currently at work on a tool that should, if all goes well, greatly enhance our ability to do collaborative research on literary texts.

108

I guess by 'real' I meant to distinguish the classroom work that often fails to align with the work that people do when they leave the institution--when they're no longer students. Idk...is it 'real' if it doesn't align with the practices of that discipline? Probably.

109

http://www.hastac.org/blogs/cathy-davidson/time-out-think

 

I love it that you've taken this side trail and linked evaluation to modes of presenting knowledge.  I am more and more interested in peer-to-peer pedagogy and I think it is intimately tied to presentation.  You may caught my recent blog, reprinted in a local newspaper today, about my KAIST conference in South Korea where talks were alternated with "cognitive mixers" and with strategic coffee breaks where the moderator would ensure that those arguing with one another would have a break to talk face-to-face and then represent their side-discussion to our group.  It was amazing.   http://www.hastac.org/blogs/cathy-davidson/time-out-think    Patrick Hayes (the brilliant mathematician and AI researcher, who has been there from the beginning of the Web) added a comment, indicating that people will quarrel with one another if talk is the deal but if they are tasked with creating a concept map of their fields, the same people will work beautifully together.

 

Think about the implications of this for all of our education, K-20 and lifelong!   Thanks to you all for such a thoughtful and provocative discussion!

103

Of course the notion of "real work" is a bit curious when you consider the extent to which publications are really referenced or "used."  Like the dissertation, much of our work might be described, particuarly by outsiders, as "unreal."  Like Cathy's crowdsourcing adventures (and the flak she has taken), maybe the question is how do we insinuate a more balanced perspective into our working lives?

113

These, and many related issues, can become immobilizers -- if you're constantly aware of these, and their tradeoffs, and the multiple ways in which they can be interpreted or gamed or valued by students, teachers, and administrators, and there's a pervasive sense of mutual exclusivity, you can end up overwhelmed and unable to even evaluate or assess your own choices!

In my courses, I try to do lots of pieces of different approaches, aware, as noted above by Widner, that different students have different preferences, abilities, and even ideologies.  Given that we know that, but don't know what those are for each class, I try to provide an early survey (anonymous) asking them about various course approaches (group projects? powerpoints? discussion in class? evaluating each other? familiarity with social media that I could use on the university course website system?) as well as content aspects of the course (in the case of my current course on Internet and Society, personal familiarity with various communication services, attitudes toward those, social relations about that use, etc.)  Then, as this is also a research course, I can provide class-level feedback, showing differences (many students in college, because they are very busy, and don't want to deal with social loafers, do not prefer group projects, even though I do) as well as examples of research, analysis, and interpretation, using their own responses from themselves and their class members, making it much more valid and relevant.  Further, many tend to think that their (strong) opinion is widely shared by the rest of the class. But this approach, a social approach to group norms, may show them that they hold a quite infrequent position, which might then provide a self-assessment about the vehemence of their personal opinion.  I explicit discuss this as representing peer norms and social influence, an implicit and latent social collaboration.

Available social media can be used in small portions, in a developing course-long process, to begin to engage them in a very paradoxical process.  They don't like to evaluate their fellow students -- they believe that is the teacher's responsibility and expertise -- yet they are constantly evaluating things via social media.  Mixing autonomy and structure, they can pick any website they like or feel is valuable, and post a description of that with the URL, on the class forum.  This makes it public, so they are more motivated, but it only involves a very small percent of their overall asssessment, and they get their credit just from doing it.  A follow-on assignment is more formal, yet still allows them choice -- they can pick any two concepts from the readings or book, and use those to evaluate a positive and negative of the site.  This is a private assignment, and they are provided clear criteria, and provided extensive feedback, but the assignment involves them figuring out how to apply what might seem an abstract or academic concept to their own familiar behavior. After they get that feedback, the third piece, involving more social media and public participation, is that they then go back to the public forum and comment on a given number of other people's website descriptions (a certain number above and below theirs) with some substantive comment -- not an evaluation, but a taking-it-seriously.  This is now not just public but also personal.  They learn a lot about the range of websites, from the perspective of their fellow students, and also provide (at least somewhat) thoughtful reflections, which each person seems excited to track down and read (as they do on Facebook).  This is just one of about 4 sets of similar assignments throughout the course.  So they get frequent feedback on a mix of public and private, and typically low-cost, activities, so they can play to their strengths.  I also keep track of everything, including their grades, their activities, when they hand things in, etc., and, in the aggregate, show how all these things are related (again, using research analysis in simple visual displays) to explaining their ongoing grade.  So, for instance, while everyone sees when everyone else hands in their midterm (which is about applying principles from the course), they tend to worry about whether if they are early or late if that affects their grade.  I have never found a significant correlation between when they hand in their midterm (or final) and their grade. But I do show them that engaging in a class review (using those cool feedback clickers, so everyone get cognitively and physically involved in processing the review questions), is positively correlated with their grades.

So, lots of little things, pervasive, explicit, using both class and online social media, avoiding the sense (often raised in discussions of course pedagogy, technology, and evaluation( that everything/anything is necessarily good or bad.

As an additional note, in our MacArthur Digital Media & Learning project -- Digital Ocean: Sampling the Sea -- we are exploring building in ePortfolios in the social-media based curriculum and evaluation of how students can become more engaged in learning about ocean sustainability.  Julie Robinson, a Ph.D. student funded by the project, is especially interested in how to best evaluate both the process (participation, collaboration, linkage, multi-media postings) and the various possible outcomes (awareness, knowledge, behavior, intention) of our project.

Some thoughts and an example, anyway.

111

This discussion has been fascinating so far, but I do wonder how this dicussion would look if we were to shift the focus of our discussion toward K-12 education. I know that the majority of us work and teach to university-level students, but our student enrollment is directly tied to K-12 education (and, in California at least, there is a huge disparity between primary education levels and university level expectations). As it is, there are such apparent inequalities and inequities when it comes to the distribution of resources and education throughout this country, and these are results of the problematic teaching and assessment standards currently used, which were introduced early on by the hosts of this forum (multiple choice tests, standardized exams, etc.).

How do we address the issues of teaching and assessment in the digital age when the larger institutions and bureauacracy want us to quantify that which should not be quantified--namely education and the intellectual development of our students? Among us here at HASTAC, there is an overwhelming desire to create and implement tools that these students would appreciate and benefit from, but how do we get the governing institutions, the ones that these schools are dependent on, to value these efforts? How can we change the k-12 curriculum to better serve our students? Can we implement crowdsourcing or other alternative forms of assessment to encourage process or collaborative centered learning? And how do we get the government to accept these forms of assessment?

I know this isn't the path the original conversation was taking, and sorry for derailing it, but I thought this might make a productive segue.

112

Anne,

You make an excellent point to recognize that we can extend our discussion to elementary education, or view the issues in light of a k-12 classroom. This is definitely an avenue worth exploring.

However, I also want to offer an alternative perspective to complement your own. While you question keeping this discussion about university education, and you propose considering the pre-college years, I would propose adding the post-college years. While "evaluation" takes on a literal "grading" connotation when applied to university and elementary education, the word "evaluation" can also apply to the average blue or white coller worker as well.

Perhaps this discussion could benefit by looking at concepts that can apply to pre-university systems, college education, and workplace evaluations as well--not to mention any number of additional extensions.

117

Hi, Anne--

I work with k-12 students in my research--mainly secondary ed. I think you're right that the issues are significantly different when you work with that age group, and that assessment strategies need to be different as well.

 

118

I think Jayme's opening comment about creepy treehouse assignments is worth revisiting. Real work demands real assessment; artificial work begets artificial assessment. Genuine tasks can be the basis of learning in the K-12 arena, as Expeditionary Learning Schools demonstrate. Authentic tasks are the nature of the work done post-graduation by blue and white collar workers. Authentic assessment is open to the community -- both as assessors of the work, and as meta-assessors of the process that created and assessed the work.

One example from an Expeditionary Learning School. In a 7th grade, as an approach to Washington State History (a required subject at that grade), the students in Summit School research a building in downtown Spokane, WA (and its context) and use the research to complete paperwork to put the building on the National Historic Register. A key assessment of this work is provided not by the teacher/mentor/coach but by the Federal bureaucrat that processes the paperwork. The class' ability is on public demonstration and their claim to competancy is a plaque on the wall of a building.

118

Nils made an interesting comment in his last post, namely:

"Authentic assessment is open to the community -- both as assessors of the work, and as meta-assessors of the process that created and assessed the work."

I wonder, how we could implement assessment in the university (or K–12) that is open to the community in this way. It seems like the biggest hurdle would be from instructors, who aren't used to having their work evaluated by outsiders. What would be at stake if we involved more people in the process of setting standards for student work? And how could we implement such a process?

130

John,

You say "It seems like the biggest hurdle would be from instructors, who aren't used to having their work evaluated by outsiders. What would be at stake if we involved more people in the process of setting standards for student work?"

I'd like to posit the opposite question. "What happens if we proceed 'business as usual' and don't include more assessment prespectives in in the way students learn?" There are several other hints in this Forum that Web 2.0 is about exposing the learner to feedback from their community, and that this is a new competency, that demands some change in instructors' practice.

In response to how to implement, see our work around "Harvesting Gradebook."

The source for this is Wiggins. The first quote is great in relation to the Harvesting concept.

1 “Assessment of thoughtful mastery should ask students to justify their understanding and craft, not merely to recite orthodox views or mindlessly employ techniques in a vacuum”
“Understanding is not displayed by “correct” answers to questions and problems out of context: on the contrary, misunderstanding is easily hidden behind thoughtless recall.  Mastery of the liberal arts is not mastery of orthodoxy but an ability to effectively justify one’s answers and arguments— to a real audience or client, and in specific contexts where generic answers are not adequate.” (pg. 47)
… it is the defense or discussion

2 “The student is an apprentice liberal artist and should be treated accordingly, through access to models and feedback in learning and assessment” (pg. 49)
“Any novice or apprentice needs models of excellence, the opportunity to imitate those models, and the chance to work on authentic scholarship projects. Students should be required to recognize, learn from, and then produce quality work in unending cycles of model-practice-feedback-refinement. They should not get out of our clutches until they have produced some genuinely high-quality work of their own” (pg. 49)

3 An authentic assessment system has to be based on known, clear, public, nonarbitrary standards and criteria.
“The student cannot be an effective apprentice liberal artist without models. There is no way to empower the student to master complex tasks if the tasks, criteria, and standards are mysterious (or are revealed, but not in advance). It is no wonder, then, that the ubiquitous one-shot, secure test and the often- secret scoring criteria undermine our educational aims” (pg.51) 

... There are 6 more of his points, but this post will get long if I include them.

and couple of my favorires from Bransford, How People Learn

Provide Opportunities to practice metacognition, self assessment, and reflection.
·        “A metacognitive approach to instruction can help students learn to take control of their own learning by defining learning goals and monitoring their progress in achieving them.” (How People Learn, pg. 24)
·        "Collectively, Dewey, Vygotsky, and Polyan define reflection as a process by which we think: reviewing, as we think about the products we create and the ends we produce, but also about the mean we use to get to those ends; and projecting, as we plan for the learning we want to control and accordingly, manage, contextualize, understand. We learn to reflect as we learn to talk: in the company of others. To reflect, as to learn (since reflection is a kind of learning), we set a problem for ourselves, we try to conceptualize that problem from diverse perspectives-- the scientific and the spontaneous-- for it is in seeing something from divergent perspectives that we see it fully. Along the way, we check and confirm, as we seek to reach goals that we have set for ourselves. Reflection becomes a habit, one that transforms. "  (Yancey, K.B. (1998). Reflection in the Writing Classroom. Logan, UT: Utah State University Press. pp. 11-12)

Foster a rich Context for learning
·        Build learning community: comfort with questioning and intellectual partnership.
Teachers must help students organize their work in ways that promote the kind of intellectual camaraderie and the attitudes toward learning that build a sense of community. In such a community, students/ [faculty] might help one another solve problems by building on one another’s knowledge, asking questions to clarify explanations, and suggesting avenues that would move the group toward its goal (Brown and Campione, 1994; Scardamalia and Beriter). Both cooperation and problem solving (Evans, 1989; Newstead and Evans, 1995) and argumentation; Salmon and Zeitz, 1995; Youniss and Damon , 1992(Goldman, 1994; Habermas, 1990; Kuhn, 1991; Moshman, 1995a, 1995) among students in such an intellectual community enhance cognitive development. Teachers must be enabled and encouraged to establish a community of learners among themselves (Lave and Wegner, 1991). These communities can build a sense of comfort with questioning rather than knowing the answer and can develop a model of creating new ideas that build on the contributions of individual members.” (How people Learn  pg 25)


 

109

Thanks for sharing these sources; there are lots of great ideas here.

89

(Apologies beforehand for the long post!)

Hi Anne,

Thank you for your comment and steering the discussion toward K-12 concerns. This is exactly the area I work in. I'd like to respond by pointing out some assumptions you make that I wouldn't necessarily agree with.

You write that "As it is, there are such apparent inequalities and inequities when it comes to the distribution of resources and education throughout this country, and these are results of the problematic teaching and assessment standards currently used, which were introduced early on by the hosts of this forum (multiple choice tests, standardized exams, etc.)."

I think it is important to note that the inequalities seen in our society, and manifest in our K-12 students, are *not* the product of standardized assessment (a byproduct of the standards reform movement). Rather the standards reform movement was a reaction by well-minded people to try and ensure that inequalities would be lessened. The idea of standards is that everyone knows what the standards are, and we expect that all students will achieve those standards no matter if they are white, black, or purple; poor, rich, or in the middle. The intention is noble - equality - however, as we all can see implementing standards based assessment is a tricky business. However, let's not equate standardized assessment as evil.

You also write that "How do we address the issues of teaching and assessment in the digital age when the larger institutions and bureauacracy want us to quantify that which should not be quantified--namely education and the intellectual development of our students?"

Your assumption here is that quantifying learning into measures is not productive and is quite bad... I would suggest that any kind of evaluation or assessment (whichever term we prefer) will always be some kind of measurement of learning. It doesn't matter if you use A, B, or C.... or 5, 4, 3.... or Excellent, Great, and Satisfactory... or "I was trying to do this, but I did this instead".... all of these evaluation/assessment schemes are some form of measurement. Let's not equate measuring learning as a bad thing.

I think your most interesting question is "how do we address issues of teaching and assessment in the digital age when larger institutions"... and here I am editing the question to reflect my focus... [are already invested in standards based education]?

I'll try to answer this particylar question in a roundabout way:

Now, I'm not a huge fan of our current standardized assessments in K-12 education, but there is a reason we have standardized tests. First, they are psychometrically tested (usually based on Item Response Theory) to accurately measure (as best we can) how much a student knows about a topic. Usually the test items are based on a rubric that estimates where you are on a scale of knowing something. They are quite good measurements of learning.

Second, most people would agree that standardized tests are limited. Multiple choice options might measure how many facts we know, and how well we can bubble in the answers. However, as many have already pointed out, it neglects the practices and demonstrations of learning that typify most of our real-world educational experiences.  Why does K-12 education insist on standardized tests?  Not because people are evil. It's because these tests are the most cost-efficient!! When you're evaluating millions of students, it gets kind of hard to watch a video creation of each student and give them a psychometrically-sound score of their learning. It's expensive to hire people to read a 10 page paper for millions of students in order to assess their writing ability.

So the real opportunity for folks interesting in changing assessment in K-12 using technology, is to (1) figure out what it is we really want to assess and create psychometric-based rubrics for it, and (2) figure out how digital/social technologies might improve the economies of scale needed to assess over 50 million K-12 students (according to Wikipedia) in a timely manner.

Anne, thank you for your post, and bravo to the fellow scholars. This is a great discussion and I look forward to further dialogue!

110

June, you make some excellent points about standardized testing. I do want to further press the point a little and to add to the context you present:

-You're absolutely right in saying that (standardized) teaching and assessment standards do not themselves create inequality in school systems across the country, insamuch as applied to the current argument. I will say that from a theoretical point of view standardized K-12 tests can and do contain biases which allow students from more-privileged communities to score better than those from less-privileged ones, which means they get a higher proportion of funding, which means that the rich get richer in a sort of self-perpetuating cycle that boils social-activist ire. Having said that, from practical experience, biased testing is one of the least-important worries for schools in underprivileged areas, which speaks to the size of the problems they have...

-Your second and third points I'll try to add to at the same time: basically, IMHO, although I understand Anne's issue with the bureaucratic need to quantify everything, education *requires* standards and measurement of some sort. Quantifying education and learning gives us a feedback mechanism for teachers to be able to make adjustments, for school administrators to spend funds where they are most needed, and for students to understand how and why they are learning. The methodology of that quantification needs to be improved, and you hit the nail on the head by prioritizing how to use digital technology to assess over 50 million K-12 students in a timely manner. Crowdsourcing grading, text mining, peer review...the possibilities are out there, we just have to come up with good ways to execute.

104

I'm really enjoying this conversation and feeling inspired by it.  I simply want to add that "or" is probably not operative here.  Critiquing standardized testing doesn't mean that we can't have better metrics.   Advocating crowdsourcing, contract grading, written evaluation and other forms of assessment (including self- and group-assessment, which studies show is often far more rigorous than external assessment if the forms of the assessment are set up in the correct way) is not to say we don't want standards.  Quite the opposite.  It is to say that there are forms of knowledge and standards of excellence that certain systems do not test, so having complementary systems is good.   I would not want to throw out either one, but would want a full complement of ways of evaluating excellence, in the classroom and far, far beyond.  Except for drivers' license exams, we rarely have to take multiple choice tests after school ends.   And yet we are all constantly evaluated--in the workplace, in life.  So then, why, in school, is the multiple choice form of standardized testing so much the norm?  What are we missing?  What can we be adding?  And why do we measure schooling so different than we measure life?   I don't mean any of these questions to be answered easily but all to be provocations to all of us to think deeply about what we want from education and how we want to measure what we want.

Thanks to all for this really stimulating and inspiring conversation.   By the way, many of those offering wise and rigorous comments here are students, undergraduate and graduates.   By the most traditional ways of evaluating knowledge, students are the last people who should be making these assessments.   That's the point, of course.   There are many modes of assessment that are both rigorous and non-traditional.  

124

Thank you everyone else for bringing up such wonderful points as well as insightful accounts "from the trenches"... There's already so much to reflect on and process! (but keep it coming!)

Jenna, I think Anne is steering us in a really interesting direction and I was wondering if you could talk more specifically about the K-12 work that you're doing? What are some successes or partial-successes you've acheived in that arena? Or perhaps at the very least, you could share some of the best links to participatory assessments or the 21st Century Assessment project?

Sorry to put you on the spot like this, but you guys have really been doing some fantastic impressive work! :)

 

132

Dixie and all,

I'm super excited about the work we're doing here around what we're calling 'participatory assessment'--participatory as in participatory culture, not as in participation. (We're doing a lot of thinking about the synergies and tensions between these.) I'm on my way out the door for a trip to Philadelphia for the National Writing Project annual meeting and NCTE--I'd love to connect with any HASTAC folks who will be there! When I hit my hotel in Philly I'll post more.

 

Thanks again for such a fantastic conversation!

111

[oops wrong place - moved it to *the* Jenna's post this was a reply to:)]

116

These are amazing comments! Thanks guys. June, your post was really informative. I must admit that as a graduate student in the humanities I pretty much try to avoid quantifying practices in favor of qualitative ones. What worries me are the material outcomes of these standardized tests today. When some schools within the same county don't have enough books, and others have smart classrooms in every class. And this level of disparity becomes cyclical if the former school continues to perform poorly and then are in danger of losing teachers and facilities, and the latter does better and is thus rewarded with more funding, additional teachers and smaller classroom sizes, etc.

This is also coming on the heels of the current budget crisis at the UCs, and especially letters from faculty that propose downsizing and closing weaker departments and campuses. I see a lot of similarity between the two situations and wonder how each might inform the other and how we might address these concerns. I know we won't be able to solve the California budget crisis here (it would be awesome if we did, though!) but I think it's fruitful to keep conversations going between educators of all levels. So it's very exciting to see this discussion continuing here.

And I'm looking forward to hearing some examples from Jenna!

115

I really like jennamcjenna's idea of participatory assessment, and it suits the Learning Record methodology that many instructors at the Digital Writing and Research Lab at UT Austin and elsewhere regularly use. In fact, I would go as far as to say that most instructors at the lab who make digital and collaborative work a major part of their courses tend to gravitate toward this model of assessment because it:

 

  • - focuses on process rather than product (though the products can also be fairly assessed using this system);
  • - allows students to create their own learning trajectory by collecting observations and work samples that are not limited to course assignments;
  • - enables students  to argue for their performance and grade based on evidence of that individual trajectory (which is almost always not captured by assignments alone);
  • - is guided by a set of learning dimensions (confidence and independence, skills and strategies, knowledge and understanding, use of prior and emerging experience, reflection, and creativity) that can be applied to any learning situation, and that can be grounded in specific learning objectives and grading criteria developed by the instructor;
  • - gives instructors rich feedback on the effectiveness of particular assignments or classroom practices.
Many posts in this terrific thread suggest that to truly learn we are pushed to ask students (and ourselves) to engage in networked environments and to become active in communities that we do not control, or even completely understand. In the spirit of participatory culture, assessment then shifts from how "we" assess "them" to how learning is a student-teacher collaboration, a truly participatory event. The Learning Record seems to be a means of scaffolding the kind of participatory dynamic that Jenna mentions in her post.
Following this line of thinking, assessment ceases to be a one-way flow, and opens up intriguing ways of thinking about the role of assessment in our daily work. My own observations of the dynamics in the DWRL suggest that the work instructors do in the classroom bleeds into blog posts, white papers, workshops, and other forms of community activity. These, in turn, guide and form individual and collaborative research projects. And much of this (net)work is grounded in a common assessment model.
All this leads me to a question I'd like to post to this forum: to what extent do assessment practices guide our evolving understanding of digital learning and frame our research questions?

 

103

I haven't used the learning record yet as an instructor, but I have used some concepts from it. I intend to use it whole hog next semester. I have used it as a student and one of the things I didn't initially get, but do now, is the literacy narrative aspect. The way it was implemented, the first part of the process is creating a sort literacy narrative by interviewing someone about and reflecting on your experiences with reading, writing, speaking, listening and whatever other course strands are developed by the instructor. This allows each student to develop a baseline. Furthermore, these individualized baselines don't force the instructor to evaluate the entire class by the same baseline, which--given the disparities among the secondary school systems--has become impossible anyway.

106

These are amazing comments! Thanks guys. June, your post was really informative. I must admit that as a graduate student in the humanities I pretty much try to avoid quantifying practices in favor of qualitative ones. What worries me are the material outcomes of these standardized tests today. When some schools within the same county don't have enough books, and others have smart classrooms in every class. And this level of disparity becomes cyclical if the former school continues to perform poorly and then are in danger of losing teachers and facilities, and the latter does better and is thus rewarded with more funding, additional teachers and smaller classroom sizes, etc.

This is also coming on the heels of the current budget crisis at the UCs, and especially letters from faculty that propose downsizing and closing weaker departments and campuses. I see a lot of similarity between the two situations and wonder how each might inform the other and how we might address these concerns. I know we won't be able to solve the California budget crisis here (it would be awesome if we did, though!) but I think it's fruitful to keep conversations going between educators of all levels. So it's very exciting to see this discussion continuing here.

And I'm looking forward to hearing some examples from Jenna!

103

ohhhhhh, you guys, i'm so sad that I can't participate more in this conversation! brb retroactively rescheduling previous commitments

 

 

I've been talking about assessment and new media and writing at the National Writing Project annual meeting all day--I'm gonna have tons to report back as soon as I have a second to breathe!

105

Oops, I was having trouble posting, and that showed up twice. Sorry.

106

While not wanting to seem too much the McLuhanist of the group:) I'm very interested in what web 2.0 technologies are proposing as far as structuring systems for metrics and assessment - and how these may support an enhancement of self-assessment in the learning process. 

In a Special Interest Group during a IEEE conference, I heard Julita Vassileva (University of Saskatchewan) present a model that, at the time, was ground breaking and (I think) still is: a classroom CMS by the name of Comtella. 

This was the first CMS / social network / online learning environment I saw that *aged*. Time was part of the complex algorithm ruling the way students’ paper contributions and overall activity was graded. In particular - and I will mention this because it is very simple idea to grasp - the system rewarded quantity per se while the community was *young*. As the community *aged* though, quality rather than quantity, was rewarded, Indeed, quantity with no attention to quality (this included non pertinence to the subject at hand) could be 'punished', lowering the participant's score. The logic? Quantity per se (for the sake of increasing grade… ) increases ‘clutter’ and difficulty in finding rich resources. 

The same system also offered innovative visualization forms of individual, team and overall community participation and learning, as a way to promote reflection on individual contribution. 

Besides wanting to acknowledge the uniqueness of the Comtella concept - the recognition of ageing as integral part of the learning experience (necessarily fast-forwarded and accelerated for the duration of a class) and of Vassileva's writings on this matter (list at the bottom), I would also love to hear about similar systems…  CMS and social networks systems that: 

-        embed maturity in their design (maturity as in adulthood - growth in wisdom:)

-        return participation to the learning community in visual, impacting, ways

In all that I write here, there is of course the implicit idea that it is difficult today to design a learning system that does not include web 2.0 in assessment/evaluation in some way shape or form.

This area may be known to some of you as adaptive reward mechanisms.

Vassileva's writings on this topic (and as I have done in the past, I'll try to bring her to the discussion:)):

Vassileva, J. (2009) Towards Social Learning Environments, IEEE Transactions on Learning Technologies, 1 (4), 199-214.

Vassileva J., Sun L. (2007) Using Community Visualization to Stimulate Participation in Online Communities.e-Service Journal6 (1), 3-40. (extended version of Vassileva and Sun, Proc. CRIWG'2006, Medina del Campo, Spain).

Wang Y., Vassileva J. (2007) Toward Trust and Reputation Based Web Service Selection: A Survey. In International Transactions on Systems Science and Applications,3 (2), 118-132. (invited paper in the special Issue on "New tendencies on Web Services and Multi-agent Systems (WS-MAS)").

 

 

112

This thread has been focused on at the level of assessment of student work. WSU has been using its Harvesting ideas to think about other community-based assessment activities. Assessment of the assignment that prompted the work and assessment of assessment activities by academic departments, for purposes of creating a system of assessment for the University's accreditation. In each case, community is being invited in, both to contribute assessments and to see the assessment work underway.

This is an invitation to try our beta version of assessment of program self-studies used in University accreditation.

To participate in the test, go here.

It will likely take you 30 minutes to look over the rubric (inspired by on the transformative assessment rubric), read the self-study written by the fictitious Departent of Rocket Science, and score the self-study using the online survey version of the rubric.

Your data and your feedback will help us refine this process, still under development at WSU. 

Thanks

101

Partly inspired by this discussion, my co-editors and I at the journal THEN have published an essay by Fred Goodman that suggests that the problem isn't just who does assessment and how, but the idea, implied by the very structure of school assessment, that every problem has exactly one correct (or best) solution.  In other words, school is most often like a puzzle, bur real life is more often like a game.

107

Thanks for posting the link to Goodman's essay. It made me think of a game that some of my colleagues at the DWRL have developed over the past few years, Rhetorical Peaks. Rhetorical Peaks is a cross-platform game designed to introduce first year writing students to basic features of rhetoric, such as the use of evidence and the different forms of argument. The game presents students with a mystery—who killed a popular teacher in the town of Rhetorical Peaks—to which there is no "correct" solution. Students explore the game environment in order to discover clues and gather information, but at the end they don't find the answer, they must make an argument for what they think is the best solution to the question of who committed the crime.

113

This line of convesation reminds me the BioQUEST curriculum project. A founding article of that group is 3P's Approach to Science Education: Problem-posing, Problem-solving and Peer Persuastion

Here is a summary of the argument

Problem Posing

To understand science as it is practiced, rather than solving already well-formulated problems from a textbook, students must be engaged in problem-posing. To appreciate this, students must learn that they could stand in the field or laboratory forever and no problems would come to them pre-posed. In BioQUEST (Quality Undergraduate Education Simulations and Tools in Biology) we try to help students understand the multiple issues involved in the posing of a problem, including “interestingness”, significance, and feasibility, as well as the problems resulting when bias creeps into problem-posing.

Problem Solving

After having posed a problem, students need to experience open-ended problem-solving. Real scientific problems do not have answers at the back of the book. The scientist entertains multiple competing hypotheses and makes inferences over a long series of experimental observations. Scientists do not arrive at a final answer; research is abandoned for a variety of reasons, including time, resources, and most importantly when the scientific research team is “satisfied”, i.e. when the solution is “useful” for some purpose. Students can, and must have this problem-solving experience to appreciate the nature of scientific answers and to develop heuristics for achieving closure to scientific problems.

Peer Persuasion

Research is not complete, no matter how many experiments have been conducted, no matter how many puzzles have been solved, until peers outside of a research team are persuaded of the utility of the answers. Persuasion is a social process and an essential one for students to experience in order to understand the nature of scientific theories and paradigm shifts. Therefore, they need to experience peer review as a professional activity. The modules in The BioQUEST Library allow student groups to easily transfer their data, graphics, working hypotheses, and analyses into word-processing, spreadsheet, and scientific graphics software to build scientific journal-style manuscripts which can be reviewed by other students and instructors.

137

[* added: folks - for the sake of scrolling space :), I deleted 1 image - you have the pdf reference anyway]

A follow up to my previous post on communities that incorporate aging or maturity: 

I did invite Julita Vassileva to join this discussion all the way from Saskatchewan, but she met with some problems with registering so I will just quote her email.

From Julita Vassileva: 

"A newer, prettier version of the system [Comtella] used physics-inspired mechanism, with an energy metaphor (different from the adaptive rewards mechanism used in the previous version). It was a discussion forum, and people were not rewarded for posting new posts but for rating them. If they rate up, they add energy to the post, or “heat it up”, if they rate down, they “cool it”. 

The reward for rating was just an animation showing a change of colors, ending with the post displayed in brighter, warmer color or cooler color.  Similar to the version of Comtella that you know, it doesn’t make sense to rate posts in a dead community, i.e. a community without any new posts.

To regulate this, we did the following: there is a certain amount of energy in the system, which depends on the number of new posts.  Each new post injects a certain amount of energy (e.g. 5 units), of which 3 are used to display the post at standard “heat” level, and 2 go in a pool of energy. When people rate posts, they use the energy in that pool, for each rating, one energy unit is taken from the pool and added to the rated post, increasing its energy /heat and making it brighter and warmer. Reversely, when rating down, one energy unit is released from the post and added to the pool. The system looses energy with time, and everything gets “colder”, if there are no new contributions.

Source of images:          Webster, Vassileva Visualizing Personal Relations in Online Communities, Proc. AH’2006. http://julita.usask.ca/Texte/AH2006.pdf

One more reference: 

Cheng R., Vassileva J. (2006)Design and Evaluation of an Adaptive Incentive Mechanism for Sustained Educational Online Communities UMUAI, 16(2/3) 

http://www.springerlink.com/content/t477ngk0wl641612/fulltext.pdf

106

Please send Julita this email address:  Nancy.kimberly@duke.edu    Nancy will work with our team to get her registered.  Because of the professional (human not bots:  that's how respected we are!) spammers who falsely register in order to be part of our list, we're having to set the filter registration points higher than we'd like.   We're happy to help people register.

108

Please send Julita this email address:  Nancy.kimberly@duke.edu    Nancy will work with our team to get her registered.  Because of the professional (human not bots:  that's how respected we are!) spammers who falsely register in order to be part of our list, we're having to set the filter registration points higher than we'd like.   We're happy to help people register.

111

Thank you all so much, for such amazing and thoughtful comments, questions and ideas! My brain is on fire just thinking about how to construct my own class next semester. i have one question that I haven't seen addressed here yet, and would love some input and thoughts, and seems a bit in line with Nils' "assessing the prompt" line of thinking above.

I call it the "chili pepper question" -- what are your thoughts around more public "rate your professor" sites (like Rate My Prof) as well as more internal professor-ratings sytsems? I know that my university has moved to include student responses online, and many many other schools have their own internal professor-rating systems made available to students. It used to be that we'd fill out multiple-choice questions on a single-sided piece of paper, on the last day of class -- of course, this resulted in students rushing, writing a few lines, and running out to celebrate the end of the semester, leaving very little valuable information. Now many of these review systems are moving online, and allow students to save their responses, and write lengthy and detailed assessments of the class and the professor.

Have these teacher-assessment tools been useful to you, as professors, instructors or students? Are there particularly useful questions you've encountered, that tease out student responses in interesting ways? How do you determine how the class is going during the semester, other than intuition of either excitement and feeling connected, or that sinking feeling (that we've all experienced, I'm sure).

One thing I learned after my first teaching experience was to do my *own* version of the teacher-assessment several times a semester, or at the very least, once in the first few weeks, and once at mid-term. Then there's time to make any corrections, and not just learn for the next time. I've also started to devise very small "mini-responses" to keep tabs on how different formats or texts are working. Little assessments that take maybe 1-2 minutes at the end of class, and can be aggregated to get snapshots throughout the semester. Otherwise those huge "class reviews" delivered at the end of the semester are simply overwhelming and a bit intimidating, especially to young instructors who've had zero formal pedagogical training and are trying to figure out how to teach as we go.

Any thoughts?

On a side note - I know there are Scholars out there who have ideas and comments but are a bit intimidated to jump in here, and I'd like to encourage you all to post your questions. The folks here are interested in all kinds of teaching and student assessment experiences - your comments will be warmly received! Do it! :-)

Thank you all again for a wonderful and thoughtful forum - I'll be bookmarking this as a reference in many semesters ahead!

103

RateMyProfessors.com is a really interesting example, because it's not run by any university. In fact, it's part of mtvU, whose goal seems to be to commercialize everything about the so-called "college experience," this monolithic thing we're all supposed to have gone through between the ages of 18 and 22. Politics aside, it's interesting how mtvU is not only defining the culture of undergraduate life, but wedging itself into the classroom through things like RateMyProfessor.com and its corollary (cue intimidating music): PROFESSORS STRIKE BACK!

I just used the verb "wedging," but I don't think this is necessarily a bad thing. Students seem to speak more freely on RMP, giving the professor better feedback, and it helps students know what they're getting into when they sign up for a course with someone who has a reputation. I'm wondering if any teachers have used it, and if so, how it's changed their teaching style.

I'm also curious what others think of the "PROFESSORS STRIKE BACK!" videos. Some of them show venerable scholars acting, ahem, rather undignified. Humanizing professors seems a good thing; playing out private classroom disagreements in a commercialized public forum, though . . maybe not so much.

 

118

Yes, RMP is a great example and it caused rather heated discussions on my department's listserv a few years back.  To quickly summarize: there was some talk that the professors'/TAs'/adjuncts' teaching evaluations would be made public to all students, a suggestion that didn't fly to well with the faculty.  (Personally, I was - and still am - in favor of this transpaency.)Many believed that since faculty has access to students' records (we can view transcripts to verify their placement in courses), students should also be able to asses our true teaching records, not just by "chili peppers" and "ease rankings" that are posted on RMP.   If we're evaluating the students' performance, why can't they evaluate our abilities (or in many cases, inabilities) to actually help them perform? Aren't our evaluations equally as critical?

I guess there's a fine line between how we react to these evals.  I think it's a fair assumptoin that junior faculty, TAs, and adjuncts will be more (re)active (not reactionary!) in response to their evals mostly because our own evals for promotion, tenure and job securtity are partially based on the comments and rankings.  But what about the tenured profs that don't care about teaching anymore? (Okay, I know that's harsh and overgeneralizing, but we all know there's at least one in every department.) Is our reaction to poor evals similar to our students' reactions to their poor grades? Do we change or not care?

Back to my original point: the relation between the private and public eval.  Would publicizing teacher evals (the "official" ones, not just RMP) increase student productivity or interest in their education?  I'd like to think it would, but it wouldn't dismiss the hits on RMP (which I do think is as close as many schools are to transparent teacher evals).

As a final thought, there's an interesting article in The Chronicle this week about Tweeting about presenters at conferences.  We, too, evaluate each other publicly.

Great topic--thanks!

101

Check out Seven Principles for Good Practice in Undergraduate Education, Chickering and Gamson (google search). There is a student inventory, a faculty inventory and an institutional one. Assess your perceptions of your teaching and ask your students to assess you. Compare. You might also ask your students to what extent they percieve your course is achieving the goals you set out in the syllabus.

107

Here's a very specific, perhaps even excessively applied question.  I have looked through some of the generously offered links so far (though I'm thinking I might offer an exploration and then integration/distallation of all the linked material as an independent study for some undergrad student interested in learning more about all this!) and haven't found something that might speak to this directly.

As I noted above, we are fortunate to be working on a MacArthur Digital Media & Learning project -- Digital Ocean: Samping the Sea -- which is developing a small set of curricula to be used by classes (grades 5-12) around the world (through ePals, but mostly in the US to start) on ocean sustainability, emphasizing the role of using social media (among teachers, among teachers and students, among students, within and across classes). 

Now, there's already a lot of literature out there on social media specifically, and, as I noted above, on using instructional media (esp. computer-mediated communication) to enable and foster more participative and collaborative activities within and outside the classroom.  And I am definitely working with that, and have a number of my own ideas.

But the question to folks here, given the very appropriate title of this forum, is what are some good ways of assessing the use and effectiveness of social media in such a learning environment.  And this might be organized, initially along (at least) four dimensions: assessor (teacher, student), assessee (teacher-teacher interactions, teacher-student interactions, student-student interactions, within-across classes), level of social media use [of course this would also vary by type of social media use, but I'm trying to keep it constrained here], and course goal (ocean sustainability awareness, intention toward action). 

So, for example:

students, assessing teacher-student interactions, accomplished through high levels of social media use, on awareness of ocean sustainability issues.

Given upwards of a 1000 students in upwards of 200 classes, this also has to be *somewhat* manageable (as you all know, teachers have inexhaustible time and energy...).  We are considering a wide range of qualitative and quantitative approaches (they can and should be complementary) including ePortfolios, textual analysis of blogs, surveys, and system usage measures, but want to make sure we emphasize the social media role.

Any suggestions would be appreciated!

113

This might be a slight departure from the main thrust of the discussion above, but I think an interesting one. One model for education that I've always had in the back of my mind as it is something that fascinates me and inspires much of my thinking on the topic, is the model presented in Neal Stephenson's Diamond Age.

Stephenson presents a world in which Nell, the main character, is essentially raised and accompanied through her life by a semi-artificially intelligent book backed by a human voice actor. The book weaves Nell's experiences into this continual narrative and teaches her everything she might need to know about people, self defense, science, and so on. Towards the end of the book, Nell is united with a large army of refugee girls who continually appeared mediated through Nell's book as an army of mice. Each of those girls in the real world possesed a similar book driven entirely by AI. The paradigm essentially was completely automated and distributed, infinitely scalable education through technology. A large group of refugee girls raised into an organized, adept group by a program. That is a paradigm I've been wondering about in the context of this discussion.
This model is one that is less about the pedagogy of the instructor than much of the earlier posts--there being no instructor--but one I think is worth thinking about in terms of the implications it holds for the scalability challenges that plague education in general as a global institution. Do we think that it would be possible to conceive of effective pedagogy for a scenario that did not involve an instructor? Can one create an object of knowledge that can encourage engagement without systemic compulsion (gaming comes to mind but not exclusively)? Can one create appropriate relationships between some independent artifice and a student's personal intellectual development? In what ways can a non human assess? I particularly mention it here because of the foil it presents for the notions of grading heretofor discussed.

 

I will note that primarily running through my mind right now are algorithmic methods of evaluation (I study lots of machine learning/NLP stuff) but I do not mean to limit the idea to that. If assesment is primarily about giving a student feedback and an opportunity to learn through practice and experience, what is changed about that transaction when the feedback is not of a human passing judgement but an algorithm evaluating or a student self evaluating? I noticed in the posts above a certain tension of students wanting to figure out what their instructors expect that I think would certainly be radically changed in the contexts I am imagining. The real question is what does the grade represent when it is not a vector of institutional symbolic power. Students always do their work at least on some level because not to work is to be graded poorly and that carries consequences. If one is self (personally, algorithmically, etc) evaluating and the grade is not something that can never be systemically dangerous, but could perhaps be demoralizing, could drive someone away from the activity, it is hardly recognizable as what we have been calling a grade. At this point I'm just wandering so I'll stop. If anyone has any thoughts on the matter, I'd love to hear them.

136

Fascinating. Makes me want to get into an extended seminar with you Evan. You have great questions.

"Can one create appropriate relationships between some independent artifice and a student's personal intellectual development? In what ways can a non human assess? I particularly mention it here because of the foil it presents for the notions of grading heretofor discussed.

 I will note that primarily running through my mind right now are algorithmic methods of evaluation (I study lots of machine learning/NLP stuff) but I do not mean to limit the idea to that. If assesment is primarily about giving a student feedback and an opportunity to learn through practice and experience, what is changed about that transaction when the feedback is not of a human passing judgement but an algorithm evaluating or a student self evaluating? I noticed in the posts above a certain tension of students wanting to figure out what their instructors expect that I think would certainly be radically changed in the contexts I am imagining. The real question is what does the grade represent when it is not a vector of institutional symbolic power."

On the "independent artifice" I have been wondering if "automated metatagging" could help make bridges between natural work and external ontologies of importance to various communities. The burden of linking would be an automated matching to whatever ontology was of interest and then other automated processes could dive deeper into the data for patterns, meanings, etc. See (http://www.semantichacker.com/) for an early stage results engine - paste in a web site or some text, etc.

I can think of several ways that automated assessment might evolve, perhaps in all directions at once. Combining auto-tagging with dynamic directed graphs give things like http://farm2.static.flickr.com/1126/1022833318_057b332f30.jpg and http://www.kartoo.com/ (put in "grading 2.0" and click the graphic dis;play layer) (imagine applying these to a person's working portfolio rather than a web site or the whole Internet). Another pathway I mentioned earlier is through neural net analysis of performance within a constrained digital space where a lot of other people have also performed and there is differientiation among novice and expert performances. I've also wonder about applying fuzzy logic concepts (which I guess might be similar to the Bayesian analysis that we've tried on rubric scoring. There is a great blog post with a variety of visualizations that a graphic guy shared with me a while back (2007) see http://www.smashingmagazine.com/2007/08/02/data-visualization-modern-app... ...and there is Many Eyes (http://manyeyes.alphaworks.ibm.com/manyeyes/) that has some ideas for us.

Concerning your excellent question about what does feedback mean when it is "not a vector of institutional symbolic power." A colleague of mine had students use an automatic essay grader that gave specific feedback about how to improve a writing sample to meet a rubric's objectives and found that they fell into 4 categories (I'll have to dregde up that work - from Sara Dexter now of UVA: ( people who used it a lot and got a lot better - and believed that a machine could provide good advice ) ( people who used it a lot and got better - and believed that a machine could not possibly help a human ) etc and people who used the scorer then had more expert case study searches and analysis and also got higher scores when humans scored the final pieces...so at the very least for some people, practice and feedback with no institutional hammer over their head did lead to more practice and better writing regardless if they believed a machine could be an assessor or not.

109

Evan,

You said two things: 'I noticed in the posts above a certain tension of students wanting to figure out what their instructors expect that I think would certainly be radically changed in the contexts I am imagining. The real question is what does the grade represent when it is not a vector of institutional symbolic power.'

I also have tried automated methods for grading short texts. Basically, it was a primary trait analysis that I tuned for each assignment. It gave a good first cut at sorting students. It did not provide feedback that would help them learn.

I'd like to get rid of students figuring out their instructors. Posting a rubric of the assessment criteria along with the assignment can help. Asking, as we do in the harvesting gradebook examples, for the person doing the review to also review the rurbic (and subsequently (but not in the particular moment) revising the rubric in response) makes the assessment criteria authentic as well. Using multiple raters, and establishing agreement among them, further moves the coversation about the meaning of the rubric into the realm of community property. Getting academic programs, not individual instructors, to use the same rubric, with inter-rater agreement, offers the potential for the student to have a much more coherent learning experience.

And let's seperate the grade from the assessment and feedback. Here we demonstrate a harvesting gradebook that gathers rubric based feedback and comments from multiple assessors (peer, faculty, industry) and provides it to the student in (near) real time. We also wrote an algorithm that takes all the numeric data and boils it down to a letter grade. The grade is a summary that is required by the university. Its not a tool to help the learner learn.

What is interesting about a grade produced in this fashion is that it has the potential to be validated and reliable because it is supported by a mechanism that can be validated and made reliable. Again, if this were practiced program-wide it would make the grades much more valuable as a summary of a student's experience that the current individualistic grading practices.

 

130

Grading and Assessment (highered.blogspot.com)

135

I want to respond to a couple threads woven into the above discussion.  I am teaching a Writing 2 course here at UCSB this year, and I have taught introductory Writing before at my MA institution.  When I first started, we were made to use  the portfolio system under the idea that process was more important than product, which, to a certain extent I totally agree with, only under the condition that the actual product is not de-emphasized.  In my new class, I've tried to evolve the portfolio slightly.  I've used USCB's moodle-like website Gauchospace for coursework.  Each of the students has a blog where they post the majority of their work, which acts as a kind of "virtual" portfolio.  Because I'm experimenting with this, I'm determined to hash this out so as to make it even more effective!  I'm intrigued by this idea of a "virtual" portfolio, for a couple reasons. 

First, it actively tracks the process of paper-writing, especially when the particular assignments are geared towards the actual paper itself (i.e. I have broken up the class into several units, and through each unit we slowly put a paper together, as opposed to more random assignments that teach various writing skills, such as thesis, transitions, logic, outlining etc.).  There's little difference here between an analog portfolio, except that the digital portoflio is less messy, and paperless.  Second, via this process, the students understand what, to be frank is one of the most important lessons college students can learn, good writing is not produced in one draft, written the night before the paper's due.  The tracking process shows the processual nature, the labyrinthine, forward-and-backward- higglety-pigglety nature of composition, which shatters the illusion about the perfect, mythological A student I imagine they imagine, able to sit down and churn out a perfect A paper.  Third, because the blogs are online, they can be actively read by other students in the class, outside of and at any time by the students, and commented on in process, taking advantage of the affordances of online media.  (I'm interested in how else this can be expanded.)  Fourth, and most importantly, the formal material of the blog informs the composition paradgim the students work with.  In contrast to paper and ink hard copies, which suggest a kind of concreteness, an inevitability with regards to the final imaginary paper the student has in mind, the "virtual" paper remains more flexible, based simply on the fact that it can be so easily changed materially on the website.  Because it isn't analog, paper, or, "real" yet, in the form of a printed paper, the "virtual" paper accurately reflects the processual paradgim the portfolio hopes to enact.  My questions here are: how else do we take advantage of this virtuality in composition?  How are computer labs appropriate "spaces" for particpatory composition?  What practical assignments and exercises have you used with these new media tools?

Finally, I know this is dragging on, but I wanted to put in some thoughts about evaluating creativity.  I was most interested in the question of creativity and what can be graded as posed at the opening of the forum.  The link posted by stanislavzza above actually responds to this question quite well.  I always encourage my students to take calculated risks (the fear being that an experimental move in their composition will fail and result in a lower grade, thus the less risky move is to produce convetional material).  But I think the best evaluation they get on these creative experiments is from their peers.  This is actually where "grades" as measures of excellence, are totally inefficient.  A grade will tell a student nothing about how something creative worked.  The best response must be in writing or oral, not too short (wow!  that worked!  great job!), and not too long (a complete analysis and breakdown), but something which, especially gauged by several to many different reactions, can give the student an appropriate evaluation.  For example, if a student wanted to weave into their conventional argument a personal experience, with the purpose of having that experiene resonate with the topic, but not explicitly explaining how it does so, the best way to evaluate this is to "test" it on different students, rather than have me, the teacher, slap a grade on it and give my institutionally informed remarks.  Perhaps what the student really learns is less whether or not a creative move like this worked or not (in the sense that it resulted in some grade), and more that experimentation, trial-and-error, and play are truly productive compositional methods.

109

I also appreciate the great experiments folks are sharing.  I appreciate the detail and thoughtful approaches to designing the various activities students are engaging and the technologies being used to support them.  It is also clear that the general consensus of this discussion points to a general inadequacy of traditional approaches to grades.  But I would still appreciate learning even more about how this group is assessing their experiements--not just to answer the simple question about whether they are "working," but to understand how this talented group is assessing (or plans to assess) their innovations. There is much we can learn about how the many efforts outlined in this forum are influencing or changing students' learning processes or learning outcomes.

104

Thanks for sharing your process for your UCSB writing course. The process you shared via blogs sounds really participatory, collaborative, and a great opportunity to encourage engagement in the writing process. Not to mention, less waste of paper! Throughout my graduate career, I have talked to other graduate students teaching writing. We discussed the possibilities of implementing new media sources, like blogs and social networking sites, and your process serves as a great model.  

I am also interested in the question of creativity. And it seems as you suggest, that creativity can be fostered through the writing process online. This happens as the students have a online workshop  space, where they can get feedback from their peers. I also feel that it also speaks to accountability, knowing that peers will read their work, may make writing more meaningful--although  the professor is important as well! but knowing you are not only writing for one person but  having a peer-based audience seems like an exciting and vital innovation in writing courses. 

What came to my mind are traditional writing workshop spaces that are very common in creative writing classes. In these workshops, students bring in a story or poem and receive feedback from fellow peers and professors in the class. I feel the innovation you've suggested parallels this process via online, and would also be able to work with all types of writing, including creative.  

I also wonder if the process you utilize also incorporates physical or in-person interaction?  How might a writing/revising process be different via a mixed methods approach of both online and in-person interaction? Regardless, I feel the process you've suggested fosters community building, creativity, and collaboration something that does not happen often in the university writing classroom. 

Thanks for sharing your process! And it seems that your students are very fortunate to learn through your commitment to innovative teaching!  

 


 

104

Thanks for the comments!  As for some of your questions: whether or not the review "incorporates physical or in-person interaction"- absolutely!  The problem is how to appropriately and effectively synthesize the two.  We have to imagine that our students are digital social network-savvy (not in all cases of course, but progressively this will be the case), thus able to easily bridge the one experience (physical, in-class) with the other (online).  In fact, I propose that this balancing act can be even more productive because it allows people to be more plastic, moving from a medium they may feel less comfortable with to one they feel very comfortable in.  This can go both ways.  "Shy" people in person can open up online, and extroverts can open up in person, and experiment with a more "introverted" medium online.


What I really want to explore though is how these open peer review sessions would work in a digital social network environment.  One example, which has less to do with undergraduate composition and more to do with scholarship, and as many of you may be aware of, is the MediaCommons project with the Institute for the Future of the Book at USC: http://mediacommons.futureofthebook.org/

This project wants to make transparent the process of academic scholarship, from idea, to revision and review, to publication.  Avi Santo from the project writes,

"a digital scholarly network would foreground the process and progression of critical thought over its finished product. It would do so by allowing scholarly work to be written in "real time," meaning that the development, drafting and revision of critical writing would happen in public. This would illuminate what is often a shrouded process for students, who are usually only presented with finished works to digest, but not the mechanisms for their creation to both learn from and evaluate. By continuing to value the current closed "peer-review" process and the romantic idealization of a secluded writing practice, media scholars fail to engage with communities that align credibility with the foregrounding of on-going and cumulative process rather than finished product."

I think this is quite constructive obviously as a graduate student having to negotiate the mysterious, clandestine and labyrinthine world of academic publishing, but also as a model for undergraduate composition and, especially with regards to evaluation.  I mean, this is the heart of evaluation, as opposed to grading, which is collaboratively attemtping to develop certain ideas and skills, rather than "grade" them.

109

As a student, I can relate to evaluating my professors since I just had to go through this process for the end of the term. This year my school switched from the fill in the bubble sheets to electronic surveys that could be done at an individual’s convenience. However, to make sure students actually evaluate their professors, grades and transcripts will not be released until the evaluations are completed. I think it is better that my institution is using electronic evaluations because I  will admit that I would usually rush through the paper fill in the bubble sheets, as well as feared the professors would recognize my handwriting from the comments section.

As for RMP, I have never used this site. I think that since I go to a smaller school, word of mouth is the media to share information about professors. Friends tell each other if a class or professor is good, as well as talk about difficult courses. However, as I start thinking about possible graduate schools, where my choice of an advisor is really important, I am considering searching this program or a similar site for information about faculty of the programs I am interested in.

In response to Kim Lacey, I am not sure I would like access to teacher evaluations or would want my feedback available to the student body. I usually give personal, thoughtful feedback because I know that only my teachers and administrators will see the comments. Also, many of my professors only shared grades with individuals rather than posted scores. I think that it would not be fair for teachers to have their evaluations open to the student body if they don’t share class grades publicly.

I think one of my biology professors had one of the most effective ways of receiving feedback. Along with her exams, she passed out evaluation forms and asked for feedback about the course. This allowed her to find out how students felt about her teaching and to determine positive aspects of her lectures as well as improve things (ex. slowing down if people felt they couldn’t keep up). Instead of just finding out everything at the end of the semester, she was able to adjust her teaching style throughout the semester. Perhaps multiple, small evaluations could be given through digital media, such as a SurveyMonkey.com form, to help teachers improve their methods. The only difficulty in this would be giving students an incentive to complete these evaluations, since many have busy schedules in which giving a teacher feedback is not a high priority.  

100

This is a very thoughtful perspective, and it is good not to have only graduate students comment.  So thanks very much.  I like the idea of the prof who hands out evaluation forms along with exams throughout the course, and I know more and more profs who hand out mid-course evaluations, not to see "how they're doing" but to see how they can "do it better."  If the best students think the material is coming too fast, then a great teacher listens and adjusts to that kind of feedback.   Thanks for contributing to this pretty amazing conversation, Claire

144

Dear Cathy,

Thank you for your support. Sometimes I am uncertain about what to write since discussions can move towards topics or issues I do not understand or relate to as an undergraduate. I appreciate all of the help and positive feedback from you, my HASTAC mentor Erin, and other HASTAC scholars. I am glad I have the opportunity to interact with so many different people. I enjoy sharing ideas in a place where I do not have to worry about criticism. These things motivate me to remain involved with HASTAC and to learn more about the ethics and uses of digital media.  

102

Dear Claire, I think people will be happy to disagree with you if they feel the need----but your comments are thoughtful and no one can criticize that even if they disagree with your points.  We love having undergraduate voices and pov's.  Thank you!  Best, Cathy

113

Dear Claire, I think people will be happy to disagree with you if they feel the need----but your comments are thoughtful and no one can criticize that even if they disagree with your points.  We love having undergraduate voices and pov's.  Thank you!  Best, Cathy

116