Monday, January 28, 2008

How Should Colleges Assess and Improve Student Learning?

First, a few caveats.

  1. I don’t believe that all we’re about in education is preparing future employees.

  2. Just because Business says something is good doesn’t necessarily make it so.

  3. I don’t know enough about the methodology of this survey to judge bias.

Okay, now that we’ve got that out of the way, I found How Should Colleges Assess and Improve Student Learning? (pdf) rather interesting:

From November 8 to December 12, 2007, Peter D. Hart Research Associates, Inc., interviewed 301 employers whose companies have at least 25 employees and report that 25% or more of their new hires hold at least a bachelor’s degree from a four-year college. Respondents are executives at their companies, including owners, CEOs, presidents, C-suite level executives, and vice presidents.

So they polled these executives about the quality of college graduates and what they’d like to change, particularly in the area of assessment. Here are a few excerpts:

Employers are satisfied that the majority of college graduates who apply for positions at their companies possess a range of skills that prepare them for success in entry-level positions, but they are notably less confident that graduates are prepared for advancement or promotion. While recent graduates are seen to demonstrate solid skills in the areas of teamwork, ethical judgment, and intercultural skills, employers are less convinced of their preparedness in terms of global knowledge, self-direction, and writing.

When it comes to the assessment practices that employers trust to indicate a graduate’s level of knowledge and potential to succeed in the job world, employers dismiss tests of general content knowledge in favor of assessments of real-world and applied-learning approaches. Multiple-choice tests specifically are seen as ineffective. On the other hand, assessments that employers hold in high regard include evaluations of supervised internships, community-based projects, and comprehensive senior projects.

Employers’ emphasis on integrative, applied learning is reflected in their recommendations to colleges and universities about how to assess student learning in college. Again, multiple-choice testing ranks lowest among the options presented, just below an institutional score that shows how a college compares to other colleges in advancing critical thinking skills. Faculty evaluated internships and community-learning experiences emerge on top. Employers also endorse individual student essay tests, electronic portfolios of student work, and comprehensive senior projects as valuable tools both for students to enhance their knowledge and develop important real-world skills, as well as for employers to evaluate graduates’ readiness for the workplace.

And, from the “Key Findings”:

. . . Most employers indicate that college transcripts are not particularly useful in helping evaluate job applicants’ potential to succeed at their company.

. . . Few employers believe that multiple-choice tests of general content knowledge are very effective in ensuring student achievement. Instead, employers have the most confidence in assessments that demonstrate graduates’ ability to apply their college learning to complex, real-world challenges, as well as projects or tests that integrate problem-solving, writing, and analytical reasoning skills.

. . . Employers deem both multiple-choice tests of general content knowledge and institutional assessments that show how colleges compare in advancing critical-thinking skills of limited value for evaluating applicants’ potential for success in the workplace. They anticipate that faculty-assessed internships, community-based projects, and senior projects would be the most useful in gauging graduates’ readiness for the workplace.

. . . When asked to advise colleges on how to develop their methods for assessing students’ learning, employers rank multiple-choice tests of students’ general content knowledge and institutional scores for colleges as conspicuously low priorities.

So, let me summarize (bias alert! bias alert!) via a single multiple choice question:

1. According to this report:

a. Grades are pretty much a non-factor in the hiring process.

b. Multiple choice tests are an unreliable predictor of success.

c. Employers are pretty much satisfied with the content knowledge of their employees and think assessments that cover content are relatively meaningless.

d. Employers want their employees to be more globally oriented, to take charge of their own job, and they must be able to communicate effectively through writing.

e. Employers prefer meaningful, relevant, experiential learning over an isolated, content-focused-only approach.

f. All of the above.

So, keeping those caveats I started with in mind, what does this mean for what we’re doing here at Arapahoe? Regular readers of this blog can probably predict what I would say about the ramifications of this in terms of the way we grade, what and how we teach, coverage of the curriculum, and how much of the responsibility for learning we put in the hands of students. But, just in case there was any doubt (and that the above wasn’t biased enough), let me add on to the choices above.

a. Since the primary way that many folks use grades appears to be superfluous, perhaps we need to take a hard look at the efficacy of grading in the first place, and perhaps switch to a focus on formative assessment versus “grades.”

b. Other than standardized test companies, politicians, and teachers who have a tight deadline to turn in final grades (see previous item), who’s really in favor of multiple choice tests? (Yes, I’ve ranted about this before somewhere, but I can’t find it at the moment.)

c. Content is necessary, but not sufficient, to be successful in the 21st century. So perhaps we should stop trying to “cover” the content, and instead focus on understanding the essential concepts and applying them in real world settings.

d. Constructivism. Blogs. RSS. Read/Write Web. Personal Learning Networks.

e. We need to take a hard look at our current system.

I’d love to hear your thoughts (especially AHS folks, but of course everyone is welcome to chime in) on what specific changes we might think about making. If you do comment, please try to not focus on the rant portion of the above, but on the results of the survey and on meaningful changes that will benefit our students.

9 comments:

  1. Hello Karl - I found your blog not 2 weeks ago via "Shift Happens"...I love, it, keep posting!

    I graduated from one of the best universities in the world (as rated by the Shanghai Jiao Tong University), and bearing Sir Ken Robinson in mind, I'd like to make to following comment:

    The academic standards at my university were rigorous - it was being trained to be an academic. So when I left, I was imbued with what could only be described as a warped, heavily socialist view of the world - government should take over everything, science is paramount to civilization, business is bad, and so on. Of course, living in the real world was something of a shock.

    We had no "exam" or "coursework" for creativity, critical thinking or applied problem solving skills: life was one big push to "publish or perish".

    I agree with a more formative approach vs grading...but I can't help but think of an oversimplified solution - many of these problems would vanish if we had a much higher student:staff ratio - staff who weren't busy thinking about their next grant proposal, but staff who cared about the growth and development of their students...

    ReplyDelete
  2. @stanley - Assuming you meant a much lower student:staff ratio, I'm with you. But I was thinking more along the lines of things we could change in our approaches to teaching and learning, as opposed to changes that our budget probably can't support at the moment (although I agree we should talk about that as well).

    ReplyDelete
  3. its very interesting that most employers find that multiple choice tests don't help much in discovering whether a student is qualified for a job. In my experience in college all of my lecture hall courses have only had multiple choice tests for ways of assessment. This makes me wonder why it is that way? Is it because that is the best way to assess us or the most convenient for the professor? I wonder if they had more people helping them grade papers if the professors would be more likely to have open-ended assessments, or if multiple choice tests are truly what they feel is the best way to assess our knowledge.
    Dietrich May

    ReplyDelete
  4. Karl et al -

    I've been visiting schools looking for interesting models, and had the pleasure of touring a New Technology school in Sacramento recently. Then I went to Science Leadership Academy for Educon. Both of these schools are project-based and New Tech does a really good job with formative assessment and portfolios.

    New Tech post here

    Besides doing a better job of preparing these kids for the world (and I share your concern that preparing kids for work isn't the only priority) I liked these schools because the kids are all really excited about learning, and they can identify learning and talk about it.

    ReplyDelete
  5. Maybe universities will finally get the message that this type of assessment is the more effective way to for students do demonstrate their learning. As an elementary school teacher, this is a process of assessment that we have been doing for years. To me it has seems that universities are the key for an educational shift to using more authentic assessment procedures. It would be nice to see a trickle down effect reach more high schools, junior high schools, and even some elementary schools.

    ReplyDelete
  6. Good article. Great discussion. I think that we need to quit trying to cover content and start covering the process of studying and learning. I did a backflip with my curriculum and delayed content coverage to teach basic study skills, and the classes then caught and overtook the pace of the rest of the classes. I'm one of the few history teachers that made it to the end of the book and beyond.... Teach process, not content.

    ReplyDelete
  7. Wonderful post and great comments. I am a science teacher in a PA high school struggling to do authentic activities in my 3 different classes. I have great ideas but the struggle is with the assessment.

    Since science is thought of as high brow and content heavy, authentic learning is met with skepticism but it is the only way to truly learn and apply. I wrote about the current problem with science education at Is Science, Math, and Technology truly for All?

    It is a call for changing content heavy education to one of what you described. This was also a major focus and agreement at Educon this past weekend. More conversations like this need to happen to change what has always been to what it really should be. Well said and adapted for others to hear.

    ReplyDelete
  8. I totally agree with your logic towards grades. I feel that multiple choice tests do not do an efficient job in testing student skills. Students can very easily memorize information for a multiple choice test and receive a passing grade, but may never really comprehend the material. I think that teachers are forced to cover a lot of material in a small amount of time. This leads to content being briefly covered but never really understood. I think teachers should be able to apply important concepts to real world setting instead of getting students ready to memorize facts for a meaningless multiple choice test and grade.

    ReplyDelete
  9. I totally agree with your logic towards grades. I feel that multiple choice tests do not do an efficient job in testing student skills. Students can very easily memorize information for a multiple choice test and receive a passing grade, but may never really comprehend the material. I think that teachers are forced to cover a lot of material in a small amount of time. This leads to content being briefly covered but never really understood. I think teachers should be able to apply important concepts to real world setting instead of getting students ready to memorize facts for a meaningless multiple choice test and grade.

    ReplyDelete