Assessing Assessments: How do we know we know what they know?

Assessment and defensible measures of assessment have been central to many discussions in higher education in the past several years. The recently published book Academically Adrift: Limited Learning on College Campuses (University of Chicago Press, 2011, by Richard Arum and Josipa Roksa) assesses higher education and finds it lacking, arguing that according to their assessments forty-five percent of undergraduates showed "no improvement in critical thinking, complex reasoning, and writing skills in the first two years of college, and 36 percent showed no progress in four years." Writing for the Chronicle of Higher Education, Thomas Benton calls this one arm of a “perfect storm” in undergraduate education, arguing that these findings along with the current political climate are functioning to further endanger the quality of education. At the core of Benton's argument is “Few people outside of higher education understand how little control professors actually have over what students can learn,” continuing to name some of those factors that have removed faculty from positions of power in shaping higher education. While most of this is beyond the daily workaday functions of the graduate student instructor or TA, it does impress upon us the necessity of ensuring that we can show that our teaching works, is meaningful, and we can prove it. Further, we all spend far more time on grading than we would like, so it would behoove us to ensure that the work we are doing grading is actually worth something.

So in this issue of TAP Talk, I'm going to give some pointers on designing assessments, identify some resources, and throw in a bit of job market advice along the way.

Some background on assessment:

  1. The big picture: You need to know what it is you're looking for, preferably before you teach it, otherwise your assessments can end up oddly divorced from your class. The goal is not to teach to the “test” (or whatever assignment you require), but rather to have the assessment be in line with the learning goals you have set for the course.

  2. What counts as assessment? Anything you do to find out what your students know/understand/can do counts as assessment, whether you formally assign a grade to it or not. Assessments are generally classified in various ways, one classification being “high-stakes,” such as exams or major paper assignments, and “low-stakes,” such as in-class writing assignments or a short quiz.

You can also add depth and weight to your teaching portfolio (which you will need for the job market if not before then), by adding examples of your assessments and how your students have handled those assessments. A pre-test and a post-test, or a paper draft with comments alongside a much improved final draft can be a great way to show that your teaching has really made a difference.

Exams: We've all taken them, but what makes a good one?

The big exam is a time-honored tradition across academic disciplines, often counting for up to 80% of the overall grades for a class, if not 100% (combined between midterms and finals), but it can be difficult to write a good one, and even more difficult to grade. Below I will discuss the different kinds of questions you can ask on in-class and take-home tests, each with their advantages and disadvantages. See the following table outlining advantages and disadvantages of different types of questions adapted from M.E. Piontek's (2008) Best Practices for Designing and Grading Exams.

Type of Item Advantages Disadvantages
True-False Many items can be administered in a relatively short time. Moderately easy to write; easily scored. Limited primarily to testing knowledge of information. Easy to guess correctly on many items, even if material has not been mastered.
Multiple-Choice Can be used to assess broad range of content in a brief period. Skillfully written items can measure higher order cognitive skills. Can be scored quickly. Difficult and time consuming to write good items. Possible to assess higher order cognitive skills, but most items assess only knowledge. Some correct answers can be guesses.
Matching Items can be written quickly. A broad range of content can be assessed. Scoring can be done efficiently. Higher order cognitive skills are difficult to assess.
Short Answer or Completion Many can be administered in a brief amount of time. Relatively efficient to score. Moderately easy to write. Difficult to identify defensible criteria for correct answers. Limited to questions that can be answered or completed in very few words.
Essay Can be used to measure higher order cognitive skills. Relatively easy to write questions. Difficult for respondent to get correct answer by guessing. Time consuming to administer and score. Difficult to identify reliable criteria for scoring. Only a limited range of content can be sampled during any one testing period.

Often the advantage of one type of question are counter to the advantages in another, and it seems that the disadvantages are sometimes counter to our goals. We all want to encourage our students to use higher order cognitive skills, but can we effectively administer and score as many essays as are necessary to effectively measure knowledge? Finding the right balance can be the difference in an exam that really pushes the course forward and one that just confuses and frustrates your students.

Grondlund and Lind (1990) advise that you make your exams “valid, reliable, and balanced.” Valid and reliable in this circumstance generally mean what they mean across scientific fields. Our exams are valid if they actually test what it is we think they're testing—in this case the student's knowledge/understanding/critical ability. Exams are reliable if they accurately evaluate the students' performance; we can best improve reliability by making sure questions are intelligible, directions clear, and expectations well-communicated. Exams are balanced when they test from the range of materials covered so that students are able to show their strengths, not just particular weaknesses.

Barbara Gross Davis, in Tools for Teaching (1993), offers a number of tips on designing exams. Here are just a few that can really make a big difference.

  1. Write questions/items throughout the semester instead of the week (night) before the exam. This will help keep your exams tied to the material you have taught, not just your memory of your lectures.

  2. Get students to submit questions for the exam. You can pick and choose which you end up using, but this will give you a sense of what the students are thinking about as well as allow your students to feel involved and effective.

  3. Prepare clear instructions. Many problems can be avoided by making sure your expectations for your students are clear. Have a colleague read your instructions before you administer the exam, they will be able to tell you if they are clear, or just clear to you.

  4. Make sure the timing works. An exam that is too long can leave your students more worried about finishing than about the content and can skew the whole thing. A true-false question might take only 30 seconds while a complex essay question might take 30 minutes.

Exams and grading will likely always be a part of our job, and as we seek to show that our teaching works and matters it is ever more important that we be able to articulate and defend these practices. Writing a strong exam will benefit you and your students.

Writing a Writing Assignment

The first key to writing a successful writing assignment is clarity and thoroughness—both in your thinking about the assignment and how you communicate that assignment to your students. You should clarify in your own mind what the best possible product might look like to you, and craft your assignment from there. If you merely tell your student to write a 10 page paper, you will likely get as many different approaches as students, and it will often end up being dumb luck whether students pick the right one.

The easy stuff: If you are particular about format, say so. Your students want to know what you want from them, so spell it out. If you want you assignments 10-pages double-spaced, one-inch margins, in 12-point Times New Roman, why wouldn't you say so explicitly? This also goes for citation format. If you want it in MLA or APA, say so—by being clear you are putting the burden on the student to act responsibly, instead of marking off points later on or feeling like you can't take points off because you didn't notify them. This kind of specificity will never harm you.

Define your criteria: A grading rubric is a great way to be consistent when you are doing the grading, so supply your students with a frame for assessing their own work. Use this rubric, adapted from http://www.allstudentscanlearn.org, or create your own.

A B C D
Clearly & effectively responds to assignment. Response to assignment generally adequate & thorough. Minimally responds to the assignment. Does not respond well to assignment.
Demonstrates specific attention to relationship between audience & purpose. Demonstrates understanding of audience & purpose. Demonstrates some understanding of audience & purpose. Demonstrates poor understanding of audience & purpose.
Main idea (thesis) very clearly stated & topic is effectively limited. Main idea clear & topic is limited. Main idea clear or implicit & topic is partially limited. Main idea unclear & topic only partially limited.
Thesis supported in body of paper by a variety of relevant facts, examples, & illustrations from experience, references to related readings, etc. Thesis well-supported in body of paper by facts, examples, illustrations though support may not be as vivid as the “A” essay. Thesis generally supported in body of paper by facts, examples, details. No more than one paragraph with inadequate support. Thesis supported in body of paper by few facts, examples, details. More than one paragraph with inadequate support
Organization & structure very evident: major points divided into paragraphs and signaled by use of transitions. Each paragraph has a topic sentence; sentences within each paragraph relate to each other & are subordinate to the topic. Introduction & conclusion effectively related to the whole. Organization & structure clear. Most major points are separated into paragraphs and signaled by transitions. Paragraphs are built on related sentences that logically develop the main points. No major digressions. Introduction & conclusion effectively related to the whole. Organization & structure mostly clear. Many major points are separated into paragraphs and signaled by transitions. Most points are logically developed. There may be a few minor digressions but no major ones. Introduction & conclusion are somewhat effective. The organization & structure must be inferred by the reader. Only some major points are set off by paragraphs and are signaled by transitions. There are some logically connected points. There may be some major digressions. Introduction and conclusion may be lacking or ineffective.
Voice & tone are consistent & appropriate to the audience/purpose. Voice & tone consistent & appropriate although somewhat generic or predictable in places. Voice & tone adequate to audience/purpose although often generic or predictable. Voice noticeably generic or inappropriate (e.g., first person narrative may predominate in an analysis assignment). Tone is often inappropriate.
Full variety of sentence structures used correctly. Word choice interesting, accurate and contributes to the writer’s ability to communicate the purpose. Variety of sentence structures used correctly despite an occasional flaw. Accurate & varied word choice. Sentences & word choice predictable. Occasional errors in sentence structure, usage & mechanics do not interfere with writer’s ability to communicate the purpose. Little sentence structure variety; wording predictable; few synonym alternatives used. Errors in sentence structure, usage & mechanics sometimes interfere with the writer’s ability to communicate the purpose.
Few, if any, minor errors in sentence construction, usage, grammar, or mechanics. There may be a few minor or major errors in sentence construction, usage, grammar, or mechanics. There are some common errors (major and minor) in sentence construction and mechanics but the writer generally demonstrates a correct sense of syntax. There are numerous minor errors and some major errors. Sentence construction is below mastery and may display a pattern of errors in usage and mechanics.
Source material is incorporated logically & insightfully. Sources are documented accurately. Source material incorporated logically. Sources documented accurately. Source material incorporated adequately & usually documented accurately. Source material incorporated but sometimes inappropriately or unclearly. Documentation is accurate only occasionally.

You can adjust any rubric to fit your needs, and by having the criteria clearly stated, your students will be able to improve their work. When you discuss a rubric with your class, you should articulate the relative weight of the different measures. Depending on your field, you might value creativity, structure, or use of sources differently—let your students know this and be sure of it yourself.

Good practices for all disciplines include giving your students an opportunity to progress in their writing. This might be requiring that drafts be handed in and assessed either by you or by peers prior to the final assignment being due. It might be turning a less formal piece of writing (a response to an essay question, a journal entry, or an in-class piece of writing) into a longer, more formalized piece. When you guide your students through these assignments you may not be teaching them to write, but you are requiring that they think critically about their own work, thus giving them the opportunity to take the initiative to improve.

Think critically about what it is you hope to achieve from this assignment and design accordingly. If you want them to work through an argument, you might want to require an assignment that requires them to present all sides of an issue. If you want them to become an expert in a topic, you might require an annotated bibliography. If you want them to reflect upon their work, ask them to do so and have them write you a letter stating what grade they deserve and why—using evidence from their own papers.

Writing strong, clear, effective assignments can be difficult, but the more you put in, the more you will almost always get out.

References:

Arum, Richard and Josipa Roksa. 2011. Academically Adrift: Limited Learning on College Campuses. University of Chicago Press.

Benton, Thomas. 2011. "A Perfect Storm in Undergraduate Education, Part 1." The Chronicle of Higher Education. http://chronicle.com/article/A-Perfect-Storm-in/126451/ Accessed 2/28/2011

Gronlund, N. E., and Linn, R. 1990. Measurement and Evaluation in Teaching. (6th ed.) New York: Macmillan, 1990.

Piontek, M.E. 2008. Best Practices for Designing and Grading Exams. Center for Research on Teaching and Learning, University of Michigan.