Posts Tagged ‘exam purposes’

Students are Gaming Your System

February 18, 2013

There was an amusing story recently about some clever students who found a way to get an A on the final without doing any work.

The professor’s policy was that all exam scores are re-normed so that the highest score on an exam becomes the new “100%” (I have a serious issue with his statement that this system is the “most predictable and consistent way” of comparing students’ work to their peers, since I think that students should judged on the basis of their knowledge of the material and NOT by comparing them to their peers. But that is a topic for a different post). His students recently got together and decided that none of them would take the final. Thus, the highest grade was 0%, and everyone got an A.

The professor did give everyone an A on the final exam, but later said: “I have changed my grading scheme to include ‘everybody has 0 points means that everybody gets 0 percent, and I also added a clause stating that I reserve the right to give everybody 0 percent if I get the impression that the students are trying to ‘game’ the system again.”

Here is the thing: students are always trying to game the system; this is because they are largely rational people. Moreover, professors mostly want students to game the system.

For example: I know of many professors who count homework as maybe 5-10% of a student’s final grade. The most common reason I hear from giving a low, but non-zero, weight to homework is “I want to make the students do the homework.” Translated: “I want my students to ‘game’ the system by doing the homework, whether they learn from it or not.” This is also true of attendance policies, participation policies, or really anything that has points attached to it.

What we (thoughtful people, at least) are really interested in is “student learning.” This is difficult to measure, so we use a proxy—”points”—to measure it. But then we fall subject to Campbell’s Law, where we confuse the proxy with the real thing.

Thus, the phrase “I reserve the right to give everybody 0 percent if I get the impression that the students are trying to ‘game’ the system again” really means
“I reserve the right to give everybody 0 percent if I get the impression that the students are trying to ‘game’ the system again in a way that I do not approve of.”

Moreover, the professor is not being clear about what are allowable ways of gaming and what aren’t. This conjures the memory of the famous case of cheating in Central Florida. The students found a test bank available online and studied from it. To me, this sounds like a completely reasonable way to study, and—to the best of my knowledge—this was not explicitly prohibited by the instructor.

The Central Florida was asking the students to ‘game’ his system by performing well on the exam. He was not clear about the allowable ways to ‘game’ the system, but he expected them to know what was allowable and what was not. This seems very unreasonable to me (there are some things that I think that we mostly agree on. It is not allowable to ‘game’ the system by writing down exactly what your neighbor wrote down, for instance. But I don’t think the Central Florida example is such a culturally agreed upon situation).

This is one of the many reasons why I switched to Standards-Based Grading: the proxies are at the very least less familiar, and most likely better associated with our ultimate goal of “student learning.” My proxies are not points, but rather “the number of times you demonstrated that you can do a particular type of problem to me.” It is tough to ‘game’ this system in a way that I am not in favor of, since most of the ‘gaming’ involves learning something well enough to convince me that you understand it.

It is still possible to game the system, though. For instance, students can demonstrate understanding through quizzes, and they can game the system by copying down their neighbors’ answers. But most of the examples of gaming SBG that I can think of falls into the “everyone knows that you are not supposed to do that type of gaming” category.

But the main point is: let’s not pretend that we don’t want students to game the system in certain ways. Let’s remember that the system is not what is important, and we must not lose sight of the reason why there is a system in the first place.


What oral exams taught me

June 8, 2012

In my course for elementary education students, I once again gave oral exams—this time for the final exam. Here are two take-aways from the oral exams.

First, I need to do some peer instruction next time. In particular, students had a difficult time understanding the difference between the “whole” of a fraction and the “denominator” of a fraction (Consider “\frac{1}{2} of a mouse” and “\frac{1}{2} of an elephant.” Both have a denominator of “2,” but the whole of the first is “mouse” and the whole of the second is “elephant.” This leads to different meanings. I think that three clicker questions would eliminate this.

Second, I was shocked at how ineffective my lectures were. The oral exam questions (which they also had to create screencasts for) were ones that were previous done in class (for example: why does inverting and multiplying give the correct answer to a division problem?). The process was this: students would figure out why the algorithm works, and then present at the end of a class period. I begin the next class period by giving the same argument. Other class periods begin with students presenting on similar questions, the class evaluating the presentations, and—if needed—me presenting the correct explanation.

Furthermore, I gave the answers to each of the oral exam questions on the last day of class. Test test So students saw the answer to each oral exam question at least three times, and probably more (especially since I had students view other students’ video solutions).

I was concerned that students would simply memorize these explanations. This simply did not happen. Either students understood the algorithm (I can tell from the oral exams—these students could answer any question that I had on the algorithm) or students did not understand any portion of the algorithm.

Most puzzling is that, in my student evaluations, some of my students complained that they were never shown how to do the algorithms correctly. This is in spite of seeing a completely correct solution to every problem between 3 and 10 times. I can only explain this in two ways:

  1. Somehow students did not understand that the solutions they saw were solutions to the problems from the oral exams and screencasts. This would mean that I did not clearly communicate the intent of presenting the solutions.
  2. Lecture was monumentally ineffective in helping them learn—so much so that students did not even remember that they occurred.

Do you have any other ideas?

On Midterms

October 22, 2009

I am in the middle of midterms. I tend to write three different types of exams: two types of in-class exams, and one time of take-home exam. I will mix the take-home with either type of in-class midterm.

The first type of in-class midterm is a check that students are able to do the basic things from the course. This includes recalling definitions and answering straightforward questions. In a calculus class, I might include a question like “What is the derivative of f(x)=x^2?” The purpose of this in-class exam is to act as an incentive for the students to take time to learn the course material.

The take-home exam has a different purpose. Here, I’ll ask questions that require students to think about concepts in novel ways. I often make these open book, open notes, group exams. In a calculus class, I might include a problem like: “Find the equation of a tangent line to f(x)=x^2+1 that goes through the point (4,8).” The purpose of this type of midterm is less to assess the student’s knowledge than to help her acquire more. I hope that thinking about these questions leads to a greater understanding of the material.

The second type of in-class midterm is like the take-home, only it is in-class and not a group test (with a couple of exceptions). The main lesson I have learned here is to only give a small number of questions, since each of the questions is fairly involved.

I have given all three types of midterms so far this semester. I tend to always include a component of “learning exam” (rather than “accessing exam”), as my main goal is to help students learn. However, I also need to assign grades, and this is the reason for the pure assessing exams.

I don’t feel great about giving the assessing exams. I do not like the idea of making the students demonstrate that they learned the material, largely because I have read psychology results that say this type of “incentive” (a bad grade is a “stick,” or a good grade is a “carrot”) decreases student learning. I would love to hear of creative ways of having students learn mathematics, assessing what they know, and having the two complement—rather than work against—each other.

Please leave comments, although please offer evidence if you say “students would never learn if I don’t give them exams/homework/etc.”

Educational Goals

September 25, 2009

My goal for today is to discuss large-scale educational goals. I expect this to be a running theme in this weblog, since it is an essential, yet under-recognized, part of education.

This theme will start with one example: standardized testing. This is a polarizing issue in education. One side, which is currently “winning,” claims that standardized testing is essential. We cannot know if students learned what they should unless students are given an unbiased exam. Moreover, standardized exams give us information about the teachers and schools; if too many students fail a standardized exam, it is evidence that a teacher and/or school is failing. Largely, standardized tests are the only true way to establish accountability.

The other side claims that standardized testing hurts education. Among other reasons, it is easiest to write a standardized exam about memorized facts; testing higher learning skills is considerably more difficult–and therefore much rarer. This gives us a skewed view of how the students are doing; “no information” would be better than “wrong information.” This problem compounds itself when teachers “teach to the test,” favoring bite-sized facts to complex problem solving. Furthermore, some standardized exams predict family income better than future grades. This could lead to promising students from poorer backgrounds to be denied access to education. Finally, standardized tests are expensive, and school districts could spend the money better elsewhere.

I did my best to be fair to both sides (I have my own opinion), although my arguments for each is by no means exhaustive.

There is debate about standardized testing in some circles, and arguments like these are thrown back and forth at each other. However, I think a more constructive step would be to delve deeper to determine the education goals and attitudes of both sides. What follows is my attempt to determine what kind of attitudes both sides might have about education.

Pro-standardized testing attitude: Students need to learn what we teach them, and we teach them things that are easy to measure–either a student knows how to add, or she doesn’t. Because of this, we need to provide incentive to the students to put in the work to learn. One way of doing this is testing–the student will learn what we teach in order to do well on the exam.

Anti-standardized testing: While facts are important, it is more important for students to develop habits and thought patterns that will make them a successful citizen. Knowing the fifty states is nice, but it is more important that students develop a habit of providing evidence when making assertions (and requiring evidence when hearing assertions).

If I were to have the pro-standardized testing attitude, it would be obvious to me that standardized testing is essential. With the other attitude, it would be clear that standardized testing would be difficult to administer, at best. Because of this, I believe it would be better for the sides to attempt to reach agreement on the educational goals, rather than standardized testing. Even if both sides were to agree on standardized testing, we would have only solved one symptom; the underlying cause of the dispute–different attitudes toward education–would linger and create new disagreements.

I propose that we all identify our educational goals and attitudes before we decide what tools (such as standardized testing) would best meet these goals.

Time pressure and exams

September 17, 2009

My wife teaches math at the public university in the area. I highly recommend marrying an academic, as dinner-time conversations can quickly turn into professional development opportunities.

Our dinner conversation on Tuesday centered around timed exams; that is, exams where you have to do many questions in a relatively short period of time. We debated their merit, and we came up with the following:

  1. Timed exams should only be used if they fit your goals and values. A discussion on goals and values will be the topic of an upcoming post; for now, I will just say that they are woefully neglected in education.
  2. Timed exams really only work if the students are only expected to either recall, or to do a very basic computation repeatedly.
  3. Timed exams are not appropriate if students are engaged in complex problem solving.

We reached these conclusions mainly by acknowledging that brilliant people can sometimes take a long time to figure things out – professors are never expected to start and finish a paper within a week. Deep thinking takes time. Therefore, adding time pressure to an exam can give a faulty assessment of one’s understanding (assuming this is the reason why the exam is being given).

On the other hand, there are other times when we do not want our students to think much, and here timed exams could give useful information. Examples of such topics include an elementary student demonstrating that they know their multiplication tables, or a calculus student demonstrating that they know how to quickly compute easy derivatives. In both cases, we want students not to have to think deeply about these questions (it is really difficult to get common denominators when both the concept of adding fractions AND multiplying integers require concentration; the cognitive load is just too much).

As I posted (years) before, I have started to give short midterms. This is because I mainly want to test problem solving and conceptual understanding, and I don’t include as much recall and computation (although they have to do computation in order to do other problems). I tend to test recall and computation in different parts of the course.