## Assessment Idea for Calculus I: Feedback desperately wanted!

I am planning an overhaul of Calculus I for the fall. I used a combination of Peer Instruction and student presentations in Fall 2012, and I was not completely happy with it.

So I am starting from scratch. I am following the backwards design approach, and I feel like I am close to being done with my list of goals for the students. Here is my draft of learning goals, sorted by the letter grades they are associated with:

View this document on Scribd

I previously had lists of “topics” (essentially “Problem Types”). These lists had 10–20 items, and tended to be broad (e.g. “Limits,” “Symbolic derivatives,” “Finding and classifying extrema”). This list will give me (and, I hope, the students) more detailed feedback on what they know.

This differs from how I did things in the past, in that I used to list “learning goals” as very broad topics (so they weren’t learning goals at all, but rather “topics” or “types of problem”). Students would then need to demonstrate their ability to do these goals on label-less quizzes.

The process would be this:

1. A student does a homework problem or quiz problem.
2. The student then “tags” every instance of where she provided evidence of a learning goal.
3. The student hands in the problem.
4. The grader grades it in the following way: the grader scans for the tags. If the tags correspond to correct, relevant work AND if the tag points to the specific relevant part of the solution, the students gets credit for demonstrating that she understands that learning goal. Otherwise, no.
5. Repeat for each tag.
6. Students need to demonstrate understanding/mastery/whatever for every learning goal $n$ times throughout the semester.

Below are three examples of how this might be done on a quiz. The first example is work by an exemplary student: the student would get credit for every tag here (In all three of the examples, the blue ink represents the student work and the red ink indicates the tag).

View this document on Scribd

The second example has the same work and the same tags, but the student would not get credit due to lack of specificity; the student should have pointed out exactly where each learning goal was demonstrated.

View this document on Scribd

The third example (like the first) was tagged correctly. However, there are mistakes and omissions. In the third example, the student failed to claim credit for the “FToCI” and the “Sum/Difference Rule for Integrals.” Because of this, the student would not get credit for these two goals (even though the student did them; the point is to get students reflecting on what they did).

Additionally, the student incorrectly took the “antiderivative of the polynomial,” which caused the entire solution to the “problem of motion” to be wrong. Again, the student would not get credit for these two goals.

However, the student does correctly indicate that they know “when to use an integral,” could apply the “Constant Multiple Rule for integrals,” and “wrote in complete sentences.” The student would get credit for these three.

View this document on Scribd

I like this method over my previous method because (1) I can have finer grained standards and (2) students will not only “do,” but also reflect on what they did. I do not like this method because it is more cumbersome than other grading schemes.

My current idea (after talking a lot to my wife and Robert Campbell, and then stealing an idea from David Clark) is to require that each student show that he/she can do each learning goal six times, but up to three of them can be done on homework (so at least three have to be done on quizzes). I usually have not assigned any homework, save for the practice that students need to do to do well on the quizzes. This is a change in policy that (1) frees up some class time, (2) helps train the students on how to think about what the learning goals mean, (3) force some extra review of the material, (4) provide an additional opportunity to collaborate with other students, and (5) provide an opportunity for students to practice quiz-type problems.

My basic idea is that I will ask harder questions on the homework, but grade it more leniently (which implies that I will ask easier questions on the quizzes, but grade it more strictly).

I have been relying solely on quizzes for the past several years, so grading homework will be something that I haven’t done for a while. I initially planned on only allowing quizzes for this system, too, but it seemed like things would be overwhelming for everyone: we would likely have daily quizzes (rather than maybe twice per week); I would likely not give class time to “tag” quizzes, so students would do this at home (creating a logical nightmare); I would probably have to spend a lot more time coaching students on how to tag (whereas they now get to practice it on the homework with other people).

Let’s end this post, Rundquist-style, with some starters for you.

1. This is an awesome idea because …
2. This is a terrible idea because …
3. This is a good idea, but not worth the effort because …
4. This is not workable as it is, but it would be if you changed …
5. Homework is a terrible idea because …
6. You are missing this learning goal …
7. My name is TJ, and you are missing this process goal …

### 29 Responses to “Assessment Idea for Calculus I: Feedback desperately wanted!”

1. Dana Ernst Says:

This is an awesome idea because…your heart is in the right place. I have no idea how this will work out, but I’m excited to read about how it is going.

• bretbenesh Says:

Thanks, Dana! I was pretty worried about previous iterations of this idea, but I think that I am at a point now where I will be willing to try it out. I think that it will have a reasonable high probability of not-being-a-disaster after I make more tweaks.

2. Moses Says:

I like this idea a lot in the generic sense. Would you allow students to choose which homework they do based on the learning goals they still need to fully accomplish?

Quizzes: I really like two-part quizzes: I give a quiz, grade separately from the quiz document so the students do not see the grading, return a %correct grade and the original quiz and give them a day or two to make corrections to the quiz (without knowing which problems were done incorrectly). I do allow students to work with others on the corrections so that the ones with stronger skills become tutors and the ones with weaker skills learn from a different teacher than myself. I then give partial credit back on the quizzes for the corrections and improved work. I think you could do this with the tagging as well (but you would have to grade pretty quickly).

Daily quizzes (how many days/week?) seems like a lot to me.

• bretbenesh Says:

Hello,

My idea is this: the homework is essentially optional, so students can do as many or as few problems as they like. I have been generally making all assignments optional, in that the worse any test/quiz/homework assignment can do is “fail to help your grade” (which is much better than “hurting your grade,” which is a distinct possibility under Traditional Grading).

That said, I am guessing that most students would opt to do the homework, since they can get help from friends/tutors/me/etc; it would be easier for each goal to get 3 problems with help and 3 problems on quizzes (without help) than 6 problems without any help on quizzes. But I leave that up to the student.

A colleague of mine did that two-pass system on an exam, and I was intrigued. I haven’t tried it yet because I cannot quite figure out for myself what the advantage is of withholding their feedback, although I think that there is an advantage. The only thing I can put into words is that it gives students a powerful incentive to (essentially) take the quiz twice. However, I suspect that there are other advantages that I cannot yet put into words; can you?

I am tentatively thinking about having two quizzes per week: one would be 55 minutes, would probably be about three problems, and would provide time to “tag” the solution. The second quiz would be something like 20 minutes to answer and tag one question. But I am planning on creating all of my assessments in the next month, and then I can see how realistic this is.

3. Whitney Says:

Hi Bret, I have been following several of your posts and taking in ideas for my “hopefully going to be a math teacher one day” idea bank. I rather like this idea, but in the context of CSB-SJU may I suggest being rather strict in the format you require homework to be submitted. Now, I am under the assumption that you will have a grader/TA for the class, which may be an incorrect one. However, as a recent TA, I would suggest that you create a format along the lines of tags must be in a separate color as in your examples above (or boxed in, something to make it stand out). This will allow for streamlined grading. Otherwise you will likely have people coming up to you (or your TA) every week complaining that you missed one of their tags. Then you must enforce the rule. Dr. Brown enforced the radical “one side of the paper” rule which greatly streamlined standard grading (although maybe a little wasteful), and points were taken off the homework set (after a few weeks into the semester) if the format was not followed. I think that if you do not have them differentiate the problem solution and the tags, it could get messy fast. Thanks for all of the wonderful ideas Bret!

• bretbenesh Says:

YES YES YES YES! Grading would be pretty awful otherwise. My plan is to provide students with felt-tip markers for the “tagging” (thanks to Dana Ernst, the first commenter, for encouraging me to buy a set). I plan on being strict on this—you only get credit if it is in felt-tip pen and boxed.

If you have any other thoughts about how difficult this would be as a TA, I would love to hear it. My sense is that it will be a little more difficult than a typical grading assignment, although not much more. I think that there will be less grading (maybe 5–10 problems per week, which seems to me to be low?), and the TA would only have to respond to the tags. Additionally, the grading is all-or-nothing for each tag—either the student completely meets the goal and gets credit, or the student does not get credit.

But I think that it will be harder in the following way: I think it will be better if I create codes for the students tags. So instead of writing “I can use the FToCI” in felt-tip pen, the student might write something like “B13.” This will cut down on work for the student, but the TA will have to be able to translate this to English in order to grade it. I think this might not be so bad if the TA grades the same problem for each students at the same time, but it might be annoying to have to constantly be looking up what the codes mean.

Thoughts?

It is always great to hear from you, Whitney!
Bret

4. Robert Talbert (@RobertTalbert) Says:

This is an awesome idea because it requires students not only to demonstrate content mastery but it also forces them into metacognition. Students not only have to master content, they also have to be able to know *when* they have mastered content, come to grips that “knowing calculus” involves both content and process, and articulate exactly what it is that they know or don’t know. And the “tagging” system seems relatively simple.

On the other hand this is a terrible idea (well, not really but this is the writing prompt I was given!) because I foresee endless nitpicking arguments with students over what exactly constitutes, say, “I can use linear approximation”. If a student does a half-assed job of taking a derivative and plugging in a number to answer a question, can they turn around and say that they “can use linear approximation”? Or is there something more sophisticated in mind?

Two more things:

+ Once you have this system the way you want it, can you post your syllabus etc.? I am desperately wanting to move towards a standards-based assessment structure in calculus but I just can’t find something I feel comfortable with, and what you’re talking about here is as close as I’ve seen.

+ David Clark is an awesome person and I’m super-glad that he’s joining us here at GVSU in the fall.

• bretbenesh Says:

I didn’t mean that you HAD to respond using one of my starters 🙂 And I will post my syllabus once I have it written. I am hoping to have it done by the end of July, but this is just an estimate.

Thanks for the feedback—I am pretty excited about this, too. I was reasonably happy with my old system (based on Problem Types) for my courses that are heavy on calculations, but I did not think that it was workable for, say, a proof-based course. I think that this idea has potential to work with any type of class with only minor modifications (this assumes that I can get it to work in calculus, first. I am aware that this whole thing could flop for a large variety of reasons).

Additionally, I could see using this in a Moore Method course, which I have been having a tough time assessing. If a student presents a solution to a problem, they should also be prepared to discuss how they “tagged” it. The class can then have a discussion about whether it is all legitimate.

I spent a decent amount of time with Campbell today trying to figure out whether this whole tagging idea was worth the effort. I can almost convince myself that it merely amounts to me forcing the students to just learn a tad more vocabulary (e.g. students would need to be able to know the difference between “exponential function” and “polynomial” in order to get credit for, say, taking the derivative of each). However, I was leaning toward this engaging the students in some serious metacognitive work. I am seriously pleased that you brought this up; I intentionally avoided using the phrase “metacognition” because I wanted to see if anyone else would think that it was relevant here. I feel more confident with this system because you also recognize that there could be such benefits.

Of course, I would love to hear from anyone else who disagrees, though.

On tagging: did you see in my response to Whitney’s comment that I am planning a coding system for the learning goals? This should serve the dual purpose of requiring less class time for tagging and less chaotic papers (but introduces potential grading annoyances, as I described to Whitney).

On nitpicking: I am not terribly worried about this, as I think that I have come up with a good way of coping with this from my previous grading system. The key is that not getting credit for a tag cannot hurt a student’s grade (it simply fails to help the student; a second key is marketing this fact to the class).

So if a student starts nitpicking, I simply say in a tone that is somewhere between sympathetic and matter-of-fact: “I can see why you thought that your tag should count. However, this is not what the learning goal means. Remember, part of the goal of these quizzes/homework/whatever is helping you understand what these goals mean. In fact, this is one of the main reasons why I make sure that your grade can never be hurt in this situation—I built that into the grading system so that we can have this conversation without you being penalized.. Now, let’s talk about what this goal really means and what you should do on this type of problem so you can be successful next time.”

This has worked well with my students, and I typically do not hear many complaints about this beyond mid-semester. That said, I now have 3–4 times as many goals for which I can have this conversation, so we will see.

Is this what you had in mind for the nitpicking issue?

GVSU really has a deep bench, and David only makes it deeper. He was a good get for you, although he is lucky to be at a place that will appreciate his talents, too. I am quite happy, though, that we were about to steal Sunil Chetty away from you guys a couple of years ago 🙂

• dcclark Says:

Following up on the “nitpicking” worry — when I used a standards-based system last, I used the phrase (and repeated it frequently): “Did you *demonstrate* proficiency with this idea *to me*?” It made it abundantly clear to students that having the idea in their head wasn’t enough, but rather that it had to be clearly demonstrated. Things were a little rough at the start, but once we all got on the same page, this worked out very well. (As a sidenote, the fact that some students did not get proficiencies when they thought they should have did not cause them any harm — they were able to demonstrate it and achieve the same level of mastery anyhow.)

• bretbenesh Says:

I seriously have a note to include “to me” in my syllabus; I completely lifted it from your syllabus, of course, David.

Robert: I am late on the promised syllabus—research got in the way. But it will be done next week (it HAS to be done next week).

5. Pinky Says:

This looks like a great summative assessment! Progress portfolios have a similar goal in mind, although they are two very different methods. This seems better for higher education.

• bretbenesh Says:

I love the idea of progress portfolios, but I have a couple of problems that prevent me from using them:

1. The one time I tried them in the past, the students reeeeeeeally procrastinated, to the point where they allegedly were skipping other math classes to get my portfolio done at the end of the semester. I think that I could help this out a little by having more milestone checks, but I am not certain.
2. My students are generally pretty awesome, but I have encountered enough cheating to make me wary of giving too much weight to assignments not done in class (hence the requirement that three of their successes must come from quizzes).
3. I am not certain that I have figured out how to give the right assignments for a portfolio yet. If I could do this well, it might help take care of my concern about cheating. But I don’t know how to do it right now.

What are your reasons for thinking that progress portfolios are better for K-12 than for higher ed?

• Pinky Says:

I think they work well for both, but this is more tailored to college age and ability than traditional progress portfolios. I am also biased, as I’m studying in a block system, and our progress spans only a couple weeks before we complete a class.

• bretbenesh Says:

Keep me posted if you figure out a great way to make them work. I hope to move to them in the future.

Hi, Bret:

Inspired by your posts in the past, I have been steadily decreasing the number of standards. For calculus, my feelings have evolved to the point where all of the “rules” go into one standard. This really helps the students to see the rules as being interrelated and worthy of being studied together. Similarly, abstract applications of derivatives to curve sketching get clumped.

For sure, I’ve lost some of the expressibility of the original list, but this is really balanced out by the ease in grading, and in student understanding of the system itself. My experience has been that if there are too many standards divided up in too many ways, then the system itself can become an impediment to student learning.

This coming semester, I am teaching business calculus and I’ve got the content standards list down from nearly 40 to a more manageable 18. Probably the next time I will cut it down further. It seems we are running the same race in opposite directions.

• bretbenesh Says:

This is both interesting and scary! I was feeling good about this plan until I read your comment. I have a tendency to make my grading system too complicated, so this is definitely a concern here.

If we both continue with our current plans, we should compare notes after the semester.

Bret

Oops. Didn’t mean to create any angst 🙂 I struggle all of the time because I see ways to make the system more “philosophically consistent”, but then I realize that I’ve just confused the whole class to the point that I might lose buy-in. Like a good board game, I am striving for that elusive combination of depth and simplicity. So far, I am more Monopoly than Go.

We should definitely compare notes after the semester. I usually follow your blog, but I haven’t had much time to post to mine anymore.

Here is a link to the current standard list. The semester hasn’t started yet, so please give me feedback on it!

https://www.dropbox.com/s/vdw2emwtvwlqxt1/f14-math-135-standards-list.pdf

The course is a probably a bit different from yours in that we cover some probability distributions and economic applications. There are also several standards related to a project, but their impact on the grade is proportionally much smaller and I’m not counting those as part of my 18 “content standards”.

• Joss Ives Says:

Hi Bret. I really like the spirit of your plan, but it certainly seems like you are creating a lot of work for yourself. Of course, you seem to be much better at maintaining a good work-life balance that I do so I may just have to trust that it will work out.

• bretbenesh Says:

A little anxiety is good! It forces me to really think about this (I also like the Go/Monopoly analogy).

Thanks for the list. I taught a business calculus class my very first semester of using SBG-type grading in Fall 2010. I only had 11 standards then (in part because I didn’t realize that I was supposed to be doing multivariable calculus until I was a couple of weeks into the semester!).

The only feedback I have for you is to compare to my old stuff; I don’t have many judgments. You do A LOT more with applications than I did (actually, this is probably better than my old set of standards, given the purpose of the course). I had numerical and graphical derivative standards, as well as a linear approximation standard (maybe it is there and I didn’t see this).

It seems really reasonable to me, though.

Do you have a link to a post on how you assess your standards? I was able to find an old post that said you plan to test each standard twice, but I don’t know if that still holds.
Bret

• bretbenesh Says:

Hi Joss!

This summer will be quite a bit of work. The semester shouldn’t be too bad for me, though—I have a grader!
Bret

P.S. Kyle Anderson to the Spurs was just about perfect.

Hi, Bret:

I don’t have an updated post on how I assess. Basically, I try to assess all (except for the standards from the last week) of the standards at least twice as part of in-class written assessments. I also let the students reassess during my office hours up to one standard per day. I used to let them reassess more standards per day but found that they rarely would do well in that scenario, and that I was spending hours every day dealing with reassessments.

One thing that should cut down on the workload a bit for me is the creation of a small test bank. I started it in my abstract algebra class midway through last semester and was amazed at how much time it saved me. I started a LaTeX document that includes questions for each standard (which I add to whenever I spot a nice problem). If a group of students want to reassess, I uncomment the problems I want to give to them, and when the document compiles, it puts one problem on each page. If I want to give the same problem to multiple students, there is an option to print multiple copies of the same problem. If you are interested in the document, I’m happy to send it to you.

• bretbenesh Says:

The test bank is almost exactly what I do! In fact, I am about to start to overhaul my test bank for calculus tomorrow. I would love your file—my way is really low-tech. I have a bunch of questions below a quiz template. When I want to create a new quiz, I copy the master file to, say, Quiz6.tex, do a control-f find for the right question, put a mark indicating that I used the question on the quiz (e.g. “Q6” for the sixth quiz) so that I don’t re-use it, and then paste the question into the template. Then I compile.

I used to let my student reassess one standard per day, and I really like that idea. However, it got overwhelming, and I replaced it with “use a whole bunch of class time for assessment.” That was before I did the test bank idea, though. I should consider doing this again to re-claim some class time for instruction.
Bret

7. dcclark Says:

Holy wah — I was offline for July and part of June and just resurfaced. This sort of organization is exactly where I’d like to take my own standards-based system. Organizing by letter grade, as opposed to the “topic-standard” organization, should make it a lot clearer for students.

I forsee one weird problem which I had: For *certain* standards, you really want students to show you mastery constantly, or at least over a very long time period (in your setup, “synthesis” is probably one of these). There’s a certain weird incentive for students to stop caring once they’ve mastered that standard. My mental solution to this — totally untried — is to change the number of “checkmarks” required for those problems. That can get complex and confusing, however.

Alternately, you seem to have built constant mastery right into some of the standards (“correctly did ALL assigned proofs”) which I think would work for some, but runs the risk of making it impossible for students to achieve certain standards if they mess up early. What do you think about how that part will work?

And finally, y’all are too kind. Minnesota wore off on me, I can’t accept compliments! 😛

• bretbenesh Says:

Hi David,

“you really want students to show you mastery constantly”

My imperfect coping mechanism for this it to be very stingy in giving them certain types of questions. I put at most one “type” of question on each quiz, and I try to put a lot of time between quizzes that meet certain standards. In addition to getting closer to forcing them to demonstrate it “constantly,” there is some science that supports this stinginess (at least, if you believe the authors of _Making It Stick_).

This system is imperfect, since I don’t do this perfectly well, and there are still students who might nail something in only 6 weeks. However, students who do that are typically the ones who could do it for longer than that, too, so there aren’t many students who “do it N times and then forget.”

Another problem with this is that quizzes are pretty short in the beginning of the semester. In calculus, it helps to include precalculus standards for this. So the beginning of the semester has mainly precalculus stuff on quizzes, and that is replaced once we get to the calculus stuff.

“Alternately, you seem to have built constant mastery right into some of the standards (‘correctly did ALL assigned proofs’) which I think would work for some, but runs the risk of making it impossible for students to achieve certain standards if they mess up early.”

I avoid the problem of students digging themselves into a whole by giving students as many attempts to re-write as they can (these are not done in a quiz setting). However, this means that I have not built constant mastery into them, since a student could do all of the proofs in one week. I can live with this, though, since these proofs will mostly be baby real analysis proofs, and my main goal is to communicate that math is about more than just computing stuff. If they do the proofs in one week, I think that they will still get that message.

Do you have any suggestions around any of these problems of constant mastery? Could you describe how your changing requires for number of successes works?
Bret

• dcclark Says:

Unfortunately, I don’t have a good answer — I set *everything* at 3 proficiencies, including some which really needed more (such as the “student success” proficiencies which represented attendance, interaction, etc.). That was so that my system didn’t get any more complicated than it already was. My best idea there was to say that certain proficiencies were evaluated for an entire week at a time, but that lead to some problems with motivation as well.

More attempts is probably a good approach, together with very careful planning. Initially, the reason I thought I should not increase the number of proficiencies was that certain topics were pretty small and would only be assessed, perhaps, 4 times. Turns out I was much more generous about giving attempts at proficiency, so it wasn’t a problem.

• bretbenesh Says:

“More attempts” is really the only tool I have. I am also in panic mode, as I am not nearly as prepared for the fall as I had hoped to be. This means that I am going to have to work on refining the solution to “constant mastery” some other semester.

8. Assessment Idea for Calculus I: Near Final Draft | Solvable by Radicals Says:

[…] « Assessment Idea for Calculus I: Feedback desperately wanted! […]

9. Update on Student-Claimed Learning Goals | Solvable by Radicals Says:

[…] am halfway through the semester where I am using a new grading scheme for Calculus I. Here is a rough summary of the […]

10. Three Benefits of “Accumulation Grading with Tagging” | Solvable by Radicals Says:

[…] I decided to give my grading system a name: Accumulation Grading (or Accumulation Grading with Tagging). I just sick of writing “this grading system” or “how I am […]