Interesting Statistics Observation

Background: My son plays soccer, and I want him to avoid concussions if possible. Virginia Tech has done a lot of research into how effective various headgear are at preventing concussions. My son has been wearing a headband to reduce the risk of concussions for the last couple of years.

He recently started complaining about the headband, saying that it seems like it might be doing some damage—he gets an unpleasant vibrating sensation when he heads the ball. I decided to look into the data a little more to figure out if I could feel good about him not wearing a headband. Here is what I found:

• The Virginia Tech studies are done with mannequins in a lab. This is a good start, but I wanted better evidence that it would work on humans.
• Delaney et al have a 2007 paper that says the relative risk concussions for wearing headgear is about 0.4 with a ridiculously low p-value. They showed that 14 of 52 players who wore headgear got a concussion during the season (about 27%), and 114 of 216 (52%) who didn’t wear headgear got a concussion. These rates are unusually high (my teams have had about 1 concussion for every 15 players over the last several years), and I think that they are a result of this being a retrospective survey. This isn’t convincing to me.
• McGuine et all did a prospective survey of about 3000 players (about half of which wore headgear—1505 players, versus 1545 who did not wear headgear), which is mostly convincing. They found a risk ratio of 0.98 (p=0.946), which indicates that headgear likely does not help.

Moreover, the risk ratio for males (474 with headgear, 546 without) was 1.83 (p=0.286), which suggests to me that the headgear might make things worse (the risk ratio was 0.9 for females). I concluded that my son doesn’t need to wear a headband.

I regularly teach introductory statistics, and I am pleased that I will be able to tell my students that I used my knowledge of statistics in a way to help me in my non-work life. This is something that they will be able to do in the future (I had to Google “risk ratio,” so it will be okay when they do that, too).

There were some quirks about that paper that made me pause, though. First, the authors state that only 13 players did not comply with the study: 6 of 1545 players (0.39%) used headgear when they weren’t supposed to, and 7 of 1505 (0.47%) didn’t use headgear when they were supposed to. They were working with teenagers, and this seems like a remarkable rate of compliance (my son forgets to wear his headband more often than 0.5% of the time—once every 200 games/practices—and he is reasonable and committed to wearing it).

Stranger yet: of the 7 who failed to use headgear but didn’t, all 7 experience concussions. The overall concussion rate was 4.3% (which matches with my experience as a coach, and which is why I do not find Delaney’s paper convincing), so having a 100% concussion rate for these 7 seems strange.

If you run the statistics on what the players actually did rather than what they were supposed to do, the results change quite a bit: the overall risk ratio changes from 0.98 (as mentioned above) to 0.63 (p=0.091). The authors of the paper were kind enough to include this, although they failed to mention that all 7 of the students got concussions—I had to sift through the numbers to figure this out on my own.

As a father, I would be very interest in a risk ratio of 0.63, even if p-value is north of 5% (9.1% is still fairly small, so there is some evidence that we should reject the null hypothesis that the headgear doesn’t help). This basically means that if, say, 5 of every 100 non-headgear wearing players got a concussion during the season, only 3.15 of every headgear wearing players would get a concussion. It doesn’t quite cut the risk in half, but it is close.

The drawback is that this changes the validity of the study, since the players were not randomized (although it was only 13 out of 3050 who “defected”).

Ultimately, I still will let my son stop wearing the headband. This is mainly because, while I like the effect size of 0.63 at a p-value of 9.1%, almost all of the benefit goes to the females: disaggregated by gender, we see a new risk ratio of 0.93 (p=0.909) for boys and 0.64 (p=0.094) for girls. Since I have a son and all of the benefit goes to girls in all cases, it seems like he shouldn’t have to wear it (although I might make my daughter wear one if she played soccer).

I emailed the corresponding author of the paper to find out if I was mistaken about the 7 players, but he has not yet responded back to me. I would appreciate if any statistics-savvy people out there can let me know if I am thinking about any of this incorrectly.

On “The Ungrading Learning Theory We Have Is Not the Ungrading Learning Theory We Need”

I somehow came across the paper “The Learning Theory We Have Is Not the Ungrading Learning Theory We Need” by Clarissa Sorensen-Unruh. I love the title, and I read the paper. Here are the bullet points.

The main point of the paper is that the learning theory undergirding Ungrading is Self-Regulated Learning (SRL), which is deficit-focused (i.e. We should try to fix the students’ gaps in knowledge). Instead, the paper argues that Ungrading is actually asset-focused (i.e. We should figure out what the students are good at, which may differ from what the instructor expects them to be good at, and leverage those assets). The paper argues that Ungrading should be view as an emancipatory pedagogy (a la Paulo Freire).

I am still digesting this difference, as I don’t know enough about the learning theories to have anything intelligent to say (although I hope that some commenters will have something intelligent to say). In fact, here are ways that I want to expose my ignorance:

• I don’t fully understand how SRL is a theory of learning. Perhaps I am reading too much into the title, or perhaps I am conflating Linda Nilson’s book with a separate theory of learning. Either is possible.
• The paper cites papers by Alfie Kohn and Jesse Stommel as evidence that Ungrading uses SRL as a learning theory. I am very familiar with these papers, yet I can’t understand why the papers point to SRL. This could be because I am immersed in SRL and don’t know it, though. (The paper also cites a paper by Gruberman, which I am unfamiliar with, and Ungrading by Susan Blum, which seems to be just a second citation for Stommel’s article).
• It is unclear to me how SRL is inconsistent with emancipatory pedagogy. The latter is about changing power dynamics so that oppressed groups can be liberated, and the former is about getting students to be involved in their learning processes (I am paraphrasing the paper for both of these). These seem like they are orthogonal to each other. Again, I am admitting to ignorance.

The idea that Ungrading should be thought of as emancipatory is intriguing, and I hope to understand the these ideas more soon.

One issue that I have throughout the paper is the definitions of the types of grading. I appreciate that the paper defines Ungrading, but I don’t like the definition.

“While the LGBQTIA+ acronym includes almost every group that is not cis-heteronormative, ungrading includes almost every classroom evaluative assessment practice that does not fall under traditional grading or normative grading (grading on a curve).”

This does not match my sense of Ungrading, nor does it match my interpretation of what was meant in Blum’s Ungrading (for every chapter except one, which I will explain below). My old SBG courses have a very different feel than when I Ungrade now, and I have a tough time thinking about how the old SBG classes have anything to do with emancipatory pedagogy (but I can see how my Ungraded courses might change the power dynamic).

For instance, the paper states that competency-based grading is Ungrading, and the paper defines this practice as follows:

“Competency is based on a level set before grading begins. Most of the time, the competency is set between 80% to 90%, with 95% being an unreasonable maximum for most contexts.”

This seems like it would be difficult to support emancipatory pedagogy via competency-based grading, although it is included as Ungrading.

The one chapter from Ungrading that did not match my notion of Ungrading comes from Sorensen-Unruh. She offers on her blog a syllabus that describes her version of Ungrading. To summarize, Sorensen-Unruh grades the exam, the student grades the exam, and the official grade is the average the two results with a bonus (or penalty) if the student’s score is close to (far from) the instructor’s score, as measured by standard deviations (which I don’t know how this standard deviation would be defined).

Summary: I think that this paper has some cool ideas, but I need to read more about the theory to fully digest it.

Time Management and Coaching

Welcome back!

I accepted an offer to be an assistant coach for a high school soccer team, which means that this month is really busy. I had to make a plan to manage my time. Here are the big ideas.

• Timeblocking: This is not new, but I am recommitting myself to time-blocking after a summer where I didn’t really use it. Cal Newport estimates that it doubles your productivity, and that seems about right to me.
• I am continuing to do weekly planning each Friday. I figure out everything that I need to do that week, and then I figure out when I am going to do.
• I am starting to be much more explicit about my goals for the week. They are now at the top of my timeblocking document.
• I then determine which of my weekly goals I will work on each day—I set daily goals.
• Once I am done with my daily goals, I can be done for the day; once I am done with my weekly goals, I am done for the week.
• I am cutting back on activities that are less essential during soccer season. For instance, I cut my research time down to one-third of what it once was. This will return to normal after the season is done.

I haven’t left early either of the two school days we have had, so it isn’t like I am ghosting my school. However, I have gotten all of my daily goals done before soccer starts each day, so this seems to be working reasonably well.

Ungrading: Early Final Grades

I stumbled into something that I like to pair with Ungrading: giving them their final exam grades early. Here is the process.

• I have them to their final grade reflection in the penultimate week of class after all assignments have been collected, and I get them feedback the same week. There are a lot of questions that help them reflect on their learning, but the two that are relevant here are (1) “According to the grading guidelines for this course, what grade do you think you currently have?” and (2) “What grade do you expect to have by the end of the semester? (and indicate which of the class assignments you intend to improve upon to get there)”.
• I can then give them feedback on how they can attain the grade from (2) (in case they were mistaken about anything). This forms a contract of sorts.
• I like how it focuses the students at the end of the semester. They know exactly what they need to work on.
• I imagine that it reduces stress for the students—they (mostly) know their grade for the semester before they leave for the semester.
• This definitely reduces stress for me. I don’t like the be the bad guy by giving an unexpectedly low grade. This way, I (mostly) don’t have to do that—I can simply tell them what they need to do to get the grade they want and help them work to that.

I say “mostly” above a couple of times because this is not perfect. My students have deadlines tomorrow, and some of them are scrambling to get their work done. If they don’t get their work all done, I am going to give them the most honest grade I can according to the grading guidelines, and this will be a surprise for the students. In some cases, I can tell them “If you do task A, you get grade X; if you additionally do task B, you get grade Y,” but there are some cases where this isn’t feasible.

The biggest advantage, though, is learning. This really leverages the motivational power of grades. Students know that if they want to get grade Y, they need to learn how to do tasks A and B in the next two weeks. They have mostly been successful at this now, which is good because they need to do the work now because they hadn’t done it earlier in the semester.

CUREs Update and Celebrating Scholarship and Creativity Day

Tomorrow is my school’s Celebrating Scholarship and Creativity Day (CSC Day), which is one of my favorite days of the year. We don’t have classes, but rather spend the day on presentations of undergraduate research. It is a lot of fun, and it is a great way to end the year.

I am doing a CURE in two of my classes, and they are doing poster sessions tomorrow. Here is a summary of what I think about both of them.

• My elementary education students are studying a game played on graphs. This has been a sensational success. We have spent about one-third of the semester on it (every Friday for a Monday-Wednesday-Friday class), and they have produced good results. We have five teams presenting posters tomorrow, each with four students. The students enjoyed the experience, I enjoyed the experience, and I think that this created an excitement for math in some of the students that will carry over when they are teachers. Moreover, one of the five teams presented their work at an undergraduate research conference, and they were likely the first elementary education majors every to present at the 45-year old conference.
• My linear algebra students are working on the NIEP. There are two teams presenting tomorrow. I am less certain that this was a success. The students did really well—each team came up with something interesting and developed a skill they wouldn’t have learned otherwise. However, I did not devote as much time to it as I had hoped, since there is a lot to learn in linear algebra. As planned, we spent about one-quarter of the semester’s time on this problem. Still, it didn’t seem to be that much. One issue is that we worked on it for a full class period every other week; perhaps we should have worked on it for half a class period every week.

I am teaching both of these classes next semester again, and I am definitely going to do a variation of what I did for my elementary education students. I am going to ask my linear algebra students what they thought of the CURE next week, although I am leaning toward continuing to do something like it again next year. They really did seem to gain something that they wouldn’t have gotten in class.

One-on-One Meetings: Update

I have been continuing doing one-on-one student meetings, and I have some new thoughts.

First, Lew Ludwig spoke at my school last weekend about AI. We are going to have to shift to more stuff like one-on-one student meetings if we want to be confident that we know what students can do. He handwrote a calculus limit on a Post-It, uploaded it to ChatGPT, and ChatGPT solved it from the scanned photograph.

Second, I rebranded these “Teaching Quizzes” this semester for my elementary education class. I instruct them to plan for the quizzes as if they were teaching the material as a teacher. This has improved their preparation since last semester. I still give them a lot of grace to think through the problems (it isn’t as polished as it should be if they were actually teaching).

Finally, the meetings still take a lot of time. Because of this, I started adding days in class where the students practice for the Teaching Quizzes. This allows me to intentionally look for students who already understand it, and I can give them credit for having done it in class. This frees up my day, since then I don’t have to meet with them outside of class.

Giving them credit in class is not perfect—I am certain that I miss some great explanations and pass some iffy ones, but it is an improvement. Also, the iffy ones that I pass are probably “good enough”—the standard should be “Would the student learn a significant amount by having a one-on-one meeting with me?” I think that I am holding to this standard well—I might pass a student who is just missing a detail or two that I could help with during a Teaching Quiz, but it is usually not worth the time.

I still miss some good explanations, although I have another improvement for that: use my awesome course assistant to help with this. He is sharp and responsible, and I trust him.

To help more students pass, I started (1) telling them that they might pass out of their Teaching Quiz in class on the practice day prior and (2) publishing a video prior to the practice day on exactly what to do. Some of the students are clearly studying to take advantage of this, which is increasing the number of students who are passing during these practice days.

Finally, there is an additional benefit to passing more students during practice days: it allows me to better us my out-of-class time. I am only giving students 15 minutes for a Teaching Quiz, since that amount leads to roughly six hours of Teaching Quizzes per class (I typically have about 25 students). Instead of working with a solid student for 15 minutes to bring them from a 9.1 to a 9.3 (on some scale out of 10), I can use that 15 minutes to help a different student for 30 minutes to bring them from a 4.7 to a 8.3 (Note: I don’t actually have a 10-point scale. I am making up the numbers as I type them). I can do more good for those who need more help, and I think it is better to give 15 additional minutes to a struggling student than to give 15 minutes to a student who mostly gets it.

Systemic Improvements: Teaching Practice Inventories (Mostly) Replace Student Surveys

I am starting to think about systemic improvements that schools can make to improve a professor’s job. I will write about this from time to time.

Getting tenure requires some amount of evidence that you are an effective teacher. My school’s main tool for doing this is to use student surveys (aka end-of-semester student evaluations). This is problematic for a lot of reasons, but it persists because it is both (1) habit and (2) cheap.

Moreover, it is unclear that student surveys incentivize professors to teach well. My personal experience says that students still enjoy being lectured to, although my personal experience says that this is diminishing quickly (students are starting to like active classrooms much more than ten years ago).

A second issue: I teach a school that prides itself on teaching. I have done two observations where I walk by all classrooms on my campus that are in use during a certain teaching period and observe what I see. In my first observation, 11 of 12 observed classes had a teacher standing in front of the class lecturing. In my second, it was 8 of 9. I only observed five seconds of each class at best, but this does seem to be a very tiny bit of evidence that professors, even at a teaching school, still lecture at a high rate.

I have a two-pronged solution:

1. We still need a venue for students to let us know if there are severe problems, so I think that we should have some sort of a student survey. There should be at most three options. The default option would be something like “This class was fine,” with no explanation required. The two other options would require an explanation, and they would be called something like “This class was absolutely exception” and “There are serious red flags with this class.” We need better language, but the former is intended for classes that are going to be, say, one of the top three experiences a student has in their college career, whereas the latter is to catch cases of verbal abuses, neglecting job duties, etc. This could be done by something akin to Google Forms.
2. A cheap solution to both measure and incentivize good teaching is teaching practice inventories. Here, teachers just reflect on how closely their teaching aligns with the literature on existing best practices for teaching. Moreover, they can plan their classes so that they get better scores—which means that they will be using better teaching methods.

We will still need class observations and such, but that is being held constant. This will help catch cases where people blatantly lie on their teaching, although that is always a risk (as there is always a risk that people will inflate their student surveys by bringing in cookies at the end of the semester).

One other drawback is that I don’t know of any teaching practice inventories for the humanities, arts, and the social sciences. I think that that is a solvable problem, though.

Feedback is welcome!

Next Step for Productivity

This week is a bit of a crazy week for me. Registration is happening, which means that I am meeting with advisees a lot. Thesis defenses are next week, so I am meeting with my thesis students. There are also math conferences this weekend and next.

This is how it is for me every April, since these are all annual events. Each year, it takes me a bit by surprise. This will change, since I am going to start reminding myself next year of the busy times. During these times, I will make sure that I am not expecting too much of myself for research/committees/etc.

Updated Time Blocking Technique

I have updated my time blocking this year. First, a brief summary of what I have been doing.

• My to-do list is a text file that I edit with Vim. I love Vim because I can quickly edit my schedule with just a few keystrokes.
• I have a time blocking template that I update every semester. This template has all of my class times, recurring meetings, and tasks that I want to do at a certain time every week. I use a simple python script and a cron job to automatically append it to my to-do list at 2 am-ish every Thursday morning.
• On Thursday or Friday (it is scheduled for Friday, but I often can’t wait) I time block my entire schedule for the next week.
• For items that will not happen in the next week, I have python scripts for each month. I also have a script called “MonthlyToDo.py” for tasks I do every month that gets run by a Cron job (and includes a line that reminds me to run, say, the python script for April). For instance, if I have a talk that I am giving in May, I will edit the python script for the month of May, and then it will remind me when I run the script.

All of the python that I do is ridiculously simple.

Here are the changes:

• I used to have a python script for every day. For instance, if I want to remind myself to grade a particular assignment on April 6th, I might put it in “Daily6ToDo.py” rather than “AprilToDo.py.” I have done away with this, though, in favor of having a list called FUTURE DATES at the end of my To-Do list. This list contains all of the reminders for the next two months or so. This is just easier than having to open up another file (daily or monthly) or record thoughts.
• Related: I now run my monthly python scripts two months in advance. It is the end of March, and I just ran the python script for May (I ran April’s last month). Basically, I want more access to the near future in my main file.
• Related: in the part of the weekly template (on Fridays) where I set aside time to time block the next week, there is space for “NEXT MONDAY,” “NEXT TUESDAY,” etc. This allows me to easily copy down things for next week exactly where I will need them.

These bullet points all have a theme of “store more information in the main To-Do list.” This has been helpful, since it is less switching between files, and I can more easily figure out what I should be thinking about in the future.

The last change is that I am starting to keep track of projects I need to do in my To-Do list. I am starting to use Trello more to keep track of projects that take more than 30 minutes to do. I simply added daily reminders that I need to check Trello in my weekly template.

Goal-less Questions

Kelly O’Shea, a long time ago, blogged about goal-less questions. I have used them before, and then I forgot about them during my sabbatical.

I am using them in linear algebra on take-home quizzes and exams. Such a quiz might consistent entirely of . A student’s task is then to write questions and answer it to show off that they know the learning outcome (there is a list of all of the 15 course learning outcomes for the semester). For instance, they might ask “Is this linear transformation an isomorphism?” (a learning goal) and then answer it. They might then ask, “What is the determinant of this linear transformation?”, write this function as a matrix, and find its determinant. Students might answer multiple learning outcomes with one question, and they might demonstrate the same learning outcome multiple times on the same quiz (they end up row reducing a lot, although I only give them small matrices for the quizzes).

There are advantages and disadvantages to this. The main reason why I am doing this for linear algebra, and I think that it is an advantage, is that I think it helps students to see the relationships among different ideas (I can find the matrix of a linear transformation, row reduce it to determine its kernel, and then determine whether the function is one-to-one based on the kernel).

One main disadvantage is that I don’t have the same control over what they do. In particular, I have noticed that students seem to avoid systems of equations that have an infinite number of solutions. However, I have two solutions to this:

1. I could create separate learning goals next semester that address this (three learning goals: I can solve systems equations with no solutions, with one solution, and with infinite solutions).
2. I could not care that they don’t have these details down.

I am opting for the latter for this semester, since I don’t want to add learning outcomes. I will probably do the same for next semester, too, though, since I think that this has been successful. The students are getting a good idea of the big picture, and I don’t want to ruin it by having next semester’s students focus too much on the details.