“It is a capital mistake to theorize before one has data.”
Arthur Conan Doyle (via Sherlock Holmes)

Several years ago, I was attending a weekly Discipline-Based Educational Research (DBER) meeting. Two senior faculty members led and organized the weekly meetings.  Both faculty members had trained in STEM disciplines.  One had received their educational research training through a now-defunct National Science Foundation (NSF) program, while the other was mostly self-taught through multiple calibrations with educational researchers.

The group was discussing the design of a research study that the Biology department was going to conduct.  One of the senior faculty members said if they were serious, they would design a double-blind study.  The other senior faculty member said that not only should they not do a double-blind study, but a double-blind study was likely a bad idea. I don’t recall the argument over double-blind studies in education ever getting resolved. We also never found out why one of the faculty members thought double-blind studies were a bad idea in educational research.

Double-blind studies are a way to remove bias. Most people know about them from drug trials.  Educational reform is not likely to accidentally kill someone if an incorrect idea gets implemented due to a bias in the research.  However, a person’s experiences during their education will certainly have a lifelong impact.  While double-blind studies might be overkill in education research, there is the question of what is enough.  As I have said before, it is the job of educators to provide the best educational experience possible; this should extend to our research.

How do faculty know how they should teach? What research should faculty members use?  Should we be concerned with the quality of educational research? Let me tell you a story (the names have been changed to protect the useless).  A colleague of mine was looking for an initial research project for a graduate student. My college told me about a piece of educational “research” that was making the rounds on his campus.  Alice, a well-respected STEM (Science Technology Engineering and Math) faculty member, had observed her class.  She noted what methods of note-taking her students were using.  At the end of the semester, she compared the method of notetaking to the student’s grade. On average, the students that used the looking glass method of notetaking had grades that avraged one letter grade lower than the other method of notetaking.

Alice told this finding to a friend the Mad Hatter, a DBER (Discipline-Based Education Research) expert.  The Mad Hatter was so impressed with the result that he immediately started telling everyone about it and including it in all his talks.  Now because Alice did her study on the spur of the moment, she did not get research approval and signed participation agreements.  The lack of paperwork meant that Alice couldn’t publish her results.  With such a huge effect, my colleague thought repeating this study with the correct permissions so that it could be published would be perfect for a graduate student.

They set-up the study; this time, to assess what methods the students were using to take notes, they videotaped each class period.  Additionally, the researchers conducted a couple of short questioners and interviewed a selection of the students.  After a full semester of observation, the graduate students analyzed the data. The result, there was no significant difference between looking glass notetaking and all the other types.  Just a little while ago, I saw a talk by the Mad Hatter. It still included Alice’s initial results.  Now the interesting thing is neither Alice nor the Mad Hatter would have excepted Alice’s notetaking research methodology if it was a research project in their STEM discipline.  However, as an educational research project, they were both willing to take the notetaking results as gospel.

While there is a lot of proper educational research, researchers have suggested that a lot of faculty and policymakers have a low bar for what is acceptable educational research.  The authors of We Must Raise the Bar for Evidence in Education suggest a solution to this low bar in educational research.  Their recommendation is to change what we except as the basic requirement of educational research.  Most of the author’s suggestions center around eliminating bias (the idea at the core of the double-blind study) their first suggestion is,

“to disentangle whether a practice causes improvement or is merely associated with it, we need to use research methods that can reliably identify causal relationships. And the best way to determine whether a practice causes an outcome is to conduct a randomized controlled trial (or “RCT,” meaning participants were randomly assigned to being exposed to the practice under study or not being exposed to it).”

One of the biggest problems with human research, which includes educational research, is the variability in the student population.  As so many people are fond of saying, we are all individuals.  By randomly assigning individuals to a group, you avoid the issue of concentrating traits in one group. 

Their second suggestion is, “policymakers and practitioners evaluating research studies should have more confidence in studies where the same findings have been observed multiple times in different settings with large samples.”  The more times you observe something, the more likely it is to be true (there is an argument against this, but I will leave that for another time.)

Lastly, the authors suggest, “we can have much more faith in a study’s findings when they are preregistered. That is, researchers publicly post what their hypotheses are and exactly how they will evaluate each one before they have examined their data.”  Preregistration is a lot like the educational practice used with student response systems were the student/researcher is less likely to delude themselves about the results if they must commit to an idea ahead of time.

If we are going to provide the best educational experiences for our students, we need to know what the best educational experiences are.  However, it is not enough to conduct studies. We need to be as rigorous as possible in our studies.  The next time you perform an educational research project, take a minute, and ask yourself how I can make this study more rigorous.  Not only will your students benefit, so will your colleagues. Thanks for Listing to My Mussing
The teaching Cyborg