“Always…uh…never…forget to check your references.”
Dr. Meredith Real Genius 1985

The scene I’m quoting made just about everyone smile. The bright young student meets Dr. Meredith who says, “a bit of advice.” The student pulls out his notepad and says “Oh, uh, thank you?” eagerly awaiting the information. Dr. Meredith says “Always…uh…never…forget to check your references.” The student smile says “Uh, OK…thank you. I’d better be going.” and wanders off without taking a note (Real Genius). The scene works because everyone knows, even the non-academics, that professional researchers ALWAYS check their references.

When I was an undergraduate, this topic came up all the time. Professors saying reviews are great places to start, However, always go back and check the sources. The PI in whose lab I worked had a quote from Frank Westheimer above his door that said, “A month in the laboratory can often save an hour in the library.” I’ll just let you think about that one.

However, something I have come to realize is that even though we know, we should we don’t “always check our references” nearly as much as we should. I know I have been guilty of it a time or two.  Why is that? As we move through our careers and lives, we have less and less time. So, we know that review went through peer review, so we don’t need to double check it do we? We have a colleague or friend that we respect, and we know they’re not trying to mislead us so sure, we don’t have to double check them, and so on and so on.

The problem is that we’re all human we make mistakes, not intentionally maliciously not even frequently, but it happens. One of the biggest reasons to check your references is to help each other. When we don’t, we let mistakes perpetuate over and over.

I will highlight this with a few examples from my own experience. How long does it take to learn something? Variations of this question come up all the time.  The number of times I have heard someone say 10,000 hours, which would be 1,250 days if you worked a solid 8 hours a day, is more than I can count. However, this is not true the 10,000-hour mark came from work by Anders Ericsson a professor of psychology at Florida State. His work looks explicitly at top-level performers, Olympic caliber athletes, chess grandmasters, etc., people at the very tip-top of fields. He asked how long it took them to reach that level of excellence and that turned out to be 10,000 hours. However, becoming an expert master is different from how long it takes to learn something. If you want to learn something, it only takes about 20 hours. I won’t go into the whole story of how 10,000 hours to become a top expert became 10,000 hours to learn something since Josh Kaufman does a much more entertaining job in his TED talk.

The next one is famous I’ve seen it referenced in books, newspaper articles, and many presentations. The study goes like something like this at Harvard in 1953 (or maybe Yale in 1979) graduates of the business school were asked about their goals and whether they had written them down.  The researchers created three groups from the graduates interviewed; goals and plans written down, goals that were not written down, and without goals. Ten years later these graduates were interviewed, graduates with goals were earning 3X as much as those without goals, while those that had written down their goals were making 10X as much as those without goals. I was going to use this study in a presentation I was giving. For reasons I won’t go into I needed the original reference. I quickly discovered that I couldn’t find it by doing a quick online search. So, I went in search of other sources that used it they all quoted other sources that it turned out quoted it from other sources.  Once I even went in a complete circle going through references and ending up back where I started.  Finding the source became a bit of an obsessive challenge with more in-depth searches and longer searches.

As I searched, I noticed several interesting things.  There were many similar but not identical stories.  The study happened at Harvard; it happened at Yale, it was conducted in 1953 or 1979.  The participants were reinterviewed 10 or maybe 20 years later.  The most telling piece of information I finally found was a post from Laura Sider a librarian and Associate Director of Frontline services at Yale University where she said,

It has been determined that no “goals study” of the Class of 1953 actually occurred. In recent years, we have received a number of requests for information on a reported study based on a survey administered to the Class of 1953 in their senior year and a follow-up study conducted ten years later. This study has been described as how one’s goals at graduation related to success and annual incomes achieved during the period.

That’s right this famous study never happened, you can see her full statement here. The goal setting study has become such a cultural phenomenon that the Yale library felt the need to state that it never happened. Where the story originally came from I don’t know. Mike Morrison has identified “two early reporters” Mark McCormack’s (What They Don’t Teach You in the Harvard Business School) and Brian Tracy’s (Goals!). You can see his full report here. I don’t know if he is correct or not once I convinced myself this study never happened I was able to escape the rabbit hole. However, it’s possible this false study might be causing harm.  Recent research actually out of the Harvard business school suggests that we have been ignoring the potentially harmful side effects of goal setting. The only thing clear to me is that we need more real research on goal setting.

Did any of this surprise you? Do you check your references? Have you ever checked a reference and discovered something you did not expect? I wonder how much we are taking for granted? Should we be checking absolutely everything? Maybe it’s time to work with some librarians and see if we can find out?

Thanks for listening to my musings

The Teaching Cyborg