Make It Stick: Test to learn

This post is one of an (intended) collection interacting with the book Make It Stick: The Science of Successful Learning. All in the collection are linked from the Introduction post.

[Edited to add, a few days after initial post: this New York Times article is an interesting read on the subject.]

Test…to learn?

First–please forget about standardized testing and “teaching to the test”. That’s a discussion mostly unrelated to “testing to learn”.

The authors of Make It Stick tell the story of a group of university employees challenged to give the locations of the fire extinguishers closest to their offices. Few could do so, but one professor–who’d worked there for 25 years–looked for the one closest to his office. He found it–right next to his office door.

Many are the times that I’ve listened to a good lecture or a good sermon, and an hour later remembered that it was especially insightful or interesting–but would have been hard pressed to tell you even the main points of the presentation. Ditto for any number of wonderful books I’ve read.

So…what’s up? Why can’t I remember what I’ve read or heard, and why can’t a university professor remember the fire extinguisher right next to his office door?

Perhaps it would help to listen to the lecture again, or read the book again–but it probably won’t help a lot. What will help, according to the evidence, is testing–or, if that term makes your skin crawl, “active recall”. When you’re forced to produce, or even to simply recall an answer, its “stickiness” in your long-term memory goes way up. The authors of Make It Stick cite one study of people asked to memorize random lists, to be tested a week later. Everyone immediately forgot about half of the list–not too bad, for a large, random list. But those who were tested immediately after seeing the lists did substantially better a week later than those who weren’t, a benefit that came through the simple act of actively recalling the information.

Even more interesting was what happened when people were tested several times immediately after initially learning the list. The result: after the initial drop-off, the repeatedly tested group forgot basically nothing more in the intervening week. Their never-tested counterparts, in the same time period, lost substantially more.

Perhaps more interesting than random lists were the stories of “testing to learn” in actual classroom settings. The upshot: it’s quite effective in that setting, supported both by controlled studies and anecdotal evidence.

The thing is, the act of remembering something is not just pulling a file out of your mental archives. Instead, when you recall knowledge or a skill, it’s as though you walked through your personal library, asked several people where the desired information might be, and finally found the file you were looking for. After finding it, reading it, and adding some new notes, you re-file the document in a slightly more prominent spot, making a mental note of where you can find the information the next time you want it. The simple act of recalling something tells your brain that it’s important, and should be retained and made as accessible as possible.

Besides strengthening what you already know, testing your knowledge is a great way to figure out what you don’t know, and to work on learning/re-learning them.

Competitors of “test to learn”

“Test to learn” does have a few competitors in the learning-techniques arena–competitors, at least, in their popularity. In effectiveness, there’s little contest. Perhaps you’ve tried some of them. One is frequent re-reading–perhaps with the aid of a highlighter, so that you can read only the important points. Another is “massed practice”, focused review on one narrow skill or set of facts. In brief: though a quick re-read after some time might be a helpful warm-up, but evidence shows that re-reading is unlikely to offer much benefit in long-term retention. Massed practice, which we’ll visit in more depth in another post, shows great short-term benefits, but those benefits quickly fade into oblivion.

I find analogies helpful here: “test to learn” is like an apprenticeship, offering lots of uncomfortable challenge but resulting in lifelong skill. Massed practice is like the “weekend warrior” approach to working out: it feels like an accomplishment, but doesn’t add a lot of value over the long term. And re-reading is like some of my magazine subscriptions used to be: magazines showed up at my door frequently, I sometimes actually read them, and those I did read seldom provided any lasting value. And so, despite my extensive exposure to magazines related to science and finance, I somehow managed to avoid developing exceptional genius in either field!

Making it practical

So…if you’re a learner, what does it mean to “test to learn”? If you’re helping others learn, how might you apply this? It’s worth pausing for a bit to think about this before proceeding. (Another lesson, mostly for another day, taken from Make It Stick.)

A few practical ideas include:

As a learner:

  • As you read, or perhaps after finishing a chapter, write your own quiz. If you’re in a rush, even “writing” (and taking) the quiz in your own mind will be far more helpful than doing nothing. What are the “big ideas” of this text? Are there facts or lines of reasoning that you need to remember?
  • As much as possible, test information in the way you’ll use it. The book offers the vivid example of a police officer who encountered an armed suspect. In a well-practiced maneuver, he broke the other’s hold on his weapon and seized it–then, in equally well-practiced fashion, he handed it back to the suspect, just as he’d done hundreds of times while practicing the maneuver with other officers. Fortunately, he recovered quickly enough to take possession of the weapon again and keep it.
  • If you can and if the information is important to remember, come back in a few days and ask yourself the same questions–an example of spaced repetition, another powerful tool we’ll talk about later.
  • Be a “mean” tester: make yourself work to answer your self-test! Some questions may have simple, factual answers–but as much as possible, lean toward questions that make you apply what you know and integrate it with the rest of your knowledge. The harder you have to work to answer these questions, the more “sticky” the learning will be.
  • As a corollary, don’t be content with answers that you don’t understand. E.g., don’t just cram a math formula into your head by rote, but understand why it it works the way it does.
  • If you’re reading a book with end-of-chapter questions, by all means use them, at least as a starting point!
  • Bonus points if you enter your questions and answers in Anki, Memrise, or another spaced-repetition system and keep on using it, for an ultimate “mind hack” using the powers of both testing and spaced repetition.

As one who helps others learn:

  • Use tests as a teaching tool. This doesn’t rule out tests as an assessment tool, but those are a different game. Take advantage of the power of testing in increasing recall, and save a bit of time for a low-stakes quiz at the end of class.
  • Use spaced repetition along with testing: have people recall last week’s concepts, and those from a few weeks ago.
  • Help learners develop the skills of “testing” themselves.
  • If you’re a classroom teacher, use comprehensive tests (in this case, we’re talking about the tests normally used for “assessment”). Yes, your students might hate you for it. 🙂 But their true learning will go up substantially. Of course, some of the other points mentioned make this much more “doable” than it might otherwise be.

Your thoughts?

  • How important is it to retain what you read or hear?
  • How can you apply the power of testing to help you?
  • If your job is helping others learn, how might you facilitate their learning through testing?
  • What other applications of this idea might there be?
  • …and where might it go wrong?

Make It Stick: What is learning, anyway?

This post is one of an (intended) collection interacting with the book Make It Stick: The Science of Successful Learning. All in the collection are linked from the Introduction post.

So, I’m thinking and writing about the book Make It Stick: The Science of Successful Learning. The title itself raises one question that’s worth thinking about: what is “learning”? What does it mean to “learn”?

Learning is most effective when it’s effortful, something we’ll examine more closely later. For the moment, though, that has one simple implication: this post will be most useful for you if you stop and think for a bit before continuing. What does it mean to learn? Why would you want to learn?

I thought of several kinds of learning:

  • The learning of a skill: welding, disease diagnosis, software design, coding in C++.
  • The learning of knowledge: the history of the USA, the mechanics of the immune system, the value of pi.

And a few things that don’t feel like they’re “really learning”:

  • Shallow memorization of facts. This is hard to pin down, but a story from my “physical sciences” class might help. In a section on electricity, the assignment was to wire up circuits in various configurations and record the relative brightness of a bulb tied into those circuits. Wanting to save some time (and, probably, being an arrogant kid at the time), I started writing in some of the results before we’d actually gotten around to doing the experiments, astonishing some of the others in my group. I’m guessing we all “knew” the facts about voltage, current, resistance, and brightness, but in that particular case, cocky-kid-Joel had that knowledge more available for use than the others in the group.
  • Extremely short-lived learning. This is the kind of thing you get when you’ve crammed for a test, did well, and lose almost all of it within days. Or, when you get to the end of an enjoyable lecture or sermon and think, “That was really good! But…um…what was it about again?”
  • Although it’s still learning of a sort, “misdirected learning” is another interesting category. Examples include learning how to read a word in another language when your top desire is to produce it (they’re connected, but different skills), or the ability to remember a fact when asked a question phrased “just right”, but not when you want to use it in real life.

Thinking of those examples, it’s interesting to note their interrelationships: skills usually depend on some sort of base of knowledge, and the ability to acquire knowledge is a skill. Shallow memorization, short-lived memory, and misdirected learning form their own constellation of interwoven cause and effect.

And so, here’s my tentative definition of learning. I…um…don’t remember whether the authors of Make It Stick actually define what they mean by “learning”, but I’m guessing their definition would at least be similar to mine.

Learning is the long-lasting acquisition of skills or knowledge, in a form accessible and useful when such skills or knowledge are relevant.

My first draft of this definition specified that the skills or knowledge should “affect the way one thinks or acts”–but that leaves a lot of ambiguity. For example, if you learn pi to a thousand digits, does that make a difference in the way you think or act? If you know that Laika was the first dog in orbit, does that make a difference? Like Scott Young, I actually like “useless” knowledge and believe that a lot of “useless” knowledge is useless only because one hasn’t yet found a use. Digits of pi can be wonderful if you need a quick proxy for random numbers. Laika’s story is one episode in the Space Race story, a part of what’s shaped Russia, America, and the world. The revised definition lets Laika hang out in the back of your mind most of the time, informing your understanding of Russo-American history and waiting to emerge when she’s needed for a Trivial Pursuit question. It lets the digits of pi wait around as static, “useless” but accessible knowledge that you’ve learned. But, if you’ve “learned” the math of voltage and current but can’t predict that a higher voltage will light a bulb more brightly, that knowledge isn’t in a useful form and thus doesn’t count as “learned”.

What do you think–what does it mean to “learn” something? What categories have I missed? Are there times when it’s worthwhile to “learn” something in a way that doesn’t meet my definition?

Make It Stick: Introduction

I recently read the book Make It Stick: The Science of Successful Learning. I found the book impressive, and it joins the short list of those I’ll too-enthusiastically recommend to anyone who will listen–especially those who are “learners” or “teachers”, groups which should encompass most people.

Make It Stick is a well-woven collection of stories and hard data on what the science actually says about how best to learn.This includes a number of things that run against our intuition, or even standard study advice. A few examples:

  • Always retiring to the same quiet place to study may actually hurt learning.
  • Learning something “easily” probably means you’re going to lose it soon.
  • Repeatedly re-reading a text (or presumably, re-listening to a lecture) is terrible in terms of learning power.
  • Tests are good–at least, if approached in the right way.
  • One of my favorites: spaced repetition is really powerful.
  • Reflection–digesting, re-stating, connecting, and applying “learning” is fairly powerful in facilitating actual learning.

And so…I plan to take the book’s advice to heart and “reflect” in writing on its contents. The benefit is primarily my own: if you wish, you can buy the book, or search for its title and find any number of summaries. I’d encourage doing both. In writing about these ideas, I hope to “learn” them, making them part of my own mental toolkit. If someone else benefits as well, I’m even happier!

For a longer preview of the ideas presented in the book, see this article. And then–start thinking about how to apply the ideas presented, and join me in the conversation!

Posts Inspired by Make It Stick: