It’s all practice

On quizzes and big exams

When I was in college, most of my grades were based on some combination of assignments, quizzes, and–heavily weighted–the mid-term and final exams. If I bombed a quiz, I wasn’t happy about it, but it wasn’t a major disaster. If I’d completely failed a final exam, though, it would have been a Big Deal.

In the years after college, I sat for the three levels of the CFA exams. Multiple months of focused study, a cost of $600 for a given exam, a six-hour proctored exam given only once* a year, driving up to Kansas City the night before to stay in a hotel close to the test center…for me, passing these exams qualified as a Big Deal.

The thing is, I didn’t actually sit for three CFA exams–I took somewhere between six and a dozen exams, across the three levels. Fortunately for my sanity, my work, and my marriage, most of these exams were “practice” exams–exams that I “proctored” for myself, that I graded myself, and that I used to figure out what areas I still needed to work on for a given level. By the time I sat for the “real” exam, I at least had a good idea of the “shape” of my knowledge, and a very general idea of how I might fare on the officially administered exam.

When I finally sat for a given level of the exam, failing it would not have been fun–but, if I’d failed and re-enrolled for the same level the next year, the first try would have given me both resources and lessons to help in the second round.

You see, life is practice. Very rarely is a “test” really a final exam. Losing a job, failing in business, damaging a relationship, missing a big opportunity–each is unpleasant, but each is recoverable

Galton boards and conscious navigators

I find it easy to think of life as though it’s a Galton board. You’ve probably seen one, though, like me, you might not have known what it was called. It’s a triangle of steel pins, where a marble dropped into the top of the triangle falls in a random sequence of “lefts” and “rights” on its way to the bottom of the board. And once a ball falls on the left side of a pin, a part of the right side of the board is forever closed off…and another leftward fall closes off more of the right forever. And there’s some truth in this. If you let yourself get addicted to drugs or alcohol, it’s really, really hard to move out of that section of the board.

Galton board
A Galton board

As like a Galton board as life may be, though, we aren’t falling marbles under gravity’s control. We can–through our own volition, and sometimes through others’ interventions, take unexpected paths through the maze. We can backtrack, take unexpected paths, sometimes even jump across gaps in the maze. We might be falling through Galton’s maze, and gravity and randomness can predict a lot about where we’ll end up–but they don’t necessarily get the final word. If I start with a couple of rightward choices, I could still end up at the far left of the board. Choices matter, but they’re usually much closer to quizzes than to final exams.

a Galton board with the ball taking unexpected paths
A conscious path

Practice well

Of course, your life will ultimately end, and it will be somewhere between “wonderfully lived” and “wasted”. But until the grades are turned in, it’s all practice.

Interesting reads

Feedback

Please let me know your thoughts. If you want to affirm or elaborate on what I’ve written, to express disagreement with it, or to help sharpen my writing or thinking, I’d love to hear from you.

Footnotes

* Actually, Level 1 of the CFA exam is given twice a year, in June and December; the subsequent levels are only in June.

“We do hard things”

This morning at the breakfast table, my eldest daughter “needed” help with her food. We helped her get started, but for finishing the challenging task of stirring her oatmeal, her mom encouraged her: “We do hard things.”

Without premeditation, we’ve started using this phrase recently in our family. It may not be as hardcore as Sara Blakely’s “What did you fail at today?” question, but it’s a lovely phrase. It encourages a growth mindset: “hard” is good, not bad. It makes it a part of our daughters’ identity, much like saying “I don’t” instead of “I can’t” re-wires the brain to help with goals. And it builds a shared identity–we, the Iwashige family, do hard things!

Project Euler: Programming skills through play

I recently learned about Project Euler, and have found it addictive. It’s a wonderful site for learning programming. It’s not powerful tools that make it great. The site’s design looks ancient; it has no programming editors; it doesn’t even have any tutorials. But it has wonder, and a series of progressively harder challenges, and lets you discover what you need to learn in order to solve those challenges. Basically, it’s a site to support exploratory learning–learning of the sort that happens when you start a new job, a new business, or a new hobby. Learning of the kind that delights when you figure out the problem, learning that sticks. Learning that happens when you dive in, get your hands dirty, and learn what you need to know along the way.

James Somers wrote a good article about Project Euler at The Atlantic. I’m still in the first dozen or so problems out of over 500, but if you want to follow my progress, my profile is here and my friend key is 858072_k8PHIvJ2fpmYfHwCObWWZ59Ql4jckfYS.

The use of incoherent fragments

When I started this blog, it was a tentative thing–a platform whose significance I downplayed, but one through which I secretly hoped I’d accomplish something world-changing. I didn’t know what, but after all…who really needs just another “data dump” made public? I wanted to write valuable things, important things, things that earned respect.

Writing content that’s worth reading is a great idea. Unfortunately, this idea has translated into my writing very little. At the same time, I’ve gained appreciation for “simmers” of ideas: experiences like watching a dozen TED talks at a time, reading Alex Steffens’s Worldchanging book, or reading half a dozen books on related topics in a weekend. In this simmer, ideas collide, interact, and combine in wonderful ways. Ideas don’t need to be long; they don’t need to be fully developed to be useful. They just need to interact productively with other ideas. A tornado of index cards might have more value than a well-organized essay–even though the former looks like chaos, and is unlikely to win any literary prizes.

So…I’m going back to the origin of “blogging”. “Blog”, of course, is a contraction of “weblog”–in my interpretation, a stream of brief thoughts released to the Web. It may be that something grand emerges, but I’m not planning on it. What I am planning is that I’ll once again write to help order my understanding of the world. And sometimes, I’ll just share with little comment the small, delightful morsels that I’ve found on my experiential plate.

Make It Stick: Test to learn

This post is one of an (intended) collection interacting with the book Make It Stick: The Science of Successful Learning. All in the collection are linked from the Introduction post.

[Edited to add, a few days after initial post: this New York Times article is an interesting read on the subject.]

Test…to learn?

First–please forget about standardized testing and “teaching to the test”. That’s a discussion mostly unrelated to “testing to learn”.

The authors of Make It Stick tell the story of a group of university employees challenged to give the locations of the fire extinguishers closest to their offices. Few could do so, but one professor–who’d worked there for 25 years–looked for the one closest to his office. He found it–right next to his office door.

Many are the times that I’ve listened to a good lecture or a good sermon, and an hour later remembered that it was especially insightful or interesting–but would have been hard pressed to tell you even the main points of the presentation. Ditto for any number of wonderful books I’ve read.

So…what’s up? Why can’t I remember what I’ve read or heard, and why can’t a university professor remember the fire extinguisher right next to his office door?

Perhaps it would help to listen to the lecture again, or read the book again–but it probably won’t help a lot. What will help, according to the evidence, is testing–or, if that term makes your skin crawl, “active recall”. When you’re forced to produce, or even to simply recall an answer, its “stickiness” in your long-term memory goes way up. The authors of Make It Stick cite one study of people asked to memorize random lists, to be tested a week later. Everyone immediately forgot about half of the list–not too bad, for a large, random list. But those who were tested immediately after seeing the lists did substantially better a week later than those who weren’t, a benefit that came through the simple act of actively recalling the information.

Even more interesting was what happened when people were tested several times immediately after initially learning the list. The result: after the initial drop-off, the repeatedly tested group forgot basically nothing more in the intervening week. Their never-tested counterparts, in the same time period, lost substantially more.

Perhaps more interesting than random lists were the stories of “testing to learn” in actual classroom settings. The upshot: it’s quite effective in that setting, supported both by controlled studies and anecdotal evidence.

The thing is, the act of remembering something is not just pulling a file out of your mental archives. Instead, when you recall knowledge or a skill, it’s as though you walked through your personal library, asked several people where the desired information might be, and finally found the file you were looking for. After finding it, reading it, and adding some new notes, you re-file the document in a slightly more prominent spot, making a mental note of where you can find the information the next time you want it. The simple act of recalling something tells your brain that it’s important, and should be retained and made as accessible as possible.

Besides strengthening what you already know, testing your knowledge is a great way to figure out what you don’t know, and to work on learning/re-learning them.

Competitors of “test to learn”

“Test to learn” does have a few competitors in the learning-techniques arena–competitors, at least, in their popularity. In effectiveness, there’s little contest. Perhaps you’ve tried some of them. One is frequent re-reading–perhaps with the aid of a highlighter, so that you can read only the important points. Another is “massed practice”, focused review on one narrow skill or set of facts. In brief: though a quick re-read after some time might be a helpful warm-up, but evidence shows that re-reading is unlikely to offer much benefit in long-term retention. Massed practice, which we’ll visit in more depth in another post, shows great short-term benefits, but those benefits quickly fade into oblivion.

I find analogies helpful here: “test to learn” is like an apprenticeship, offering lots of uncomfortable challenge but resulting in lifelong skill. Massed practice is like the “weekend warrior” approach to working out: it feels like an accomplishment, but doesn’t add a lot of value over the long term. And re-reading is like some of my magazine subscriptions used to be: magazines showed up at my door frequently, I sometimes actually read them, and those I did read seldom provided any lasting value. And so, despite my extensive exposure to magazines related to science and finance, I somehow managed to avoid developing exceptional genius in either field!

Making it practical

So…if you’re a learner, what does it mean to “test to learn”? If you’re helping others learn, how might you apply this? It’s worth pausing for a bit to think about this before proceeding. (Another lesson, mostly for another day, taken from Make It Stick.)

A few practical ideas include:

As a learner:

  • As you read, or perhaps after finishing a chapter, write your own quiz. If you’re in a rush, even “writing” (and taking) the quiz in your own mind will be far more helpful than doing nothing. What are the “big ideas” of this text? Are there facts or lines of reasoning that you need to remember?
  • As much as possible, test information in the way you’ll use it. The book offers the vivid example of a police officer who encountered an armed suspect. In a well-practiced maneuver, he broke the other’s hold on his weapon and seized it–then, in equally well-practiced fashion, he handed it back to the suspect, just as he’d done hundreds of times while practicing the maneuver with other officers. Fortunately, he recovered quickly enough to take possession of the weapon again and keep it.
  • If you can and if the information is important to remember, come back in a few days and ask yourself the same questions–an example of spaced repetition, another powerful tool we’ll talk about later.
  • Be a “mean” tester: make yourself work to answer your self-test! Some questions may have simple, factual answers–but as much as possible, lean toward questions that make you apply what you know and integrate it with the rest of your knowledge. The harder you have to work to answer these questions, the more “sticky” the learning will be.
  • As a corollary, don’t be content with answers that you don’t understand. E.g., don’t just cram a math formula into your head by rote, but understand why it it works the way it does.
  • If you’re reading a book with end-of-chapter questions, by all means use them, at least as a starting point!
  • Bonus points if you enter your questions and answers in Anki, Memrise, or another spaced-repetition system and keep on using it, for an ultimate “mind hack” using the powers of both testing and spaced repetition.

As one who helps others learn:

  • Use tests as a teaching tool. This doesn’t rule out tests as an assessment tool, but those are a different game. Take advantage of the power of testing in increasing recall, and save a bit of time for a low-stakes quiz at the end of class.
  • Use spaced repetition along with testing: have people recall last week’s concepts, and those from a few weeks ago.
  • Help learners develop the skills of “testing” themselves.
  • If you’re a classroom teacher, use comprehensive tests (in this case, we’re talking about the tests normally used for “assessment”). Yes, your students might hate you for it. 🙂 But their true learning will go up substantially. Of course, some of the other points mentioned make this much more “doable” than it might otherwise be.

Your thoughts?

  • How important is it to retain what you read or hear?
  • How can you apply the power of testing to help you?
  • If your job is helping others learn, how might you facilitate their learning through testing?
  • What other applications of this idea might there be?
  • …and where might it go wrong?

Make It Stick: What is learning, anyway?

This post is one of an (intended) collection interacting with the book Make It Stick: The Science of Successful Learning. All in the collection are linked from the Introduction post.

So, I’m thinking and writing about the book Make It Stick: The Science of Successful Learning. The title itself raises one question that’s worth thinking about: what is “learning”? What does it mean to “learn”?

Learning is most effective when it’s effortful, something we’ll examine more closely later. For the moment, though, that has one simple implication: this post will be most useful for you if you stop and think for a bit before continuing. What does it mean to learn? Why would you want to learn?

I thought of several kinds of learning:

  • The learning of a skill: welding, disease diagnosis, software design, coding in C++.
  • The learning of knowledge: the history of the USA, the mechanics of the immune system, the value of pi.

And a few things that don’t feel like they’re “really learning”:

  • Shallow memorization of facts. This is hard to pin down, but a story from my “physical sciences” class might help. In a section on electricity, the assignment was to wire up circuits in various configurations and record the relative brightness of a bulb tied into those circuits. Wanting to save some time (and, probably, being an arrogant kid at the time), I started writing in some of the results before we’d actually gotten around to doing the experiments, astonishing some of the others in my group. I’m guessing we all “knew” the facts about voltage, current, resistance, and brightness, but in that particular case, cocky-kid-Joel had that knowledge more available for use than the others in the group.
  • Extremely short-lived learning. This is the kind of thing you get when you’ve crammed for a test, did well, and lose almost all of it within days. Or, when you get to the end of an enjoyable lecture or sermon and think, “That was really good! But…um…what was it about again?”
  • Although it’s still learning of a sort, “misdirected learning” is another interesting category. Examples include learning how to read a word in another language when your top desire is to produce it (they’re connected, but different skills), or the ability to remember a fact when asked a question phrased “just right”, but not when you want to use it in real life.

Thinking of those examples, it’s interesting to note their interrelationships: skills usually depend on some sort of base of knowledge, and the ability to acquire knowledge is a skill. Shallow memorization, short-lived memory, and misdirected learning form their own constellation of interwoven cause and effect.

And so, here’s my tentative definition of learning. I…um…don’t remember whether the authors of Make It Stick actually define what they mean by “learning”, but I’m guessing their definition would at least be similar to mine.

Learning is the long-lasting acquisition of skills or knowledge, in a form accessible and useful when such skills or knowledge are relevant.

My first draft of this definition specified that the skills or knowledge should “affect the way one thinks or acts”–but that leaves a lot of ambiguity. For example, if you learn pi to a thousand digits, does that make a difference in the way you think or act? If you know that Laika was the first dog in orbit, does that make a difference? Like Scott Young, I actually like “useless” knowledge and believe that a lot of “useless” knowledge is useless only because one hasn’t yet found a use. Digits of pi can be wonderful if you need a quick proxy for random numbers. Laika’s story is one episode in the Space Race story, a part of what’s shaped Russia, America, and the world. The revised definition lets Laika hang out in the back of your mind most of the time, informing your understanding of Russo-American history and waiting to emerge when she’s needed for a Trivial Pursuit question. It lets the digits of pi wait around as static, “useless” but accessible knowledge that you’ve learned. But, if you’ve “learned” the math of voltage and current but can’t predict that a higher voltage will light a bulb more brightly, that knowledge isn’t in a useful form and thus doesn’t count as “learned”.

What do you think–what does it mean to “learn” something? What categories have I missed? Are there times when it’s worthwhile to “learn” something in a way that doesn’t meet my definition?