God Had It Easy

God had it easy compared to me.

In the movie "Oh, God" John Denver could not believe the unprepossessing George Burns was God, but he at least believed God existed.

I do not have that edge: no one believes Algebra failure rates can be reduced from 60% to 10%.

God did not win John over by levitating a sugar bowl in the diner, but He got him later when they were driving in the car and He made it rain. Inside the car.

All I can do is quote letters like this (with Margaret's kind permission):

Dear Ken, 

Back in the '90s I used your Homework Tutor program successfully with my Algebra I students who were behind. I was excited about the efficient progress they made and how quickly they got to the same level as my better students after using the program.

Now I have some new teachers in my school who need help with their poor algebra students. Do you know where I could purchase your software? 

Of all of the software I have used in the evolution of school math and computers, your Algebra I Homework Tutor was the most elegant and practical. 

Sincerely,Margaret LekseRoundup High School, MT

Nice, but no rain.Or so I thought. Maybe you missed it, too.

When I reached out to Margaret (I was already resurrecting the app) and asked her to say more about her experience. She made clearer what her letter said: a half-dozen kids had fallen so far behind in Algebra that they no longer could follow her class presentations. She put all the kids on my software so the strugglers would not be embarrassed, then each kid worked at their own level. In one month the strugglers got up to speed and they put the software aside until new material proved challenging.

Wow. Same (exclellent, btw) teacher, same content, same kids. They fell behind under the traditional chalk talk/homework approach, then overtook the class when left to their own devices on my software.

But still, no one believes me, even long enough to let me show them the software. In this case my audience has definite information that what I am saying is impossible: we already tried software -- AlgeBlaster, Algebra Solved!, Cognitive Tutor, MyMathLab, MyMathLabPlus, ALEKS, Khan Academy, Mathspace, LearnBop -- and the Algebra failure rate is unchanged.

So what makes my software different? We will look at the details next, but the bottom line is that it was designed for real, struggling, unhappy, unenthusiastic students with shaky basic skills -- for the kids I worked with as a math teacher and private Algebra tutor.

Let's break that down by looking at my personal check list when assessing competitors:

Does the software check intermediate steps on multi-step problems?

I like to say that students do not do problems, they do steps of problems. Even if they do them in their head, they solve 3x-2 = 13 in two or more steps. Likewise, they do not get stuck on problems, they get stuck on steps of problems. Or make careless errors or reveal weak prerequisite skill on steps of problems.

If software is not checking steps, it cannot deeply engage the student. Sure, having the answer checked immediately instead of the next day is a big win, but it is not enough to create an immersive environment in which students are free to interact with Algebra the way a beginning programmer can interact with certain computer languages.

The ed theory buzzword for this is "active learning". With highly granular (step-by-step) feedback and hints and even videos, we enter a zone where even an indifferent student can apply native intelligence to sorting out the Algebra beast. They can try this, try that -- Algebra is talking to them saying "Fine" or "Not so much" or "Watch me", in detail.

Checking intermediate steps is hard because kids can type anything, and correct work can come in many forms -- the only way to do it is to create an artifical intellignece expert system that can parse and understand Algebra -- so most software checks only answers, explaining why come up short in research. 

I was a private tutor, and worked problems in the classroom with the students. Step by step. I knew that students can have trouble with any step, but most often the first step is where the new topic had to be applied.

Looking back at 3x-2=13, that would come after students could handle 3x=15 and x-2 = 13. So the first step presents the new challenge: which of the two familiar balancing transformations do I apply first?

Where is the vast majority of Algebra tutorial software while the student is contemplating that new concern? Nowhere to be found: "Come see me when you have the answer." followed by "Sorry, wrong."

Note that this is not so bad for the strong student, who then gamely rechecks all their steps looking for their mistake and usually finds it without much trouble. Great, but those are not the students failing Algebra.

Does the student generate answers, or is it multiple-choice?

Multiple choice just does not work. My French teacher liked to say (in French) that there is all the difference in the world between recognition and recall. She said this after we all came up empty on a bit of vocabulary but as soon as she started the word we all finished it with her.

Think about it. Multiple choice always provides the answer! It tries to challenge us by providing three or four other choices, but it still provides the answer. As for the several other choices, well, students have twenty ways to go wrong in Algebra. Providing just three silently corrects seventeen of those mistakes.

By contrast, let students enter their own answers and ironically we help them learn by making success harder. That blank piece of paper, be it of pixels or tree, forces them to generate the mathematics. Yes, help is available, but there again they must do the heavy lifting of asking for and analyzing the help.

The current buzzword for this is "active learning", but I prefer Confucius's "I do and I understand".

Whatever the term, multiple-choice makes this impossible.

OK, the student supplies all work. How? By typing (x^2 -1)/(x+1)?

My target students are strugglers. They will not get far trying to type or read stuff like
(x^2 -1)/(x+1), known as "ASCII math". 

And if the software can render maths nicely (as would TeX), does it include a slick maths editor? Or does it have a graphical keypad with icons for different operations forcing strugglers to build equations click by click? 

A slick editor is hard, btw, which is one reason software does multiple choice.

Fail on either of these and the struggle with composing or reading math will prevent immersion in the maths.

Algebra and Tom Sawyer's Fence

[Thanks to PBS for hosting this excerpt.]

We educators -- especially those math educators who more and more are apologizing for math and even suggesting we should not teach Algebra -- could learn something from Tom Sawyer. So could Mark Twain. Twain wrote the unforgettable passage in which Tom Sawyer coaxed other boys into paying him to do his fence-painting chore, then drew from it the wrong lesson on human motivation:

In order to make a man or a boy covet a thing, it is only necessary to make the thing difficult to attain.

True, the story begins with Tom trying to buy Jim's assistance to no avail. Later Tom refuses to let Ben paint the fence, and later still he has the whole neighborhood lined up to paint the fence, paying him for the privilege. So what makes me think Twain got it wrong?

Twain missed a critical link from his own tale, one germane to the importance of doubling down on Algebra even as we struggle to find a way for kids to master it: why did they want to paint the fence? Because Tom said, No? Um, no. Look closely at the seminal exchange and see if you can spot the true motivational engine at work. Beginning with Tom:

“What do you call work?” 

“Why, ain’t that work?” 

Tom resumed his whitewashing, and answered carelessly:

“Well, maybe it is, and maybe it ain’t. All I know, is, it suits Tom Sawyer.” 

“Oh come, now, you don’t mean to let on that you like it?” 

The brush continued to move. 

“Like it? Well, I don’t see why I oughtn’t to like it. Does a boy get a chance to whitewash a fence every day?” 

That put the thing in a new light. Ben stopped nibbling his apple. Tom swept his brush daintily back and forth – stepped back to note the effect – added a touch here and there – criticised the effect again – Ben watching every move and getting more and more interested, more and more absorbed. 

Presently he said:

“Say, Tom, let me whitewash a little.”

Tom has not said, No, but the bait has been taken. Now Tom will set the hook (and drive monetization) by saying No, but his success hung on first highlighting the intrinsic reward of work done well. Here is the key bit again:

Tom swept his brush daintily back and forth – stepped back to note the effect – added a touch here and there – criticised the effect again  – Ben watching every move and getting more and more interested, more and more absorbed. 

Twain's cynicism, I wager, would be lost on anyone in the painting craft.

"Of course," they would think. "There are a hundred details to get right, details one knows only after painting a thousand fences."

And Mark does seem to know that:

You see, Aunt Polly’s awful particular about this fence – right here on the street, you know – but if it was the back fence I wouldn’t mind and she wouldn’t. Yes, she’s awful particular about this fence; it’s got to be done very careful; I reckon there ain’t one boy in a thousand, maybe two thousand, that can do it the way it’s got to be done.

Thus Tom is saying No, but it is a reluctant No driven by concern for performance: this work is hard and it matters and I would love to say Yes but this has to be done right.

Tom did not turn fence-painting into a game, nor did he make fence-painting somehow relevant to the boys' larger lives. He did not reward them -- he charged them! -- and he did not say they would need fence-painting skills later in life. He simply made it challenging.

We all take pride in a job well done and thus are drawn to challenges at which we might succeed. Look at video games, the single-player kind. The worst thing you can say about a game is that it is too easy to beat. Graphics, music, sound effects, and all the gamification in the world cannot save a game that is too easy to beat.

Why is that? Because the key lure of single-player games is the endorphin rush of success after substantial struggle, and the more struggle the bigger the rush. (Enhancing learning retention, research shows..) There is nothing like mastering something hard, nothing like performing well, meeting an unbending standard. When I complete a level it is because I can perform at that level: one does not pass the Professional Driver's License exam on Gran Turismo by luck.

Tom understood how to make hard work attractive: magnify for your audience the satisfaction of excellence. But US educators today are apologizing for Algebra, the doorstep to one of our greatest cultural wins, mathematics. We strive to make this purest of sciences relevant, as if it were not. We create games that teach mathematics and from that kids learn that we do not think math is worth learning in its own right. We even argue that one should be allowed a college degree without passing Algebra.

That is a losing game. If we want kids to stop showing up for college without fundamental skills in the three Rs, we will need their help: their hard work, their satisfaction in mastering skills that do not need to justify their worth.

We are apologizing for learning being hard. Tom knew that that is half of learning's appeal.

What Made HomeworkTutor So Good?

The Background

The Tilton's Algebra web site (TA) is a reincarnation of a desktop application sold for Macs and PCs back in the 80s and 90s.

That application was called Algebra I HomeworkTutor. I am a programmer, not a marketer.

It was reasonably successful but I am no businessman, either. I tried to do it all without raising money, and this software is challenging (think years of development, not weeks). Periodically I went to work on other great projects which lasted years before I could hunker down for another push.

The most recent push has lasted fifteen months. (This app is intense.)

The Question

While working on the new version, several times former users of HwT tracked me down (with no little effort) to find out if the software was still available, or in one case to ask me to give a talk on the product. The message was consistent: there was a lot of Algebra software out there but nothing like HwT.

I was asked recently to document as best I could why HwT was so powerful, even though it had none of the new features of TA such as embedded video, an on-line forum, and a "levelling up" process to ensure mastery as well as draw students in.

The Answer

Here is what I have learned, for what it is worth, from anecdotal feedback, occasional published research, and a small amount of experience myself observing students using my software and other tutoring systems. In order of increasing importance, here is my understanding of why HwT succeeded in the 90s to the extent it did and why there is still nothing like it in 2015.

Step-by-step error detection and forced correction of mistakes. 

With HwT, students entered each step of their solution, not just the answer, then checked their work for correctness. When correct, strugglers got encouragement if they were in doubt, greatly reducing anxiety.

If their work was wrong, it just said so, and students could not proceed to the next step without correcting the errant one. This forced correction was transformative and universally popular with teachers. Without this, struggling, discouraged students just plowed through worksheets making mistakes interested only in producing something to turn in so they did not get a zero, the ball now in the teacher's court to correct all the papers and try to reteach the material.

Student engagement.

When mistakes were made, it was up to the student to fix the mistake. Hints and solutions of examples were available, but they had to ask for them and understand them. Importantly, the hints were just that; with most software so-called "hints" actually tell the student the next step.

Furthermore, HwT simply waited until they fixed their mistake, allowing unlimited retries.
Most software stops the process after a certain number of failed efforts, presumably to avoid discouraging the student. HwT trusted the learner to decide for themselves when to ask the teacher for help, and anyway, research shows the struggle is important to the permanence of learning once the student has their "Aha!" moment.

Limited help--but not too little.

The software does offer subtle hints and solved examples for students to learn from, but these require the student to dig into their memory of the teacher's presentation for any benefit to be had. The student struggle is guided, but in the end they have to assemble the solution. I have recently seen research on the value of frequent quizzes: apparently the act of pulling content from memory strengthens the command of that content.

Student control.

I learned this one from the first student I observed using my software on-site at a school. She had the difficulty level set to "Easy" and was doing problem after problem successfully. I suggested she try an "Average" problem and she nicely let me know that she would do a few more easy ones before advancing.

I realized her comfort level was simply different than mine. For the many students traumatized by math it seems better for them to practice "until they themselves knew they were proficient", as one teacher put it.

There is much talk these days of data mining and adaptive learning and software customizing the learning experience automatically. I would be curious if the implicit loss of student agency reduces engagement, effort, and finally results.

Quantity of practice. 

While "time on task" (ToT) is being challenged as necessarily correlated with greater learning (Kohn being a good example), even Kohn acknowledges that if the student engagement is there then the learning does follow.

I am speculating here, but I suspect students using HwT did more problems, achieving a greater quantity of practice as well as the quality discussed above. For one thing, the problems just pop up ready to be worked -- no copying them out of the book onto paper.

And now with TA and its so-called "missions" (summative tests) generated at random, students can do problem after problem trying to pass a mission, much as they play video games for days trying to get past a tough level. The old line about "I do and I understand" is as true as it is old, so I think increased time "on task" is vital.

Speaking of which, the DragonBox Algebra people reported some fascinating numbers from their "challenges" in which thousands of students did tens of thousands of problems. They said something like 93% of the students completed the game, but some took ten times as long and did six times as many problems as the fastest.

Of course for that we need software generating problems and evaluating student work or the load on teachers would be untenable.

Is the anxiety eliminated?

I was not a math major in college so when I decided to move from elementary education to math I had to take a few courses at a local college. One course was differential equations, and I distinctly remember being half-way down a page on one problem doing calculus as a shaky prerequisite skill and feeling terribly uneasy about the whole thing.

It turned out in each case that I was doing fine, but there it was: without any feedback on each step, and with a lack of confidence in my calculus, I experienced math anxiety  for myself.

So I am curious how students will react to step-by-step correction: is knowing they have made a mistake OK, as long as they know? Or will they still report anxiety? Also, how much does it help to have the software say (in training mode) "OK so far."? As a private tutor many a time my clients did a step correctly but then looked at me in doubt. That was the anxiety I experienced doing differential equations.

I think some students will still be upset when they get mistakes, so the cure may not be perfect, but I will be curious to see if the reports are of anxiety or frustration. My hope is that getting the anxiety out of the way will draw out more perseverance and ameliorate even the frustration.

Two Sigma Problem?

Bloom (1984) identified a so-called Two Sigma Problem: how do we come up with an instructional method as effective as a combination of mastery-based progress and good individual tutoring, without hiring a tutor for every math student?

I have mentioned that I have done scant observation of my software in the field. This lack of field-testing was possible because I came at the design from the other direction and simply did my best to recreate in software the experience I provided as a private tutor.

One teacher reported that she put stragglers who had fallen off the pace on HwT and after a number of weeks they actually caught up with and rejoined the mainstream. Large-scale tests of Cognitive Tutor and Khan Academy have failed to demonstrate much benefit at all, but HwT helped failing students catch up with and rejoin the mainstream.

It would be interesting to see if students report any sense of being tutored privately by the software.

Good tutoring

Van Lehn (2011) did not find the two sigma effect Bloom et al reported. Instead, he found effects of 0.76 for software tutors and 0.79 for human tutors. But Bloom explicitly claimed to have used "good human tutors", while Van Lehn documented a wide variation in the quality of the human tutors. For example, often the one-on-one sessions involved very little student contribution -- the tutoring was more of a one-on-one lecture.

My style even when lecturing is to constantly call on students for contributions to the work I am doing at the board, and my style as a tutor was to help students when they got stuck by asking them leading questions they actually had to answer and then if all went well realize how they could get unstuck.

While a good human tutor will always be better than a good software tutor, embodying quality tutoring pedagogy in software makes it more reliable and more widely available. Over time it can be enhanced in the light of experience, capturing even better tutoring in the software asset.

Gaming, good and bad.

On one rare on-site visit I saw students also using Where In the World is Carmen Sandiego? Students had learned that if they asked for enough hints the software would just tell them the answer, so they did that until they ran out of hints. Then they would call out to the teacher to tell them, so they could build back up their reservoir.

One study I saw on Cognitive Tutor said students did not ask the software for help because CT deducted points for hints. Instead, students asked the teacher for help (who, again, provided it). CT  also provides answers after three mistakes, while HwT just lets them flail.

One factor we see is that a teacher can inadvertently undercut the software. That happened as well in the infamous "Benny Incident" where the teacher neglected to keep an eye on a student's fascinating internal dialogue with a tutoring system. (Shaugnessy, 1973.) The student was indeed bright, the questions were often multiple-choice, and the passing threshold was only 80% so the student was able sail through the automated instruction despite grievous deficits in understanding.

So that is the bad sense of gaming. On the good side, new for TA is a so-called "Mission Mode". Here there is no multiple-choice, the standard of success is higher than Benny encountered, the help non-existent, and the tolerance for error nil. But students can try a mission as often as they like, quit a mission going badly any time they like, and even when they fail they get a "Personal Best" certificate if they get that far. So like video game missions, the standard is unbending but the failures are simply stepping stones to eventual success, with progress drawing the student into more and more practice.

Mastery-based.

Last but (not as advertised) not least: a lot of the above boils down to students proceeding at their own pace in what is commonly referred to as the mastery-based model. Bloom felt the mastery model was more important than the private tutoring, perhaps precisely because of the variability of tutor quality documented by Van Lehn. As the DragonBox Algebra results show, over 90% of students could master the game given enough time.

This in turn aligns with what most math educators believe: Algebra is not that hard. I would be interested in whether students who struggled with Algebra before using the software reported a change in attitude in which they changed their assessment of the difficulty of Algebra.

Summary

There is a lot of powerful Algebra software out there, but Algebra failure rates are as bad as ever. Two-year colleges are forced to offer multiple courses even in arithmetic, and the AMATYC has just recommended dropping the Algebra requirement for non-STEM majors, a step already taken by the California college system and elsewhere. So why is all that Algebra software not working, when HwT did back in the 90s?

One easy reason: the only other software I know of that checks intermediate steps is MathSpace. Without step-by-step feedback, most of the wins delineated above disappear.

Other than that we have a good research question: which other elements of HwT were so instrumental in its success? Above are my guesses as to where lies the answer, but it is all anecdotal and seat-of-the-pants. More experience and data on how exactly students and teachers use the software is needed.

I suspect we will find the following:

• Students report less anxiety.
• Different students will use different kinds of help (trial and error, video, canned hints, solved examples, and the public forum).
• Students will do many more problems.
• Student performance will be better and more uniform, but with a normal distribution of how much practice is needed to achieve that performance.
• Students will enjoy math in and of itself, as a puzzle.
• Comparison with adaptive tools will show student agency is more important than precise, automatic throttling of content.