Is There a Better Way to Test Math?

Casey Hawthorne, assistant professor of mathematics education

Olivia McGuirt ’21 estimates she watched every clip more than 100 times.

One hundred times, noting the words the kids used.

One hundred times, watching them write.

One hundred times, evaluating their confidence or confusion.

“It wasn’t just about what they were saying,” McGuirt says. “It was: ‘Show me your work.’”

McGuirt was one of four students who spent the summer of 2020 researching with Casey Hawthorne, an assistant professor of math education at Furman. Their project was designed to analyze what standardized tests are assessing, specifically by looking at how students think about math problems involving fractions and diagrams.

“Kids love – humans love – to talk about their thinking,” says Hawthorne, who conducted video interviews with the elementary school students.

But math classes traditionally have been focused on answers, not thought processes. Hawthorne’s experience suggests many teachers feel they have to devote the bulk of classroom time to methods for calculating those answers, with little time exploring the different ways students might think about the material.

“Once you do that, it just profoundly changes how you view education,” he says.

“You have to stop and really listen to what the student is saying,” McGuirt says.

The Furman Advantage includes funding for undergraduate research. As COVID-19 began to shut down the country in Spring 2020, Hawthorne knew he’d need to devise research that could be conducted remotely.

The idea for this project grew out of one of his math education classes. Hawthorne had changed the course to add a weekly teaching experience in a local elementary school. The change seemed to create a new level of interest for his Furman students in how the younger students were thinking about math. “That’s the first time students have wanted to do research out of that class,” Hawthorne says.

“It was our first experience really getting to play teacher,” says Katie Taylor ’21, now a fifth-grade teacher at Roebuck Elementary School in Spartanburg, South Carolina.

The student researchers were all education majors, people who didn’t identify as mathematicians. In fact, McGuirt, an elementary education major, had been nervous about the class because she didn’t consider math “her thing.”

Hawthorne “showed us a lot of cool ways to help kids better understand math,” she says. And the methods helped her, too.

“I just fell in love with the way kids were thinking about numbers,” McGuirt says. “I found that to be fascinating.”

Olivia McGuirt ’21 gives a TEDxFurmanU talk titled, “Children’s Mathematics: Creating a New Generation of Learners.”

The research

Standardized assessments are ubiquitous in schools throughout the United States and are used for everything from ranking schools and assessing teachers to measuring individual student progress over a year.

With the help of a South Carolina school district, Hawthorne identified 20 fifth-grade students from different classrooms and with a range of abilities and had them take a 10-question math test. The questions were taken directly from a variety of standardized tests, including those the district uses.

Once the initial results were in hand, the researchers narrowed their inquiry to four problems, all of which asked students to use models or diagrams to answer questions involving fractions. The diagrams either represented the answer to a question or both the process and the answer.

Hawthorne met with each elementary school student for an hour over Zoom and asked them to talk through their work. He had their answers in front of him; he wanted to know how they got to those answers.

Some never even looked at the diagram. Others skimmed the question and then ignored it while trying to connect the diagram to one of the multiple-choice answers.

“Some kids would really focus on the drawing of the fraction, some would focus on the wording of the question,” Taylor says.

Some demonstrated as they spoke that they understood the math but had trouble identifying what the question was asking them.

“Now, being a teacher, I see that all the time in my class,” Taylor says. “I just keep this research in the back of my mind.”

Most students were choosing answers, even correct answers, for superficial reasons, Hawthorne says. Others picked wrong answers while giving explanations that revealed a solid grasp of fractions.

“It just boils down to, there’s no one right or wrong way to solve things,” Taylor says.

The team used a coding process to identify ways students were thinking about a problem or expressing their ideas. They looked for patterns and themes within one problem and then across multiple problems.

“We spent so much time just trying to make sense of students’ thinking,” Hawthorne says.

The results: a disconnect between students’ scores and their understanding.

Hawthorne wasn’t surprised by the big picture on the tests themselves: “These things are pretty bad at measuring students’ understanding of fractions,” he says. On the flip side, the questions worked as “fantastic” tools to draw out students’ thinking when he talked with them and probed their understanding of the material – what educators call “formative assessment.”

“But as a summative assessment, they were just really miserable,” Hawthorne says. The assumption behind standardized testing is that the more thoroughly students understand the material, the better score they’ll get. That’s not what this research revealed.

“There was limited alignment between understanding and scores,” Hawthorne says. “It really makes you question.”

Part of the issue is that many questions aren’t written in the way a teacher would ask them in a classroom, Taylor says. So “the score doesn’t reflect fully what they know,” she says. “It’s something to remember when you do get test scores back that aren’t what you hoped for.”

When the research was completed, Taylor and McGuirt worked with Hawthorne on a paper published by the Research Council on Mathematics Learning. Charles Michael ’21 and Mikayla Miley ’20 worked with him on a piece published in the Ohio Journal of School Mathematics.

Charles Michael ’21, a fourth-grade teacher at Tigerville Elementary School

The real world

The chance to do research was valuable for the Furman students personally, as some immediately began pursuing master’s degrees during their first year of teaching.

“It’s been a pretty easy transition because we did get this experience,” Taylor says.

McGuirt gained confidence about applying data in her classroom.

“I feel like I have a better understanding of how to use research to guide my teaching,” she says. And that means the work is having an impact on dozens of elementary school students across South Carolina’s Upstate, too. All those hours of listening to a handful of kids on video shaped the way McGuirt listens now.

“I really learned to listen to understand and not listen to respond,” she says.

Michael, who teaches fourth grade at Tigerville Elementary, says the summer research has helped him anticipate his students’ answers in the classroom. “It’s pretty important for me to understand what students might say,” he says.

Hawthorne’s long-term vision is to continue this study and get the results into the hands of many more classroom teachers.

“I don’t have the answers to these tests,” he says, “but I would like to bring this information to the teachers and administrators who use it and start conversations.”