Sharing Frank Feedback

In two reports to be published this week, Macmillan Learning offers a surprisingly in-depth look at how students and faculty responded to the first iterations of its new digital learning platform, Achieve.

The first report, previewed by Inside Higher Ed, shares how instructors perceived Achieve during early testing and describes the product development process in detail. The second report focuses on the student learning outcomes of a spring 2019 pilot that involved 40 instructors teaching courses in biology, calculus, chemistry, composition or economics using Achieve. The product is slated to launch commercially in fall 2020, with more subject areas to follow.

While instructors’ impressions of Achieve were generally positive during the spring semester -- the first of three semesters of beta testing -- some statistics clearly showed room for improvement. Asked how likely they were to adopt Achieve once their participation in the pilot had ended, instructors gave a moderate response of 6.6 on a scale of 1 (“will definitely not adopt”) to 10 (“will definitely adopt”).

Though all instructors said Achieve improved student collaboration, comprehension and preparedness for class, just 8 percent agreed or strongly agreed that Achieve supported mastery of their subject more than if the product had not been used. Achieve incorporates existing Macmillan textbook content into a digital platform with additional features designed to improve the student experience and assist instructors.

Alastair Adam, co-CEO of low-cost textbook publisher FlatWorld, said it was “really brave” of Macmillan to publish the results of its beta testing. While mostly encouraging, the data are not the staggering statistics you typically see in publisher marketing, he said.

Most digital learning platforms look pretty similar, said Adam. On top of curricular content, they offer adaptive learning technology, auto-graded assessments and student performance tracking. “It’s really hard to move the needle in this space,” he said. “This is a game of inches, not revolution.”

Fiona Hollands, associate director of the Center for Benefit-Cost Studies of Education at Columbia University’s Teachers College, welcomed Macmillan’s transparency. “It’s great they were willing to put themselves out there and share these data this openly,” she said. Hollands has previously published research that suggests many higher education institutions choose to adopt classroom technology on the basis of “less than rigorous” evidence.

Macmillan’s reports look like they were written by “serious researchers,” said Hollands, though she questioned whether instructors considering Macmillan’s products will really need the level of detail included in the reports. “I’m not sure who these reports are intended for,” she said.

Macmillan’s research would have been more credible if it had been conducted by a third party, said Hollands. But the inclusion of some modest results gave her more confidence the research is legitimate, she said.

Kara McWilliams, vice president of impact research for Macmillan, emphasized the company’s commitment to ethical, peer-reviewed research. Co-designing a digital learning platform with faculty and student input was, at times, challenging, she said. Incorporating weekly faculty feedback into the product took time and required close coordination between different teams at Macmillan. And it soon became clear that instructors from different disciplines wanted different things from the platform. But the end result is a product that instructors and students actually want to use, she said.

Involving faculty early in the design of Achieve helped to make its design much more efficient and user-friendly, she said. The team incorporated more student performance metrics into the platform at faculty member’s request. Faculty also asked for more course management flexibility, clearly labeled learning objectives, and a simplified grade book layout. 

While Achieve is still in development, McWilliams is pleased with the reception of faculty and students so far. The most recent data suggest 89 percent of participating instructors intend to keep using Achieve when the trial is over, she said. Students were required to actively consent to participate in the study, with 74 percent, or 2,251 students, choosing to do so. Students were offered the software for the duration of the pilot for free.

Jeanette Wiseman, senior associate at MindWires Consulting, said it is not uncommon for publishers to offer students free copies of their product when the product is in development. Offering a free trial is a good incentive for instructors, as well as students, and helps to drive early adoption, she said.

In addition to offering students a free trial of Achieve, Macmillan paid instructors an honorarium of up to $1,500 per class per semester for their weekly evaluations. Wiseman, Holland and Adam said it is not uncommon, nor inappropriate, to compensate faculty for their time. Hollands does, however, recommend that instructors check with their institutional ethics boards before accepting such payments.

Solomon Namala, economics professor at Cerritos College, a community college in Norwalk, Calif., and Solina Lindahl, an economics professor at California Polytechnic State University, both participated in the beta testing of the Achieve platform. They intend to continue using the product, which will cost students around $99 per semester once the pilot has ended.

Both Namala and Lindahl said that they had been impressed at the way Macmillan worked to incorporate their feedback into the product, though the process of providing weekly evaluation was repetitive and a little more work than they bargained for.

“There are still some improvements to be made, but I would recommend the product,” said Namala.

Lindahl, who has been involved in the testing of numerous digital courseware products, agreed. She said Macmillan's process was thoughtful and their communication was good. “I feel like they really listened,” she said.