Edtech has an evidence problem

Edtech brokers have begun producing new evidence and measurements of the impact of technologies in schools. Photo by Alexander Grey on Unsplash

Schools spend a lot of money on edtech, and most of the time it’s a waste of their limited funds. According to the Edtech Evidence Exchange, educators estimate that “85% of edtech tools are poor fits or poorly implemented”, indicating very weak returns for the $25 billion or more annually spent on edtech in the US alone. The problem is that school procurement of edtech is rarely based on rigorous or independent evidence. The Edtech Evidence Exchange is one example of a new type of organization in education that is aiming to address this problem, by constructing an evidence base to support edtech spending decisions.

In a new paper just published in Research in Education, Carlos Ortegon, Matthias Decuypere and I conceptualize these new edtech evidence intermediary organizations as edtech brokers. Edtech brokers perform roles such as guiding local schools in “evidence-based” procurement, adoption, and pedagogical use of edtech, and have the mission to support teachers and school authorities to modernize in safe, reliable, and cost-effective ways. Edtech brokers are appearing around the world yet they have not, as yet, captured much critical attention. We kicked off our project on edtech brokers a couple of years ago, with Carlos Ortegon taking the lead for his doctoral research and lead-authoring the paper entitled “Mediating educational technologies: Edtech brokering between schools, academia, governance and industry” as the first major output.

Edtech brokers are significant emerging actors in education because they are gaining the authority and capacity to shape the future direction of edtech in schools, at a time of rapid digitalization of the schooling sector in many countries around the world. They can also be powerful catalyzers of the edtech market. As expenditure in edtech from governments, companies, and consumers has increased in the past decade and as the edtech industry continues to seek new market opportunities, such as the application of AI, edtech brokers play a role by connecting technical products to the specific social and political contingencies of different local settings.

Edtech brokers

In the paper we identify three distinctive kinds of edtech brokers:

Edtech ambassador brokers, which act as representatives (or ambassadors) of specific edtech brands. Edtech embassador brokers encourage the procurement of their products and promote their educational potential. Ambassador brokers are a global phenomenon, as the growing number of Google and Microsoft specialized organization partners across different countries makes clear, and they usually offer services such as streamlined procurement and professional development for teachers.

Edtech search engine brokers operate as search portals that focus on providing on-demand evidence about “what works” in edtech, thereby shaping procurement and usage from a wide range of market providers. They place strong emphasis on providing “bias-free advice” and “evidence-based recommendations” that can prevent problems of over-expenditure as the Edtech Evidence Exchange puts it. Edtech search engine brokers often combine multi-sector mixtures of academic, industry, policy, and philanthropic expertise, though some are commercial companies and others directly goverment-funded.

Edtech data brokers support schools in managing, regulating, and analyzing their digital data. Edtech data brokers are gatekeepers of the data produced by schools when using edtech, whose core activity is securing data flows between schools and vendors. Data brokers offer distinct tools for schools to analyze their data, facilitating school-level educational decisions. 

Though they are relatively unknown in the digital education landscape, edtech brokers are therefore becoming important figures that make claims to expertise in edtech effectiveness, filter purchasing options, shape edtech procurement decisions, manage data flows, and lead the professional development of teachers in schools.

Beyond this seemingly straightforward definition of their role, we also see edtech brokers as strategically mediating between schools, industry, evidence and policy settings. In this mediating role, this means edtech brokers construct relations between a variety of different constituents. For example, they connect vendors to schools, act as relays of evidence produced in research centres, and they strengthen policy agendas on evidence-based edtech. They also act as transmitters and brokers of normative ideas about tech-enabled transformation and reform, assisting the circulation of powerful imaginaries and expectations of educational futures into the attention of school decision makers. One initiative even brokers relations between startups, learning scientists and investors for evidence-based edtech financing.

But this means edtech brokers also have some capacity to affect each of the constituents they connect. First and foremost, edtech brokers take up powerful positions in determining which and how edtech is used in schools, according to particular standards of evidence. This means, second, that edtech brokers can influence edtech markets, shaping the financial prospects of startups and incumbents, as they either promote or devalue specific products, and thus affect the procurement decisions of schools. And third, they can influence policy settings and priorities, by positioning themselves as arbiters of “what works” and thus amplifying policy attention on certain affordances and functionalities.

Mediating edtech

In the paper we highlight the mediating practices of edtech brokers and their implications. The first set of mediating practices we refer to as infrastructure building. In their documents and promotional discourse, edtech brokers frequently invoke the idea of school modernization, and of using evidence-based edtech to update and upgrade schools’ digital infrastructures for teaching and learning. In the case of ambassador brokers, this updating of digital infrastructure also involves synchronizing schools and teachers’ pedagogic practices with the broader digital ecosystems of big companies like Google and Microsoft. Edtech data brokers emphasize interoperability and the synchronization of student data flows across different edtech applications. More than merely offering technical products and support, these efforts shape the digital architecture of the school through the promotion of rapid, easy, and safe processes of transformation.

The second key brokering practice is evidence making. Edtech brokers use different evidentiary mechanisms and instruments to produce evidence of “impact” and “efficacy”. By doing so, edtech search engine brokers in particular guide the adoption and usage of edtech in schools, ultimately mediating and shaping the production of “what works” evidence and its circulation into school decision-making sites. One edtech search engine broker studied in the paper, for example, operates as a kind of database of edtech products that are ranked and promoted in terms of online reviews provided by teachers. The broker calls this kind of evidence “social proof”, with its legitimacy derived from front-line teachers’ active participation in its production, though it is also shaped and constrained by a series of specific criteria the organization has derived for “assessing impact”.

Another search broker, by contrast, rates edtech according to specific variables and measurement instruments, enabling schools to define their needs and receive contextualized recommendations through a “matching” program. As such, edtech brokers reinforce the political ideal that “what works” can be repeated in diverse settings, by incorporating educators themselves into the evidence-making process and by producing locally contextualized guidance via new instruments. Edtech brokers’ evidence is not neutral but imprinted by specific assumptions and interests.

The final practice of brokers is professionality shaping through professional development and training programs. By mediating between edtech vendors and pedagogic practice, brokers aim to transform teachers into knowledgeable edtech users, while simultaneously extending edtech vendors’ reach into everyday professional routines. Edtech brokers therefore project a particular normative image of the digitally-competent teacher who, armed with evidence and training, can capably choose the right edtech for the job at hand and deploy it to beneficial effect in the classroom.

Examining edtech brokers

The article is now the basis for ongoing empirical work with edtech brokers across Europe. They are mediating edtech into schools, and while doing so laying claim to expertise in edtech evidence and practice. This makes them significantly powerful yet little-studied actors in shaping how and which digital technology is promoted to schools, how schools make procurement decisions, and how teachers incorporate edtech into their routine pedagogic practices.

In turn, these brokering practices open up important questions about the nature and production of evidence about edtech impact, about the role of little-known intermediary organizations in shaping the future of edtech use in classrooms, the interests, assumptions and financial and industrial support underpinning their judgements, and their capacity to affect the market prospects of edtech startups. Edtech brokers may be putting efforts into solving the evidence problem in edtech, but by doing so they are also positioning themselves as powerful influences on the digital future of schooling.

The full paper, “Mediating educational technologies: Edtech brokering between schools, academia, governance and industry”, is available (paywalled) from Research in Education, or as an open access version.

This entry was posted in Uncategorized and tagged , , , , , , , , , . Bookmark the permalink.