Getting the evidence right: What quality looks like for AI in education

What counts as quality evidence when the development of artificial intelligence moves faster than the trials can measure it? Discover why rigorous studies alone aren't enough and what the sector needs to do to solve this.

|
5 minutes read
A somewhat commonly-used artistic trope that can be used to illustrate both pro-AI and anti-AI sentiment. Credit: Zach M/Unsplash

A somewhat commonly-used artistic trope that can be used to illustrate both pro-AI and anti-AI sentiment.

Artificial intelligence (AI) has emerged as a promising opportunity to accelerate foundational learning, particularly for learners in low- and middle-income countries, but the evidence base on AI in education and shared understanding of what good evidence looks like remains thin.

Governments and funders are being asked to decide which digital tools to back with little more to go on than promises, product demonstrations and anecdotal reports of success.

In the introductory blog of this series, the Global Coalition for Foundational Learning set out three tactical priorities for accelerating foundational learning with EdTech and AI: strengthening the evidence base, investing in digital public goods and building a more coordinated ecosystem.

This blog takes up the first, strengthening the evidence base, and starts with a question that sounds simple but is not: what counts as robust evidence when we talk about AI in education?

The type of evidence we need

The global education sector largely agrees that the most rigorous method to generate evidence is a randomized controlled trial (RCT).

RCTs provide a strong research design for measuring causal impact, but they are costly and are best placed once an EdTech tool and its implementation design have sufficiently matured with the basics checked: the soundness of the pedagogy, the usability of and engagement with the tool by teachers and students, and whether it is safe and age-appropriate for the children it is meant to help.

An RCT on an untested product is premature. Even if methodologically robust, it cannot tell us what we need to know: whether the tool failed or whether it was never ready to test.

Robust evidence for an AI tool in education is contextually relevant, generated at every stage of the tool’s development and kept current.

The evidence base is built by evaluating a tool throughout its development lifecycle, using what each stage reveals to improve it and decide whether it’s ready to advance further for use in practice. This requires a sector to be clear and agree about what good evaluation looks like at each step of the development lifecycle and not just at the point of measuring impact that comes much later.

For the use of AI tools in education, a useful way to structure the tool evaluation process is the four-level evaluation framework developed by the Center for Global Development and the Agency Fund for AI-based development interventions that considers:

  • Model evaluation: does the AI model behave as intended?
  • Product evaluation: is the tool being used as designed?
  • User evaluation: does using the tool change what learners know and do?
  • Impact evaluation: does the tool improve learning outcomes, equitably and cost-effectively?

The framework was designed for the development sector overall, but partners are now working on an education-specific version that addresses foundational learning needs.

Within the existing framework, RCTs sit at the impact level.

The earlier levels call for different evaluation methods, each with its own standard of rigor, that also matter because positive performance at one level does not guarantee quality performance at another. For example, a model can pass a pedagogy benchmark (the level of model evaluation) and still confuse a young learner (the level of user evaluation).

Evaluation across the lifecycle can also detect risks that traditional quality assurance may miss including algorithmic bias, unintended use, widening inequalities, data privacy and security and effects on children’s confidence, effort and independence. Robust testing before deployment and continuous monitoring after it is how these issues are caught.

Still, evidence for an AI tool in education that is robust but not meaningfully interpreted leaves a decision maker where they started: with a number, but not a decision.

Consider the World Bank’s pilot of AI tutoring in Nigeria. The headline result was striking: learning gains equivalent to nearly 2 years of schooling in just 6 weeks. But the analysis showed those gains were unevenly distributed.

Students with stronger, prior academic achievement benefited more as did students from higher-income households compared to many lower-income students who were encountering computers for the first time.

A trial of Google DeepMind’s Gemini Guided Learning tutoring tool in Sierra Leone showed the same pattern of uneven effects determined by students’ prior academic achievement.

A positive average effect can mask outcomes that matter for the very decisions the evidence is meant to inform: whether to procure a tool, where to deploy it and what to pair it with as part of a strategy for teaching and learning.

Tool evaluation at the speed of AI

Evidence generated once and filed away quickly becomes outdated. AI does not hold still: models are updated, swapped and replaced on a timescale of months.

The largest AI labs often test updates extensively before release, but most EdTech developers building on top of those models don’t have the same capacity. This means any change to the underlying model can alter how an AI tool performs in a classroom without being detected.

Keeping pace with AI requires the sector to come together to develop cheaper, faster yet robust evaluation methods from earlier signals on whether a tool is working and single-session checks of whether a child is actually learning or just going through the motions to rapid trials that produce impact evidence in months rather than years.

The recent RCT of Google DeepMind’s tutoring tool in Sierra Leone is a move in the right direction: significant learning gains (+0.26 standard deviations) under real classroom conditions in just 8 weeks.

Building global public goods

The sector needs shared infrastructure to develop quality AI tools in education including: AI benchmarks, standardized evaluation methods and study designs, common datasets and published playbooks—concrete, reusable tools that any developer, funder or government can use.

The components are starting to emerge across the Coalition for Foundational Learning, supported by multiple Coalition funders: Fab AI’s AI model benchmarks for education (with the AI Observatory) to the World Bank’s evaluation work, UNICEF’s EdTech for Good Framework 2.0 and the Learning Cabinet and the Quality Assurance Facility for AI in Education’s education use-case benchmarks and field evaluations.

AI’s potential to accelerate foundational learning is real, but realizing it depends on evidence that keeps pace with the technology, is generated at every stage and is interpreted carefully enough to inform the decisions it is meant to support.

We invite funders, governments, developers and evaluators to build appropriate evaluation into procurement and grant making so that what works can be scaled responsibly and that evidence is treated not as a final hurdle to clear, but the foundation on which everything else is built.

#####

Read other blogs in this series

Related blogs

Leave a comment

Your email address will not be published. All fields are required.

The content of this field is kept private and will not be shown publicly.

Comments

  • No HTML tags allowed.
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.