Right now, society is worried about the impact AI will have on education. Educators are seeing students offload their thinking to AI, employers are concerned about the readiness of graduates who used AI during their studies, and higher education institutions are having to rethink their teaching processes to accommodate the inevitability of AI being used by students (and being criticized for it!).
I see education as AI’s greatest near-term opportunity for positive impact on society. Unlike other AI big bets (such as curing all disease), there are problems in education that current models and tooling can solve or improve now. This opportunity is particularly huge for positive impact because of the economic and civic changes that improvements to education can bring.
While we know AI can negatively impact a student’s learning (while making them feel like they’re learning more!), we also know that AI tutors can be tuned for a teaching task to mitigate that negative impact and potentially improve the learning experience overall.
Additionally, much of the current AI use for learning seems to forget that society has done a lot of important research into the education and learning processes themselves to understand not just what needs to be taught, but also how best to teach it. By remembering this research, we can more appropriately adapt AI use to benefit education.
In this article
How AI is affecting learning
Why education is such a massive opportunity for AI
Prompting versus thinking: Where AI is failing students
What research into education has already shown to improve the learning process
Why building effective AI learning tools is such an engineering challenge
Fenzo has joined the AI Dev Pass!
As part of a way to bring high-value partnerships to you, I’m putting together the AI Dev Pass. This is a collection of exclusive perks and offers for paid subscribers to make AI learning, development, and tooling more accessible to you. You can find the current offers here. Readers can read more about partnerships here.
I’m happy to announce that Fenzo has joined the AI Dev Pass and is offering 10% off of its annual Pro plan as a perk for AI for Software Engineers subscribers.
Fenzo is Educative’s AI tool to help you learn career-building skills by constructing bespoke courses, books, and articles via AI. I tested Fenzo and was very impressed by its ability to produce explanations that build upon first principles and create interactive elements. I was also impressed when I left some feedback with the development team and they fixed it very quickly.
Give it a try and unlock 10% off here.
How AI is used changes what students learn
An excellent study into math education in a Turkish high school does a great job of explaining the current state of AI in education.
Students were given unrestricted access to GPT-4 when studying for a test (“GPT Base”). Access to this model was then revoked when students took the test. The study found a 48% increase in performance on practice problems with GPT Base but 17% lower performance on tests when GPT Base was no longer available (all compared to students without access to GPT Base throughout).
This is a very clear indicator of what we’ve already expected: Relying on AI to think hurts learning. Interestingly, the students impacted by using GPT Base didn’t perceive a reduction in their learning. They felt it was as effective as previous study methods.
Another group within the study was given access to a GPT variant that had been configured for tutoring through its prompts (“GPT Tutor”). Specifically, it had been given teacher-prepared solutions, common mistakes students make, and instructions on giving hints to aid in the students’ education. Similarly, this group was given GPT Tutor for studying, and it was taken away at test time.
The GPT Tutor group improved their practice scores by 127% over their counterparts studying without AI. However, test scores showed zero significant difference between the GPT Tutor and non-AI groups.
In this study, we didn’t see a direct improvement from using AI, but we did see that tuning the AI to be an education-specific studying tool mitigated the negative impact it had on learning.
Potentially more interestingly, we can see the impact this tuning had on how the students studied. The researchers’ analysis also found that students increasingly asked for help and attempted answers themselves, rather than simply requesting solutions.
This shows the importance of tuning the AI experience for education and the impact that can have on the actual learning experience. Even a marginal impact on the learning experience could be a massive net positive for society.
Why education is so important
A 2018 World Bank report analyzed the impact better education has on individuals, their families, and society as a whole. Its main takeaway was that better education for the populace has benefits across the board. These benefits are summarized in the table below.
The same report also highlights two very important distinctions for providing great education:
Providing education to only a subset of the population “can deepen cleavages between favored and disadvantaged groups.” Improving education is good, but we need to ensure that education is available to all or we face potentially negative societal impact.
Access to education and great learning are two separate things. While we can create an education system available to everyone, the impact of that education system is only as effective as the learning that takes place within it. Whether learners actually gain the skills and knowledge they need is what matters.
Interestingly, a report from UNESCO, the OECD, and the Commonwealth Secretariat projects roughly US$10 trillion in annual global social costs by 2030 from children not gaining basic skills. This estimate includes lost earnings and public costs, rather than just money spent making up for the skills gap.
I’ve always been a huge proponent of improving education to improve other aspects of society. An educated populace tends to help solve other problems society faces and potentially even prevent them from appearing. There’s a huge opportunity for AI to make this happen.
Prompting versus thinking: Where AI is failing students
Back in February 2024, Andrej Karpathy made an important point about learning in a post on X. His point is that a lot of short-form educational content is entertainment masked as education. Viewers feel like they’re learning something, but watching alone doesn’t necessarily involve the work needed to truly learn.
His most prominent point was that for learning, the “primary feeling should be that of effort.” LLMs are excellent at producing answers, but having the answer supplied often means we don’t have to do the work or apply the knowledge to learn something ourselves. That work is the learning process.
As the founder of Fenzo puts it:
“Most AI tools are great at spitting out answers. But getting an answer and actually understanding something are very different things.”
Currently, generative AI tools are marketed and developed to do things for us, generally conveying the message of let AI do that for you. We can see this in current marketing for the most prominent AI tools:
OpenAI’s ChatGPT agent announcement promises to “complete tasks for you.”
Gemini’s scheduled actions offer to handle recurring tasks: “Just tell Gemini what you need and when, and it will take care of the rest.”
Notion promotes agents as a “24/7 AI team”, doing things whether a person is online or not.
These applications aren’t necessarily wrong (in fact, many are genuinely useful), but it’s no wonder that students use AI tools this way when this is the messaging they receive. While this strategy might work for things “that just need to be done,” it doesn’t work for things where the act of doing is the takeaway, such as learning.
So how do we ensure our AI educational tools provide the right amount of effort to induce learning?
What education needs to be effective
When it comes to applying AI to education, the how is more important than ever. It’s too easy to take the path of least resistance instead of the path of greatest effect. Below are the items I’ve found to be critical in learning to produce the effort required for substantial, long-term benefit.
Structure
The structure of learning a topic is perhaps the most underrated aspect of grasping a given topic’s fundamentals. One of the reasons my AI for Software Engineers roadmap focuses so much on learning via technical books is that it lets experts create a guided learning path that walks through topics in the proper order.
The National Academies discuss learning progressions from a teacher’s perspective, explaining how to use them to ensure students understand a topic from the underlying principles. They give the example of a student needing to understand that the Earth orbits around the sun and rotates on its own axis before understanding why the Earth gets colder at night.
Structured learning roadmaps do a good job of “indicat[ing] common misconceptions students have about a topic.” This structure enables revisiting those misconceptions to ensure proper understanding from the ground up.
A modern-day example I’ve experienced of AI negatively impacting the learning process via lack of structure is the way current Deep Research tools work. I’ve tried to use these to help me understand complex topics in the past, but they just generate reports that don’t help me learn the underlying concepts that are vital to getting a deep understanding.
Interaction
The World Bank report emphasizes the teacher-learner relationship as central to learning. This relationship is a familiar part of classroom education.
This relationship is so important because it enables interactivity in the learning process. A student can clarify questions directly with someone who understands a topic better than they do. Feedback helps students understand and correct their thinking—directly assisting with the work that needs to be done to effectively learn.
Interaction can also take place directly inside the learning material. Learn Your Way, built by Google Research, transforms textbook material into interactive learning formats. Its questions and activities show how AI can turn teaching material into things we respond to and get feedback on.
Interestingly, the World Bank report argues that technology needs to strengthen the teacher-learner relationship to improve learning. In the math study above, teachers helped direct the AI through examples. I think parts of this teaching exchange can also occur solely between the AI and learner when done correctly.
However, doing this properly is a difficult task. It leaves questions such as: What needs to be generated, and how should it be interacted with? What topics are most important for interactivity for the learner to best learn? Many of these are more intuitive for a real-world teacher to judge than for an AI tutor.
Recall
Many studies show that recall helps retain information in memory. One example discussed in How People Learn II found that students who recalled a text from memory retained more of it one week later than students who only reread it.
There are many strategies in modern education to make recall a standard learning practice: testing a concept more than once (such as on a midterm and a final), having learners recap a topic after going through a section on it, study plans that force learners to go back over previously learned topics again (for large-scale standardized testing, for example), and more.
Returning to material and trying to recall it is an effective way to retain information over time. Recall gives learners the opportunity to struggle through the information again and commit it to memory.
Between the ephemeral nature of AI chats and other generated assets and the “let me do this for you” messaging of AI assistants, recall doesn’t come naturally during LLM learning. Great care has to be taken to build a tool that enforces or encourages it.
Personalization
All people come to education with different backgrounds, needs, and goals. This means some subjects, teaching styles, concepts, etc. are a better fit for some individuals than others.
In modern non-AI education, we try to address this through different learning paths. There are special and gifted education programs, remote curriculums, accelerated curriculums, private versus public school, and more. These programs aim to help individuals learn better based on their needs.
In my opinion, this is the biggest potential application in education for AI. We can create robust user profiles that enable the AI to understand a user’s needs, such as their grade level, interests, and learning goals. Google Research’s Learn Your Way is a great example. It adapts textbook material to a student’s chosen grade level and interests, generates quizzes, gives feedback, and offers multiple learning formats.
In a Google-funded randomized experiment with 60 US high-school students, one group studied a chapter using Learn Your Way and the other used a digital textbook. On the test immediately afterward, Learn Your Way students averaged 77%, compared with 68% for the textbook group. On a follow-up test 3–7 days later, the averages were 78% and 67%, respectively. Fifty-eight students completed that follow-up.
Students using Learn Your Way also spent longer studying than the textbook group. The study tested the whole experience—including quizzes, feedback, and different formats—so it doesn’t show how much personalization alone contributed.
A true AI tutor needs to understand the user and adapt to what they already know and how they learn. This is something I see as very possible in the near future and one of the biggest improvements that can be made for modern education systems.
What this means for engineering
Building AI products for education means turning what we know about learning into decisions the software can actually make. What does the learner need to understand first? When should the AI explain something, offer a hint, or let them work through it?
The GPT Tutor example above gives us a starting point. Teachers supplied correct solutions, common mistakes, and guidance on feedback. Their expertise became part of the information and instructions the model used to teach.
We can apply that thinking to a custom skill with supporting references or an entire learning tool. This is something I’m working on with my own Deep Research skill (break-it-down—announced in the last article). I want to use guidance on learning to help me work through the research and build on what I understand.
For example, the skill could ask me to explain a paper’s result using the concepts we’ve covered. It could then use the source material to give feedback on my reasoning and ask me to apply the concept to a different example. Saving those attempts would give future sessions a basis for deciding what to revisit.
This creates concrete engineering work: selecting references, defining how the model should respond to different attempts, and preserving learning history. A full product could also control the order of activities and schedule later practice. But if the tool mistakes a familiar phrase for understanding, it could build the rest of the lesson on a gap in the learner’s knowledge. We need to test whether its feedback is accurate and whether learners can explain and apply the material independently afterward.
This is why I think the most difficult part of applied AI is often understanding the problem space one is working in as opposed to building the actual AI solution. While there are many easy ways to “learn” with AI, there is complexity in ensuring a solution creates the long-term durable impact learners are looking for.
I firmly believe applied AI needs more engineers because the applications in which AI can make a positive impact are only going to grow. I also believe we’re as deeply in need of experts in other fields to make this impact as well.
Thank you for reading! Going through these studies and thinking on this topic has made me reapproach the way I use LLMs for learning, and I hope it does for you too.
I’d love to hear more about your experiences using LLMs to learn and understand what has worked for you and what hasn’t.
Always be (machine) learning,
Logan
If you want to learn more, check out these sources:
Generative AI without guardrails can harm learning: Evidence from high school mathematics by Bastani and colleagues
World Development Report 2018: Learning to Realize Education’s Promise by the World Bank
The Price of Inaction by UNESCO, the OECD, and the Commonwealth Secretariat
How People Learn II: Learners, Contexts, and Cultures by the National Academies of Sciences, Engineering, and Medicine
The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research, published in Frontiers in Psychology
Towards an AI-Augmented Textbook, the Learn Your Way research paper
An experimental evaluation of an AI-powered interactive learning platform, the Learn Your Way classroom experiment published in Frontiers in Artificial Intelligence








I think exactly the same!