Research has been one of my most common use cases for AI over the years. As AI has gotten better, I’ve found myself more apt to reach for AI agents and LLM apps to learn about topics and search for sources.
One tool I’ve often reached for is the built-in Deep Research feature in LLM apps. Over the years, though, I’ve become more and more disappointed with it.
For those unfamiliar, Deep Research takes a query from the user, reformats it to be more agent-workflow-friendly, finds related sources, and generates a single report that synthesizes an answer from those sources.
This sounds like a brilliant idea, but I’ve come to realize just how limiting the process is when it comes to learning. It isn’t collaborative, and it follows a fixed workflow that reaches an output after a single query, regardless of what the user is trying to learn. It also seems to me that Deep Research hasn’t drastically improved in a long time.1
Since agents have become much more capable and can now follow multi-step workflows on their own regardless of the application (see Codex, Antigravity, Cowork, Pi, and many other harnesses), Deep Research can be recreated to be much more effective, free-flowing, and collaborative.
And that’s exactly what I did. Along the way, I learned quite a bit about the Deep Research learning process, how LLMs fit into it, and how to keep my process focused on my actual goal: making sure I learn what I want to learn instead of just collecting sources like Pokémon.
Below, I walk through the philosophy behind my research skill (which I’m calling break-it-down), organized around the goals I set for it based on what I’ve tried before.
1. Sources come first
By far, my biggest complaint about built-in Deep Research features is how they find and include sources. The most beneficial research sessions I’ve had (and articles I’ve read) generally reference fewer, more comprehensive resources rather than many less comprehensive ones.
Most Deep Research reports find more than 120 resources (sometimes up to 200!) and include anywhere from 15 to 40 of them in the final report. At first, I thought this was great. Vetting sources is a time-intensive human task, and it’s something AI should theoretically be able to do without limits.
However, the more I manually vetted the resources the AI found, the less confident I became in the report. Frequently, the AI would reference Reddit as a source. I see this as the equivalent of millennials or early Gen Z-ers referencing Wikipedia in their grade school reports. Granted, if the query was asking for opinions, Reddit might be a good resource. But I found many AI tools prefer Reddit even for concrete technical facts.
Citing Reddit was made worse by the fact that most of these reports don’t cite inline (i.e., hyperlink the specific information taken from a source). Instead, they put links to every source used in a paragraph at the end of that paragraph.
Combine that with 30 total sources, and it becomes really difficult to vet the facts for yourself and trust the information the AI pulled.
To fix this, the primary focus of my research skill is to find fewer, more reputable resources and put them front and center before any sort of synthesis. It literally lists sources and why they were included before writing anything else.
This enables a few things:
It’s much easier to vet sources, understand where information is coming from, and decide whether what the AI found is reliable.
It encourages reading the sources for yourself. There’s an interesting notion in the AI world that complete sources are no longer needed because AI can read them for you. I’ve especially seen this claim made about books. That’s entirely wrong, and we’ll get into why further down.
It makes the synthesis more reliable. I’ve found that the more sources the research workflow includes, the more confused the agent gets when synthesizing them. That leads to more hallucinations and a greater chance of pulling bad info from a bad source.
The focus on sources carries through the rest of the report, too. The agent is encouraged to cite the source of each claim directly where it’s made.
This means literally pulling the graphic that supports a claim, the quote behind an outcome, or the specific metric that shows a result, and placing it in the final report for the reader to see.
Along with making the agent’s information easier to vet, this instruction has done a great job of making reports easy to follow, information-rich, and clear. It’s almost like the original source authors included metrics, graphics, and phrases to make it easy for readers to understand their work. Pulling those directly into the synthesis has made the summaries much better. Where agents would otherwise paraphrase and lose context, they now retain the full context needed to communicate an idea.
Interestingly, pulling source material was the hardest instruction to get the agent to follow. Agents favor text and markdown they can whip up on their own, which you can see in the pre-built Deep Research tools too. It’s also where I’ve seen the biggest difference between Flash- and Pro-tier models when using this skill. Flash models don’t like doing the extra work, and Pro models are willing to.
2. Start from an ELI5 foundation, no jargon
My second biggest complaint about current Deep Research tools is just how jargon-y their reports are. Most of my learning is on technical topics, and technology can be full of jargon, but I see this in research reports regardless of topic.
Part of me thinks this is simply because the agent is asked to create a technical report, so it leans toward jargon. But I also think LLMs in general seem to be post-trained to be verbose and “sound smart.”
One theory is that smarter-sounding LLMs are generally preferred on platforms like LMArena, so post-training biases toward that preference. That’s plausible, since back in the day LMArena was the go-to source for model comparisons. Now we have many benchmarks across many tasks, so overfitting to that one comparison seems unwise. Either way, most LLMs still prefer verbosity and a specific way of speaking.
When you’re learning something new, jargon and verbosity make it much harder. Jargon often requires the reader to have more background to understand a report, but does nothing to tell them what that background is.
So in my skills, I focus on:
Taking an ‘Explain it like I’m five’ approach. This seems silly for complex topics, but it’s actually the best starting point for learning them. It lays a strong foundation, gives a high-level understanding, and doesn’t assume much prior knowledge.
Building on first principles related to the topic. The agent identifies what the reader should know before getting into the topic at hand and finds sources for that background as needed. Those sources are included in the report too.
Being direct and answering the question first. The agent answers the immediate question before anything else instead of burying it in facts. This helps the reader find what they’re looking for and read the rest of the explanation in context.
This works because my skills also tell the agent that its output isn’t a final report. It’s a starting point for further learning.
3. Let curiosity drive the learning
There are very few questions that don’t lead to more questions. In research and learning, I’d venture a guess that there are almost none. Using the report as a starting point for those questions enables a real research and learning process, customized to the user and their goals.
I mean this both in terms of what they want to learn and how they learn it. Learning is incredibly personal, and AI learning tools are capable of working with each learner’s strengths. They should prioritize doing so.2
Traditional Deep Research tools do let the user ask follow-up questions, but the report generally doesn’t change, and some of the most important information stays ephemeral within the conversation. They’re usually a single-shot process: one input, one output.
My first version of the skill did the same. I thought it would beat the existing Deep Research tools because I could customize the workflow and guide source collection. I also figured I could give it plenty of samples of my writing, which reflect my preferred output style, and have the agent emulate it.
It turned out the single-shot method wasn’t sufficient for advanced topics, and the AI was much worse at emulating my style than I expected. Surprisingly, Pro-tier models and higher reasoning didn’t help much. The agent would always fixate on certain phrasing and structure, even at the expense of explaining the topic well.
My goal with the skills is to capture this information and keep the agent guiding the user through high-quality sources. My primary methods for doing this are:
Building interactive elements where possible. These are tough to get right and make meaningful, but they’re one of the greatest benefits of agents explaining concepts. To enable this, the agent creates the final report as HTML, so it can generate any sort of web asset in it.
Telling the agent to expect follow-up questions and encouraging it to ask the user how they want to follow up. This keeps the follow-up reader-led.
I also highly recommend using a harness, app, or tool that lets you highlight elements in the report and ask questions about them directly. This helps direct the agent and build out the report.
4. Be careful what you summarize
Summarization is an application ripe for LLM impact. So much so that back in 2023, I wrote an article encouraging readers to look into other summarization methods because of the cost of LLMs and how effective those other methods were.
I don’t think that article applies much anymore because LLMs are now so effective and so cheap, but there’s still merit in putting more thought into how we summarize and when we do it, to make sure it’s actually useful. This became especially apparent while developing these research skills.
As mentioned before, I’ve read some crazy takes about reading sources, books, and articles being irrelevant because AI can now read these things for us and distill them to just what we need to know.
In reality, LLM summarization is incredibly fallible, both in how accurately LLMs summarize and in whether summarizing with them serves a useful purpose at all.
My first takeaway is that LLMs often summarize incorrectly. In longer contexts, they tend to recall information unevenly, favoring what’s at the start or end of the text. This biases what ends up in the summary.
LLMs also tend to hallucinate, especially at longer contexts. Incorrect information is obviously antithetical to an effective summary, and it’s made worse by how hard hallucinations can be to detect in a Deep Research context. If you’re using these tools to learn, that’s directly counterproductive.
When most of us use LLMs, we expect them to do things for us. When that happens, we don’t experience the effort needed to learn.
In learning, summarization is the process. The effort needed to learn comes from reading information, connecting related ideas, and simplifying concepts for our own understanding. In Deep Research, we need to be careful not to offload all of that synthesis to the agent.3
I’ve got an article coming later this week going more in-depth into learning with LLMs. Subscribe to get it in your inbox!
This mostly needs to be addressed by how the user works with the agent, but I’ve set up the output of break-it-down to help users synthesize for themselves.
First, sources are listed before anything else, emphasizing where the information came from and what the user can learn from them (see the sources section above).
Second, the report is largely built from the figures, quotes, and other supporting evidence in those sources, which points the user back to them.
Third, the agent is instructed to build the report from first principles and explain things simply first (see the ELI5 section above). This makes for a more complete explanation, and one that encourages the user to dig deeper and keep learning on their own (see the curiosity section above).
Even with these stipulations, it still comes down to the user using the agent intentionally.
My general advice for summarizing effectively while learning: use summaries to decide if you should read a source, not to learn what is in it.
For example, I use an agent to filter my Readwise Reader feed (subs get an extended free trial of Readwise Reader) down to things I’d find interesting. This works well. Using an agent to summarize those important sources and draw out their insights is significantly less beneficial.
5. Good input means good output
My last lesson is less about my skill and more about me. Models have become so intelligent that I’ve found the limiting factor for what they can do is the context and direction I give them.
When I get garbage output, I often find my instructions were incomplete or my context was muddled by overlapping skills or bad instructions within them.
While creating break-it-down, I also ended up cleaning up the AGENTS.md files in the apps I use and in many of my repos. I tightened up many of my skills too, making them shorter and more direct, and making sure skills used together don’t misdirect the agent.
I’ve previously written about building skills for long-running agents, and that article covers most of the guidelines I use for writing good skills. If you haven’t audited your own skills in a while, I suggest doing so:
I’ve found it much better to be less prescriptive with current frontier models. Instead of giving the model explicit, step-by-step directions, I tell it what I want it to do and what output it should produce, then lean on its intelligence to get it right. The model often finds better solutions than I would. Leaning on its intelligence also means that as I upgrade models, or as the context available for a task changes, the model adapts on its own instead of relying on me to rewrite its instructions.
One thing I’m still working out is how to make the most of model intelligence without being too prescriptive, while still letting agents work with less expensive models. The gap between frontier and cost-effective models is currently very large, and the cheaper models still need extra guidance.
Please take a look at break-it-down and let me know what you think. Feel free to fork it and edit it to suit your needs. Even if you don’t use it, I recommend developing your own research workflow, even from scratch. I’m super bullish on the educational gains and accessible knowledge LLMs will provide, and a good research workflow is key to getting there.
Thanks for reading!
Always be (machine) learning,
Logan
As an aside, now that I’m writing this, I’m realizing that Deep Research’s stalled development might be because it was laying the groundwork for multi-step agent workflows, and the developers realized a separate feature might not be needed. This is just a thought from me, though.
Personalization is a big part of why I built my research workflow as skills: anyone can import them wherever they prefer to work with their agent of choice.
A good question at this point: why include a summary or report at all if LLM summarization isn’t good for learning? Why not just build a source collection skill that produces only the first part of the report and tells the reader to go read the sources? There are two reasons: 1. Writing the report helps the LLM make sure it has enough information to answer the user’s query. 2. Summarization isn’t entirely bad here; it just needs to be used effectively. The report goes a long way toward guiding the user, piquing their interest, and getting their cognitive gears turning.
Look forward to trying it out! Also a very common use case for me and I’ve definitely gritted my teeth in frustration more than a few times (and wondered if it’d be more efficient to search for sources myself) at the pitfalls you mention.
Very interesting from a HS educational perspective. TY