When AI Compiles Research and What Gets Lost in Translation

The Experiment

A few months back, I was having coffee with a friend and he was talking about how some of the kids on his son’s baseball team had such a better understanding of baseball because they played MLB’s The Show. He said they had a better understanding of the rules, certain faux-pas-plays, and best practices in certain situations. They couldn’t always execute (I guess that’s the difference between 10-year olds and professional athletes), but they had a better understanding of what could happen and what they wanted to happen.

Then we started talking about how it seemed obvious, but was there any research that had been done that supported it? I decided to give Claude a shot at compiling the research. I think it was Claude’s Sonnet 4.0 that did the research and it took about 45 minutes.

Baseball laying beside a bat - Wix Studio

Being my first time doing something like this, I though it was really cool because it brought together a whole bunch of research that sort of pointed to what I was looking for, but was a bit more deviated from what I would have looked for had I done the search myself. Anyways, here is the Claude article and this post is going to be my interpretation/critique of it.

What AI Got Right

Credit where it's due. The compilation accurately reported some genuinely useful findings.

The Jenny & Schary study (2014) is real and the statistics check out. Forty international university students played Madden NFL 25 for eight 30-minute sessions over four weeks. Football knowledge improved from 60.6% to 67.8% on comprehensive tests. Field layout understanding jumped from 54.0% to 62.7%, and player position knowledge went from 54.8% to 66.7%.

That seems legit.

The basketball research using NBA 2K16 exists. The soccer attention training study with young players is real. The pattern holds across multiple sports and multiple studies, so AI didn't fabricate the core narrative.

For someone who just needs to know "does this work or not," the summary delivers a defensible answer backed by actual academic sources.

What Got Lost

Context collapse: The compilation mentions "215 high school students" and "young soccer players" and "community college students" like they're interchangeable populations. They're not. High school students playing basketball games behave completely differently than community college students learning football rules. Developmental stage, prior knowledge, motivation, gaming literacy, etc. All of it matters when you're designing actual learning interventions.

Methodology handwaving: We get told there were "randomized controlled trials" and "validated psychometric instruments," which sounds rigorous. What we don't get is any detail about how long the interventions lasted, what the control groups did instead, how they measured transfer, or whether effects persisted beyond the immediate post-test. I know that I could go through the studies that were pulled, but this was one of the things that stood out to me. It almost seemed intentionally vague.

The Jenny & Schary study ran for four weeks with structured sessions. That's a completely different intervention than someone casually playing Madden at home. But the summary presents it as evidence that "playing sports video games" generically improves sports knowledge.

The expertise gap: This is the big one. The summary tells you that sports games improve tactical awareness. It doesn't tell you why, or what that means for selecting games, or what skills transfer and which ones don't.

An AI can report that something works. It didn’t explain the underlying mechanisms in ways that help you make better design decisions. Maybe that’s a fault of the prompt. I guess we’ll know more next time.

The Real Problem for L&D

I'm not picking on AI research compilation because it's fun. I'm picking on it because every L&D professional I know is drowning in this stuff.

Google Scholar alerts, AI-generated summaries, ChatGPT literature reviews, research briefs that promise to "synthesize the key findings" so you don't have to read the actual papers.

It's efficient. It's convenient, but it's systematically stripping out exactly the contextual details and methodological nuance you need to make informed decisions about learning design.

When you read that sports video games produce "12-19% knowledge gains," your next question should be: Under what conditions? For which learners? Doing what, exactly? Measured how?

AI summaries consistently fail to preserve that context. Not because they're lying, but because they're optimized for synthesis and compression. They give you the headline finding and move on. This is really important to remember. Large Language Models predict which word comes next (simplified explanation), not necessarily whether the word fits within the larger contextual meaning.

They are getting way better though, so maybe this research can be a benchmark every few months to see how the process is coming along.

What This Means for Evaluating Research

Whether you're reading AI-compiled research or human-written summaries, here's what to watch for:

Specificity matters: "Video games improve learning" is useless. "Eight 30-minute sessions of guided Madden NFL gameplay improved football rule comprehension by 8.2 percentage points among international students with no prior football knowledge" is actionable.

Methodology reveals applicability: If the study used highly structured interventions with trained facilitators, you can't assume casual gameplay will produce the same results. If participants were college students, be cautious about applying findings to corporate learners or middle-years students.

Look for mechanism explanations: The best research doesn't just report that something works, it explains why. Pattern recognition? Deliberate practice? Immediate feedback loops? That's the knowledge you can transfer to other contexts.

Check the missing pieces: What didn't the researchers measure? What limitations did they acknowledge? Often the most valuable insights are in what the study couldn't prove, not what it could.

The COTS Game Connection

Here's what I actually believe about sports games and learning, based on both research and facilitation experience:

Sports games absolutely teach sports knowledge. But they teach it inconsistently, and the mechanisms matter more than the medium.

When someone learns football rules from Madden, they're learning because the game provides immediate consequences for rule violations, because they're making hundreds of tactical decisions per session, because they're getting visual feedback on player positioning and movement.

Those mechanisms (immediate feedback, high decision density, visual representation of abstract concepts) transfer to other COTS games. They're why Portal 2 teaches spatial reasoning and why Overcooked teaches communication under pressure.

But you can't just drop people into a game and expect magic. The Jenny & Schary study used structured sessions. Participants knew they were there to learn. Someone selected the game deliberately and created conditions for knowledge transfer.

That's facilitation, not passive absorption.

When AI Research Summaries Are Useful

I'm not telling you to avoid AI-compiled research. I use it. I just used it as the basis for this entire article.

AI summaries are excellent for initial exploration, for getting oriented to a new topic, for quickly checking whether research exists on a question. It’s also way better at finding articles than I am. What would take me 10 hours probably took Claude 30 minutes. Are they all legitimate sources? No, but it’s better than nothing in most cases.

They're terrible for implementation planning, for understanding nuance, for making design decisions that require contextual judgment.

Use them as a starting point. Then go read the actual studies.

Test These Ideas

Sports games teach sports knowledge. COTS games develop soft skills. But only when someone who understands both learning design and game mechanics structures the experience deliberately.

That's the part AI can't compile for you.