Tecnalia Paper Review

Intro

The beginning of this study describes a state of the world that, in my opinion, won’t surprise anyone. Soft skills are important, soft skills are difficult to train and teach, soft skills are even more difficult to assess, but with the advancements in technology over the past few years, we have new ways to address these things. We just need to actually do it.

I often have conversations about assessments and evaluation when I talk or present about the work I do. I get a lot of “How do you know it works?” which is great. It’s a question we need to ask and a question that needs to be acknowledged as very difficult to answer.

How do you measure someone’s patience? Besides giving specific examples, how do you tell one person that their communication is worse than their co-worker’s? What scale are you using that isn’t grounded in your own experience or perception? This is a common trap a lot of people (L&D folks, looking at you!) find themselves in. Institutions want “better teamwork” or “reduced conflict”, but actually identifying the scale which they would use to acknowledge when things are “better” or “reduced” is nebulous at best.

I remember having a written test in elementary school that was supposed to gauge how well we followed instructions. It was a page with about 20 different instructions on it and if you followed the instructions “correctly”, you ended up handing in a page with your name in the top right-hand corner. If you didn’t follow them correctly, you ended up making a bunch of doodles, writing a bunch of random words, and taking an extra 20 minutes to complete the task. Was that a good measure of my critical thinking abilities?

What about the scripts and role-play we see in training scenarios these days? Are they accurate representations of how someone will problem-solve in a group?

This paper is essentially saying that video games can help us get a better understanding of where people are with soft skills and how much they are improving. More specifically, they’re exploring whether participants’ gameplay data (via Steam) plus a model that maps in-game achievements to soft-skill constructs can be used as a form of “stealth assessment.”

What the paper tried to do

So we know that soft skills are important. If anyone tries to argue that, they clearly have never had to participate in a “team huddle” or heard the term “circle back” or “we’re building the plane while we’re flying”. Ugh.

In this study, the hypothesis was that commercial video games and Steam data could help assess soft skill development.

This is a big ask and I think the group did a fantastic job at meticulously analyzing the data they had available to them. There’s a ton of information in this paper and I highly suggest a thorough read through. For the sake of this article, I’m going to skip to the findings.

They wanted to attempt to measure complex problem solving, critical thinking, cognitive flexibility/adaptability, and time management using what they called a “Stealth assessment” which meant their AI model would take the gameplay data from Steam without the players actually having to do a test.

They were being assessed without doing an assessment.

Study design

The participants and groups are set up pretty standard for this type of research from what I found. Maybe it’s because I tend to read a lot of the same research that seems pretty normal.

What they wanted to do was have people play certain games for about 15 hours. The average was 18 and then they scrubbed the Steam data with their AI.

They looked at achievements and ranked achievements based on what skills needed to be used to gain those achievements and used that information to feed their results.

Keep in mind this is a gross oversimplification of the complexity of this study. I really encourage anybody who’s interested to get in there and read it because they go really in-depth and it’s awesome.

The results

The final conclusions of this study were that there were no significant statistical differences to indicate that video games, measured in the way that they were, improved soft skill performance in players. There were a few situations that were close to significance, which is a statistics way of saying “almost”.

I think that it’s important research and personally I find it very promising that the gameplay in this way didn’t result in statistical differences in soft skill development. That may seem counterintuitive but hear me out.

In my practice, in my own experience, and in my own amateur research, especially compared to this, I have found that video games are the tools and there needs to be some sort of facilitation to ensure the soft skills are beig addressed. An analogy I like to use to explain this is that of a novel.

Reading a novel alone is a different experience than reading with a group for a specific purpose.

For example, someone who reads a historical fiction book might miss the political intricacies that occurred at the time out of that work. However, if there was a conversation during the reading of that novel, or in my case if there is a conversation during the playing of a game, then the focus shifts to the development of the skill or, continuing with the analogy of the novel, the political intricacies of the time.

It’s a very convoluted example but what I’m trying to say is that in this study people were meant to take the game, go away, play it for 15 plus hours, and they were hoping that the skill was going to develop naturally. This is where they found no statistically significant difference.

I think it has a greater impact if there is a layer of facilitation to the experience where someone can guide the focus back to the skill at hand.

The mechanism

There were some really interesting connections between what this study focused on and what I’ve seen in practice. One thing in particular was that someone’s gaming history might imply skills that they’ve practiced or that certain achievements or tags on Steam or Epic or Riot might show somebody what skills they should train.

I really love this connection between skills and gaming as a sort of connective identity. I think that that’s really interesting and exciting.

What this means for L&D

For learning and development or education as industries I think that this research paper and the surrounding Megaskills Initiative is very meaningful. I think that it highlights commercial video games in a way that we can leverage the good work that other teams are doing in creating these masterpieces so that we can help build up our workforce or the workforce in general in a good way.

I don’t think that this paper was meant to necessarily prove that video games teach soft skills. I think that what this paper does is show that there are different ways of assessment that haven’t quite been explored yet.

When we’re working with something as difficult to authentically assess as soft skills, trying to approach the practice of assessment and evaluation in a different way is increasingly important.

The paper suggests we might be able to measure soft-skill-adjacent behaviour in games, which I find important because measurement is usually the bottleneck.

I do think it also opens the door for further research and further conversations about the role of commercial video games in soft-skill practice and assessment, when paired with deliberate facilitation.

— J