Yesterday, I went on a rant about data. As you probably guessed from the various columns wherein I attempt to use data to analyze future points, I like data. I like math. It’s really interesting to see the ways that data will line up, converge, and ultimately give us if not a story at least the outline of one – even when it’s not really adding much of use to the discussion. I wrote an article about predicting the next expansion date for Final Fantasy XIV based on some math, and now I have firm math. It’s about four months! What do these data mean? Nothing.
Data are interesting. Subscriber trends, player participation trends, and overall engagement are all really interesting to me. These are things that I will stare at, analyze, and dissect for hours on end. But the thing is that all of these data don’t tell us nearly as much as you might think. There are limits to what data can do. And both people who love data and people who hate data can miss the fact that ultimately, data are just a tiny portion of the overall picture.
Case in point, I’ve done several columns of data sets for World of Warcraft based on potential dates between expansion launches for the game (here’s the most recent, with Dragonflight). These columns are genuinely really interesting to me. They provide a neat picture and a detailed analysis of how this is progressing. I really like doing them. But they don’t actually tell me the most important piece of information, which is whether or not the next expansion is going to be good.
This isn’t a failure of the data, though. The data were never going to tell me if the expansion was good or bad, just like the data cannot tell me whether I’m bored or not. Data don’t tell a story; data denote firm, quantifiable facts. And those facts can suggest a story, but it’s important not to confuse the two.
We all know that an individual data point does not tell a story. Saying, for example, that The Elder Scrolls Online has 500,000 subscribed players does not say anything by itself. Adding a separate data point starts to provide an arc, like saying that a month ago it had 2 million. But you need something more to really make that tell a story, like how three weeks ago the developers started a new program where players have to send ZeniMax a pint of human blood to stay subscribed.
Most games obviously will not actually ask you for blood. (I wouldn’t put it past Mortal Online 2, to be fair.) But the real thing here is not the obviously facetious example. Most people aren’t really concerned with subscriber numbers; they’re concerned with what those subscriber numbers mean – or more accurately, what those subscriber numbers can be made to mean.
If the game is good, you get more subscribers, and if the game is bad, you get fewer. That’s how it works, right? And the answer is… no.
It’s not entirely false as an assumption, of course; in broad terms it’s reasonably accurate. When a game sees a sudden massive drop in subscribers, that is usually in part or in whole because the choices that have been made for this particular game have run counter to what players want. But it’s not entirely certain. A game could lose a whole bunch of players because, say, another game launched and it’s sucking all the air out of the room at the moment. Or another game just had an expansion launch. Or your game is 20 years old and people are just looking for other things to do.
And what’s really going to put the pumpkin spice in your latte is that all of these things can be true at the same time. Data are only going to tell you about some of this.
This is where feedback from players can be vital, but the problem is that this feedback is also far from a sure thing. It’s possible to get a bunch of feedback that is all… you know, garbage. You could have players eagerly shouting in your forums about how this latest change is the worst thing since that time the game required shipping human blood in order to subscribe, but your actual numbers don’t show fewer people playing.
But does that mean those people are wrong? Well, if their posts are complaining about something being “woke garbage,” probably. But it’s also possible that a current change conceals negative long-term elements. And none of that information will actually tell you… you know, whether or not it’s good. Is this actually fun? Is this making the game better or just making it different?
Data cannot tell you these stories. Data can only give you points along the way. Data will tell you if your patches are coming out on a reliable cadence, but that’s only a good thing if those patches are well-received.
It is possible for something to be popular and bad. There are reasons for it, but it’s happened a lot throughout human history. It’s also possible for something to be popular and good but still not be to your particular tastes. I’ve never really cared for the Foo Fighters as a musical act, but I recognize that it’s not because the band is bad, it just doesn’t do it for me personally.
And there is a human tendency here to turn to data to try to point to something that is to my taste and show how it’s better. But that isn’t how data work. It doesn’t prove anything if, say, a movie I personally liked made more in box office returns than one I didn’t care for. By that metric, the movie Ratatouille is only 88% as good a film as Michael Bay’s first Transformers film. Can you believe that?
Do you believe that? Because you definitely shouldn’t.
It’s tempting to say that data paint a picture, and the fact is that I’ve probably described it that way before, but that’s honestly disingenuous. Data do not paint a picture. Data suggest where the outlines of a picture may lie. They provide a framework, a scaffolding. They give you a clear place to hammer a nail into the wall and hang a picture, and after you’ve done that you can look at say, “Well, that picture was made to go here, it really ties the room together.”
But these two things are actually separate. It is not that you put up the picture in the only place it could go. And while there are pictures in my apartment that I look at with satisfaction, as if it that was the best possible place to hang that picture, I also know that I am lying to myself. Those pictures could go almost anywhere. I could even replace them and the functions of the room would not change.
Data cannot tell you when the game is good, only how many people are here and spending money. It’s still useful, it’s still fascinating, and it’s still fun, but you cannot scientifically measure fun. Let data show the actual things they’re good at – and recognize their limitations.
Sometimes you know exactly what’s going on with the MMO genre, and sometimes all you have are Vague Patch Notes informing you that something, somewhere, has probably been changed. Senior Reporter Eliot Lefebvre enjoys analyzing these sorts of notes and also vague elements of the genre as a whole. The potency of this analysis may be adjusted under certain circumstances.
