Vague Patch Notes: MMO players are a bad substitute for professional testers

And it's not the players' fault

Oh snaps.

Test servers are, let’s face it, kind of a terrible idea that just keeps getting trotted out there in the MMO space. And the older I get and the more I understand about the industry, the more I see them as unnecessary at best and actively harmful to the testing process at worst.

This was on my mind during the most recent World of Warcraft interview roundup, in which it was mentioned that part of the problems with Torghast weren’t found by the people in the test. Now, the point here isn’t to drag that particular statement (which sure sounds like it’s patently false if you were paying attention to the actual beta feedback, but whatever) but rather to look at some of the philosophy of testing that went into this and how it might have contributed to the problem.

Real talk: Part of the problem here comes down to relying on players testing the system to provide useful feedback. And players have different priorities than actual testers, and you can’t just unleash a bunch of them and act like you’ve gotten a good testing session taken care of.

Let’s take a step back. What, exactly, is testing supposed to accomplish? Because in broad strokes, there are three things that fall under the broad header of “testing,” and they tend to all be conflated as one job. In broad strokes, we can look at testing as having three main components:

  • Functional testing: This is the nuts-and-bolts, kick-the-tires sort of testing. Does the server hold up under load? Are there glaring bugs with clicking on something? Can you get locked in infinite loops, good or bad? Do the systems perform like they’re supposed to? This is the process of locating bugs and existing problems and making sure that they’re addressed and corrected.
  • Conceptual testing: Is this fun to do? Is the gameplay loop satisfying? Do the quest parts lead into one another and have all of the appropriate connective tissue? Are the rewards commensurate for the time put in? The goal here is to look beyond the immediate elements of whatever is being tested and place it into a larger context, especially when referencing design documents.
  • Executive testing: Do the various parts of this content load consistently? Is everything working properly and giving you an idea of how to play it? How do these systems interact with others? This is the basic nuts-and-bolts path of making sure that the systems and content are explained clearly and are approachable for players.

You might see the problem just from looking at these categories, but let me make this explicit. Most of this stuff are things that players don’t actually care about. No one who is playing on a test server cares about where this new content slots in on a design document, just whether or not it’s fun to play on its own.

Oh, COME ON, what did that accomplish?

In fact, a lot of these things are actively antithetical to the way that players on test servers are looking into all of this. Yes, players want to find bugs and make sure that they’re not there, but they also don’t always know if something is a bug or actually the way it’s supposed to work. Players can tell you whether something is fun or not, but they can’t tell you if there’s a reason something isn’t fun. Heck, most people on test servers are their primarily to try out new content, not really to test it.

Mind you, I’m not faulting the players here. If you’re not getting paid to test this stuff, it is by definition a hobby rather than a job, and there’s no way to fault players for treating a hobby as something to do for fun. But that’s going to naturally affect the quality of testing, and it’s also going to affect how well studios communicate back and forth with the players.

For example, let’s say that hypothetically a new patch for a game adds boat-building. Let’s just have fun and claim it’s, oh, Star Wars: The Old Republic. There’s a whole system for building your own boats in there now. Why? Who cares. The point is, you’re building boats.

Immediately, players start coming back and explain that boat building isn’t fun, it takes too long, and there aren’t enough places to use the boats. There are several possibilities about what could be happening here. It could be, for example, that the boat building mechanics aren’t explained well enough, so players don’t know how to build boats properly. It could be that boat content isn’t ready for testing yet, so boats are still limited in their usage. It could be that boat building is supposed to be slow because you’re supposed to be done after building one boat.

But absent any of that information, players are going to take away that boat building isn’t fun and isn’t useful. And without having an understanding of why players are finding this system unfun – because players don’t have a full picture of how boats are supposed to work – it’s very easy to fall back on just reassuring players that it will be fun once they have the full picture, even if maybe the problems are very real and need to be actually fixed.

No, this is not a boat.

Now, to be fair, there are some things that are best tested with masses of players. Server stability, for example, is much easier to test when you have a whole lot of people stampeding onto your servers with a variety of connection speeds and locations. Similarly, opening up the game into the wild is often the best way to ensure that players at different hardware levels can run the game decently. That’s just basic logic.

The trick is differentiation. Players are useful for testing these things when the game has already undergone most of the other testing. You’re pretty sure that the systems work and are stable. The testers are finding things fun. The game is playing the way it’s supposed to. This is what you hire testers for, people whose job is actually to play the game and provide useful feedback.

And here’s where test servers start to be actively malicious. Because the more the function of tester is moved over to players, the more tempting it is to skimp on the QA staff who can actually do the important work of testing the game in its totality, getting into the areas where players tend to neglect, actually taking a full and comprehensive look at the game and testing it in all the ways it needs testing.

Not putting investment into that, of course, leads to a cycle of hurting. Updates come out with more bugs, less polish, and less understanding of what players want. The blame gets foisted on the people who actually were doing the testing, and then you cycle back to the next update, and…

Yeah, you get the idea.

Test servers are a neat idea. But they’re a neat idea that doesn’t really work as a free battery of professional QA testers and cannot replace them. And the more a company tries to replace proper testing with player efforts, the worse things are going to go as a whole.

Sometimes you know exactly what’s going on with the MMO genre, and sometimes all you have are Vague Patch Notes informing you that something, somewhere, has probably been changed. Senior Reporter Eliot Lefebvre enjoys analyzing these sorts of notes and also vague elements of the genre as a whole. The potency of this analysis may be adjusted under certain circumstances.

No posts to display


Please Login to comment
newest oldest most liked
Subscribe to:

Not surprising. QA is a job. Players just want to get in, see how they like it, maybe tell their friends that they have access, etc. I guess it provides some publicity since a lot of content creators are a big part of it.

QA people need to know what to look for, how to communicate what they found. I don’t even know if a lot of player beta testers even put IN bug reports.

To me, this is where you put in the polish, (and sometimes you leave big, glaring mistakes in it) and when you skimp on it, you are skimping on development. Quality assurance isn’t something that’s apart from development, but a part of it, like proofreaders are part of the publishing process.

Dankey Kang

Counterpoint: MMO players are either free or pay you to test their product.


No kidding.

Loyal Patron
Patreon Donor
Kickstarter Donor
Dean Greenhoe

You get what you pay for.

Right now, people are actually paying the developers to test their games via alpha and beta access. So the testing you get is limited.

When the developers’ actually start paying* players to test and provide the tools required only then will you get true testing done by players.

*Trinkets, titles and other cosmetic items is enough for many folk for testing.


As a player, I’ve always hated when game devs rely on players to test their games, rather than hire a qualified testing staff to handle it. I’m there to have fun, not have the game be a buggy mess which should have been fixed before it was released.


The ironic thing about this article is the fact the “three main components” of testing you list, you literally pulled out of your ass. I invite you to go google “QA Executive Testing” and see what you get back. It’ll be a bunch of results about how to be come a professional QA Executive. It’s not a thing. Neither is conceptual testing. Functional testing is a thing, but you don’t understand it.

The types of software testing are:

Unit testing – done by a developer testing a specific function or method to make sure that one piece of code works. There’s also automated unit testing and test driven development or TDD, but it’s not necessary to go into that for this post.

Functional testing – Done by developers and QA. Making sure some feature works. A functional test crosses multiple unit tests in terms of the code involved. “When I cast fireball, do I see the proper animation, spell effect? Is the initial damage correct? Is the burning DoT applied and the correct value?”

Integration testing – Done by developers and QA but primarily by QA. Making sure that when you take all of your individual modules that passed functional testing and put them together that everything still works… that module A doesn’t cause module B to break even though module A and module B passed functional testing individually. “Fireball and icebolt both work individually, but if I cast fireball and then try to cast icebolt while the fireball animation is still playing does icebolt also cast or am I forced to wait until fireball finishes?”

Performance testing – Done by QA, typically specialized QA with specific performance testing training and skills. Making sure the software operates under load. “How many users does it take to make the server buckle?”

UAT or User Acceptance Testing – Done by stake holders (executives) and a subset of users (beta testers. in the case of MMO’s us). This is to validate the software meets the requirements set out at the start of the project. In the case of MMO’s this boils down to ‘is it fun’, ‘is it engaging’, ‘is it immersive’, ‘does a warrior feel like a warrior’, ‘does a mage feel like a mage’, ‘is it balanced’, ‘is the itemization engaging’ etc.

Regression testing – Done by QA. Making sure when a new feature or bug fix goes in it doesn’t break something else or ’cause a regression’. “We fixed the fireball animation, but that fix broke the icebolt animation for some reason”

To sum up, testing isn’t even what the author thinks it is making this article a case-in-point of why MMO players are a bad substitute for professional testers.

Robert Mann

It’s a bad idea for one very simple reason: Players tend to be there to PLAY, rather than to specifically “try to break things”. Any good tester is going to be actively trying to break everything.

Kickstarter Donor

“MMO players are a bad substitute for professional testers”

This is just an unavoidably true statement.

Players just want to play a game for fun, the sooner the better, and if that means under “the guise” of testing it, then that is what they will do.

Whereas testers, are doing a job. their concern is the continuation of their job, which in turn leads to the continuation of their paycheck allowing them to meet the costs of living and so forth..

Even back in the day when they had proper Beta testers, even though we weren’t on the payroll, we still got incentives for doing the job. Namely a free copy of what we were testing and our names in the credits accrediting our participation. And when testing we had NDA’s and strict guidelines on what they wanted testing, and deadlines to get feedback reports in, etc.. it was treated seriously.

These days there is none of that, instead, they charge US for the pleasure of testing their game, give us no accrediting for the work and call it “Early Access” like they are somehow doing us all a favour by letting us fix their broke a**, in development projects.

Wilhelm Arcturus

It depends on what you want out of a test server. If you think it is going to be a general testing ground where players are going to reduce your QA needs, you are sadly mistaken. If you want player feedback on specific new features, and are willing to put in some effort directing players, you might get your money’s worth out of a test server.

EVE Online does a good job with its test server. They put up new features there, do a dev blog about the new features describing them in detail, ask players to go try them out, and have a forum thread setup for feedback. You may not like every change CCP makes to the game, but they do step up and let you give feedback. They sometimes ignore that feedback, and many an old school capsuleer has an “I told you so” story, but everybody gets a swing and the new mechanics and CCP does often change or tune features based on that feedback.

But, while people do sometimes find bugs, the main purpose is for people to try the feature and give feedback. It is more of a play test than anything. And EVE Online, as a sandbox where most new additions are mechanics and not PvE content like raids and dungeons, this system mostly works.

Likewise, when Blizz puts new raids on the test server, this is more of a playtest exercise, getting feedback on difficulty and enjoyment rather than a bug hunt. They’ll happily take bug reports, and no doubt they get plenty, but they are not primarily there for that sort of testing.

The bigger problem when it comes to testing is that if you sit down for a bit, open up a spreadsheet, and start making a matrix of things to test in any sort of open world game, if you aren’t quickly swamped by the vast number of combos that can occur in the game, then you aren’t doing it right. Be nice to the QA people. They are understaffed, under payed, under appreciated, overworked, and almost always invited too late to the party to head off disaster.


Are ‘public beta’ tests really EXPECTED to derive any meaningful reports from the TESTERS?

My understanding was that they have two purposes:
– load testing. Basically like opening the habitrail to hundreds of gerbils at once. You don’t expect them to actually do anything themselves except be gerbilly: eat, shit, fornicate, and generally do everything you expect them to do later, but in large potentially system-straining numbers.
– (now) it’s a way to farm stupid…er, ‘eager’ fans of their willing cash to be the first to try your new thing even if it doesn’t really work.

I did beta testing for a couple of years when I started the reviewing gig.
REAL beta testing is frustrating, boring, repetitive work. You test shit to break it, meaning little things, repetitively, trying to find the boundaries of functionality and failure, and then report exhaustively.

It is NOT “just playing the game earlier than your friends because you’re kewl”.

Bruno Brito

I did beta testing for a couple of years when I started the reviewing gig.
REAL beta testing is frustrating, boring, repetitive work. You test shit to break it, meaning little things, repetitively, trying to find the boundaries of functionality and failure, and then report exhaustively.

Hence why it was paid ( because it was work, and not really the kind you enjoy ), and why it had a way better chance of finding issues. Nowadays, games launch on a ridiculous state and spend a lot of the future development time fixing that state.