Massively Overthinking: The state of early access, alpha, and beta ‘testing’ in the MMO genre

I remember years ago when then-Massively-columnist Rubi Bayer let loose with a blistering rant on the state of faux beta MMOs. She helmed Betawatch back then, see, and she was fed up with (mostly imported) MMOs claiming to be in beta when in fact they’d soft-launched. A lot of readers didn’t understand her fury at the time, but boy have things changed, right? Now, every game’s in on that very old trick, only they call it early access now, while some are still pushing the boundaries, charging $1000 for pre-alpha.

MOP reader Pepperzine proposed a topic for this week’s Massively Overthinking that’s right on point. “I was thinking it would be interesting if we could discuss when people consider a game to be in alpha/beta versus a final launch as a topic,” he wrote to us.

“Back in the day, this was easy to determine. Selective testers were extended invites into beta who were experienced testers who had the computer hardware to handle the software. The primary purpose of being in the testing phase was exactly that, to test and bug report. When the game was made available to the public at a price, a game was considered launched. Now, players are granted access to pre-launch titles by ‘donating’ or purchasing access. For the most part, the primary purpose of participating in the pre-launch experience for these players is not testing or bug reporting but rather to experience and play the game. The division of purchasing a game and donating to test has become so blurred that it is no longer a valid way of determining if a title is at a state to where it is launch ready. These titles can stay in this pre-launch phase for as long as they deem necessary, easily deflecting criticisms by reiterating it is still in development. So when do you consider a game to be launched? Is it when the producers declare it is? Is it when there is no longer the possibility of wipes? Is it when cash shop monetization is implemented? Is it as soon as the company begins selling access?”

Where’s the line in 2017? Let’s dig in.

Andrew Ross (@dengarsw): For me, as soon as you charge a consumer accessible price for your game, it’s released. You can put “early access,” “beta,” or heck, maybe even “work in progress,” and I’ll still talk about your game as a finished product. For you sausage-factory fans, I’ll tell you now that my position does get me in trouble, but frankly, I don’t care. I take this position even with games I like.

There’s a job called “QA Tester,” and those are the people who should be playing games when they’re unfinished. Those people are finding the bugs because it’s their job to do so and the company’s job to fix them. It is not the playerbase’s job to catch developer errors, and frankly, I find it both lazy and abusive. Games are expensive to create. I know, I get it. But I also feel like a lot of what we get are cash grabs. Old school MMOs not only had actual betas but launched buggy as hell without asking the consumer to consider the game a beta. They took a subscription fee and fixed things while adding content, and at least the ones I played did this at least on a monthly basis, often with weekly maintenances and hotfixes as needed.

Not everyone is going to pay for a subscription. I understand that. But you don’t need to insult your own game by releasing it and saying, “Don’t judge it yet, just pay for it while we fix it!” If you give us non-abusive store options (maybe additional customization options, character slots, reasonably priced non-stat-granting monocles…), we’ll pay. But as soon as you tell me your game isn’t finished but you want my money? “Released” is all I see, and that’s exactly how I’ll judge it, because that’s how casual customers see it.

Brendan Drain (@nyphur): I’m completely biased on this because I have a game in early access currently (a non-MMO space 4X game called Predestination on Steam), but here’s a viewpoint you won’t hear very often: As useful as it is for developers to have a game in people’s hands for testing and feedback purposes, I don’t want to do early access at the alpha stage or even crowdfund my games before the core gameplay is in place. Many new indies are practically forced to do it because games cost money to develop, and almost nobody will take a financial risk on new independent studio. I’d much rather develop in secret for several years and then have a big surprise announcement when the game is ready to show to the public, when it’s polished enough that the press might take it seriously, or at least when the core gameplay is all in place so we can act on feedback immediately. Broken early access games are a symptom of a broken industry.

Regarding definitions, I’d call a game an alpha when there’s a fully playable vertical slice of the core gameplay but it still has missing features and content, there may be game-breaking issues, and the core gameplay may even change based on feedback. Most devs do not want to let the general public play their games in this state but I think it’s perfectly acceptable to enter something at this stage into Early Access, providing of course that you’re up-front and honest about the development state and future plans and you need feedback this early to help shape the development of the remaining features and content. Once all the core gameplay is implemented and the devs are focused on iteration, balancing, tweaks, polishing, cosmetic changes, or adding more content above the basic requirements, that’s a beta stage. The expectation is that it’s too late at this stage to make fundamental changes to the core gameplay, and again it’s perfectly acceptable to be selling the game at this stage if devs are up-front about it.

The notion that we need to rid ourselves of in this industry is that the labels “alpha” and “beta” always mean the game is still pre-launch. Several games have pushed their last server wipe or opened full sales and microtransaction stores while still claiming to be in a beta state, but that’s kind of a launch by definition. Plenty of games have also engaged their full release strategies during alpha or beta, some to capitalise on the millions of sales made in an unexpected viral push and others running sizable marketing campaigns while still claiming to be pre-release. Those games have launched. Let’s not kid ourselves that when RUST officially hits the button and leaves early access that it’ll suddenly sell more units than it has already in early access, or that PlayerUnknown’s Battlegrounds won’t hit market saturation in its current development state. Those games have launched.

Brianna Royce (@nbrianna, blog): I think this determination has become increasingly harder to make than it used to be. At one point, it seemed clear to me that any game charging real money was at the very least soft-launched, and those of us who’ve worked on Betawatch over the years have treated it as such, stripping those games from the list. Now it’s so prevalent that treating those games as launched when in fact they are still hobbling around on some early access or beta crutch seems like doing them a favor, and I consider it more my duty to point out that crutch than ignore it.

I think the combination of Kickstarter and the decline of AAA MMORPG development changed the MMO scene more than early access, truthfully; it primed us for willingness to pay for games years before we’d even secured the “privilege” of becoming unpaid testers, which is worse than early access. It does seem absurd to me, on some level, to criticize a game like ARK for charging for a full-scale expansion during early access when we have MMORPGs charging actual subscriptions (or $1000 pledge packages or pixel ships) before launch.

This is more or less why in our Betawatch and Make My MMO columns, we’ve switched away from trying to judge strictly on whether money is changing hands or whether there will be future wipes or partial wipes and instead focusing more on whether legitimate testing is the chief goal of live servers, while also noting the business model situation too so that gamers can make up their own minds. I’m not thrilled with not being able to categorize more clearly, and personally I will keep right on judging games as paid products once paying begins, but until players stop allowing themselves to be taken advantage of, this is the state of our genre.

Eliot Lefebvre (@Eliot_Lefebvre, blog): Considering that I now run the Betawatch feature, it might not surprise people to know that I have strong opinions on the matter. They’re probably stronger than our “official” line, but I’m going to stress this just for emphasis: If an online product is taking your money for persistent access, it has launched.

The point here is not that the game is persistent and free of wipes; the point is that you can access the game on the regular and play. H1Z1? Launched. Star Citizen? Launched. Shroud of the Avatar? Launched ages ago. Any “beta” game with a cash shop has launched, even if there’s an upcoming server wipe and you’ll be refunded your money. The game has launched and should be judged as such. Specifically, I’m taking aim at games that tend to get away with “early access” as a marker; the launch date is when that access opens.

The rationale is that you’re being sold something, and it really doesn’t sit well how many games are willing to sell themselves to players based on an early development stage that may or may not reflect what the developers are actually going to do. H1Z1 is a particularly awful example; the game basically launched as a totally different product before Daybreak decided to chase money for a free-for-all gank festival, leading to Just Survive, rebranding, and multiple abandonments. It is, in a word, dirty pool.

Yes, I’m willing to catch games in the fallout that kickstart and then sell early testing access along the way. You are selling the game. Stop pretending you’re testing the game; you’re trying to launch without having to actually finish the game. And if your defense is “well, the game isn’t actually finished yet,” then maybe you shouldn’t be taking money for it yet.

Expansions I’m a bit more forgiving for; there’s a clear division between the two. It’s not hard to see that the next World of Warcraft expansion is still in testing when it’s not on the live servers. But I feel like early access in general gets used as the same sort of defense that “it’s just a beta” used to be; we should be judging these games on the basis of launching in mostly-finished or unfinished states a bit more vigorously.

As with many things, I’m willing to judge on something of a curve, and a lot of it depends on how aggressively a given title is going for money. Titles like Camelot Unchained, for example, really do feel like they’re going in for testing; Star Citizen has been selling people overpriced ships that only exist as “we thought of this ship.” But I’m more willing to come down hard on early access as a functional launch, and open betas are really a thing of the past.

Justin Olivetti (@Sypster, blog): Maybe I’m basic in how I see this, but an MMO is launched when (a) the world has switched over into a persistent (non-wipable) state and (b) the studio is taking money from players. I really think it’s that simple. Studios can debate semantics on this and try to apply labels in the hopes of giving them more PR leeway, but I’m not going to swallow that this is alpha or beta or whatever when both of these conditions are met. You’ve launched: Deal with it and don’t deny it.

Backing up a step or two, I honestly have no idea what constitutes test stages these days. To me, alpha means “mostly broken” and beta means “mostly functioning and currently polishing,” but I acknowledge that there are many degrees between the two.

Early access seems to be its own thing that could mean alpha and could mean beta but definitely means “we’ll take your money now, yes please.” But since there’s still going to be a wipe or two coming, it’s not quite launch. I think the industry and community has long arrived at the consensus that open beta is “we’re not actually testing anything, this is just a public demo to gain publicity,” but now it’s gotten a little more confusing with early access (which, as I said, is pretty much the same only with money changing hands).

MJ Guthrie (@MJ_Guthrie, blog): I have a few little indicators that tells me a game is launched, whether or not a company will call it such.

  • If you have a cash shop, you are launched.
  • If you sell paid expansions, you are launched.
  • If there are no more wipes because development of all core things are set, you are launched.

I don’t consider early access launch because some companies really do use that program as it was designed: as a way to further core development that couldn’t happen without additional funding. Unfortunately, there are those that abuse the program; some companies really do want to keep that “We’re not launched yet!” excuse going as long as they can! Not cool, and not fooling me. I also don’t like the proliferation of the term soft launched: launch is launch. I am, however, OK with people spending something extra on a package for a headstart and still calling the official launch the date everyone else can access the game.

Some games don’t play the “We’re still in development” card as an excuse for lack of progress, content, or other more nefarious misdeeds. For instance, I look at Warframe and wonder, now why in heavens hasn’t this launched? It looks like a launched game, it acts like a launched game (even better than a number of launched games!), but it is as far as I can tell still in open beta. Why? I haven’t seen that used as a reason to deflect criticism, and Digital Extremes is putting out constant content, including a massive– and might I add free — expansion. So I have no idea what is stopping that little verbiage jump to launch, other than maybe having to re-title the EULA you sign when you start the game. You can’t wait until the game is complete because an MMO-type game is never complete… until the day it dies (or maintenance mode, which is essentially the same thing). For what it is worth, Steam does note the game was “released” March 25th, 2013, so there’s that.

Patron Archebius: The official “launch” has become blurred in more than one direction – with the ease of patching and adding/enabling additional content, there are quite a few games that I wouldn’t consider launched regardless of what the studio says in public. A prime example of that is Battlefield 4, which “released” as such a buggy mess that they didn’t push out any major balancing patches until half a year later. The first six months were spent fixing a delightful range of game-breaking bugs, from instant-kill bullets (kinda fun) to randomly dying when jumping over stuff (less fun). This did not, however, prevent them from releasing new DLC!

For the big studios that can afford lots of QA and are going to cancel massively anticipated games because of “market forces,” betas have become ways to generate hype and see how everyone reacts to their latest lockbox shenanigans. I’m sure some valuable knowledge is generated, but how often do we hear “The beta is an old build, all the problems you’re reporting have already been fixed?” Everything is very near completion, at that point. The studio is rarely going to make any changes based on your feedback. The game has been researched, developed, and marketed – the beta is just breaking champagne against the bow.

And yes, early access has blurred this line as well. At some point, you’re paying for a complete project, and then watching it all come together. Sometimes the game, as it exists at launch, isn’t as fun as it was a few months back – ARK, for instance. Sometimes I never go back to games that I backed in early access, content with the time I’ve spent in them.

So, in short – I don’t think it matters any more. Games have moved away from being one-time releases that you dig out your SNES to play one more time. A game can be ruined or improved by changing developer focus years after launch; it can be a blast from the first moment of early access or a legitimate alpha-test slog. If they continue to improve and update their product, so much the better – but I consider a game to be launched when I can get in and play and have a good time with it, regardless of whether that’s a year before or a year after the official launch.

Your turn!

SHARE THIS ARTICLE
Code of Conduct | Edit Your Profile | Commenting FAQ | Badge Reclamation | Badge Key

LEAVE A COMMENT

61 Comments on "Massively Overthinking: The state of early access, alpha, and beta ‘testing’ in the MMO genre"

  Subscribe  
newest oldest most liked
Subscribe to:
Reader
tony quinn

I really like this website its updated and straight to the point with reviews, about time someone took notice of the mind boggling take advantage of rip offs for charging for a beta, I remember my first beta test it was Aion you had to sign a NDA with no charge for time, those where the days, I spent a lot of money for so called beta’s which are just released games with no intentions of getting better most notable of late Revelation online wish I had got my money back for that one. Hopefully people have caught on to this ridiculous take advantage of mentality. Thanks for the great articles

Reader
Zen Dadaist

How apt – using an illustration from the recent Warframe comic release. The game is claimed to be “still in Beta” as an excuse that gets trotted out now and then, even though they’ve dropped the visible “Beta” tags everywhere…

Reader
the_balance

I haven’t commented here since the unfortunate situation that turned these hardworking individuals into MassivelyOP.com, instead of the former just massively. We’ve had our spats (the staff here and I) over the years. Sometimes for good reasons, sometimes constructively, but always because we share the same passion for games and the articulation of news surrounding them. I’ve checked this site, in whatever form it existed, almost daily for the better part of a decade now.

Way back when Mechwarrior Online (MWO) dropped a (disappointingly) huge content patch in 2012 I wrote a post on their forums with the title, “No, MWO. Your game is not in beta.” It got me instantly banned from the game, a ban that persisted for over a year.

After that ban and the inability for me to respond/defend my post on the MWO forums, I came to a betawatch here and posted a similar perspective in the comments section. I doubt it contributed much, if anything, to the stance that this site started to take towards the beta moniker from that point forward. MWO was, however, off the betawatch list the next week. I’m not looking for credit here, or bragging – just trying to relate to the staff of this site and how much we’ve seen eye to eye over the years in spite of how much we’ve also disagreed.

The timing was likely coincidental with MWO. I’m telling this story because as coincidence would have it, this article came out the very same day I posted a similar argument on the Shroud of the Avatar forums, basically echoing the sentiment here that this game launched “ages ago” and the beta moniker the developers still use to describe the game is meaningless at this point.

I went on to say that the age of “forever beta” is finally coming to an end. We, as a community (gamers) have started to understand that what developers are doing is basically a delay tactic with the aim of getting another free bump in PR when you decide to use the word “launched” to describe a game everyone knows really launched years ago (in this case). I can’t tell you how much I’ve been frustrated both experiencing, and seeing my friends experience what this tactic does over the years. Promise the world. Deliver an island.

I doubt my post on the SOTA forums, something I like to refer to as the graveyard of hope, had anything to do with this article. It just shares a similar sentiment. The forever beta movement is coming to an end, a death it needed to die years ago. Seeing this article, and keeping it in the back of my head over the weekend, made me re-sub to these forums and post for the first time since the OP was added.

I did that, and told this story, because I need to say thank you. And not just the writers in this article, but everyone who takes the stance against this kind of bullshit aimed to do nothing but take advantage of people for as long as possible. This isn’t just a problem in gaming, but that’s a story for another time.

The gaming community is passionate. We spend our lives making memories that will last forever with friends we’ll (almost) never meet. We let hope get the best of us in the last few years, supporting projects that were already completely doomed to failure with the intention that just a little more money or time will suddenly help a developer get their act together and make good decisions.

For every developer that was successful using this tactic, that delivered on their promises and provided a solid product, there are dozens that preyed on the promise of better days to come. They took the money, didn’t change anything, and continued to fail on the hope that we, passionate gamers, cared and would continue to support them and their inability to learn from their mistakes.

This needs to stop. And articles like this, professionals writing about how developers using the forever beta tactic are essentially just liars and abusing the good will of the gaming community, go a long way.

As gamers, we deserve better. We always did.

Reader
Nathan Aldana

Its the same place its always been. The line is wherever the devs think they can push it to for maximum money from people who have bought into hype

Reader
rafael12104

Ah, Pepperzine right on the money as usual.

Ok, most here know this situation quite well, right? But here is what really pisses me off. No matter the state of the game, devs now quite liberally use the old “we will fix it at launch” excuse. And in many cases although you have paid close to full price if not more, you end up with a sub par product for weeks, months, maybe years. And even worse, devs can ignore your feedback to improve the game until that nebulous date of launch and you have to just grin and bear it.

There is no accountability any more. Ark is a great example of this. Fun game. Great in many ways but the bug squashing? Heh. “At launch, don’t you worry.” Content update didn’t go well? Shrug. It’s all part of the plan. Really?

So, from my perspective, a game is not launched until there is an actual launch date and it is met. Because at that point, the devs are completely accountable. No more bullshit.

I understand that bugs and issues are part of early access. That is fine, but with the prices and time games are in early access; with the opportunities and feedback devs recieve to improve the game before launch, it is really frustrating when they don’t. Betas are even worse in that sense. The last few Betas I have participated in launched with bugs we, the players, reported in Beta 1, 2 and Open.

Screw this. I’ll wait until launch, thank you. And it better be a good launch, damn it! Games are in early access long enough these days.

Reader
Robert Mann

I… have two viewpoints on testing with people. On the one hand, for developers who are on smaller teams, with lower funding, and who don’t want to get a publisher with inane demands on release schedule a year before a game is actually ready? I think getting fans to test is a great idea. Where I don’t want to say they should charge, I do want to note that some limiting factor for people coming in needs to apply in many cases for online games, since servers will only handle so many people prior to a lot of the optimizations that will come as time passes. So that one I think is reasonable for them, and I’ll consider it (and even give a delay until such time as they are further progressed for me to call it launched.)

On the other hand, the moment some big studio or game with a publisher that has massive funding to work with does anything where people outside testers that are hired for the job gain access? That’s launched, and it better be essentially ready even if a little rough. They don’t have any reason to need the help.

Reader
Utakata

You know I was going to say something, until Mr. Eliot made me realize how much game developers where moving goal posts and stringing us along with this:

“The point here is not that the game is persistent and free of wipes; the point is that you can access the game on the regular and play. H1Z1? Launched. Star Citizen? Launched. Shroud of the Avatar? Launched ages ago. Any ‘beta’ game with a cash shop has launched, even if there’s an upcoming server wipe and you’ll be refunded your money. The game has launched and should be judged as such. Specifically, I’m taking aim at games that tend to get away with ‘early access’ as a marker; the launch date is when that access opens.”

…now my pigtails and I have nothing more to add to the matter. o.O

Polyanna
Reader
Patreon Donor
Kickstarter Donor
Loyal Patron
Polyanna

If anyone who pays you some money can play your game, then it is a released game. Regardless what label you slap on it, if you’re charging money to play it, for anyone who wants to pay, then you are selling a game.

If it’s half-assed, half-done, low effort hot garbage, but you’re letting people pay you to play it, then it’s half-assed, half-done, low effort released hot garbage.

Everything else is just so much marketing bullshit.

Reader
Loyal Patron
Kickstarter Donor
Patreon Donor
wonder_llama

Looking at the situation from the other side of the fence, it’s really sad to see how this time-honored tradition has been sullied over the years. Being a beta tester used to mean you were entering into a mutually beneficial agreement with the game developers where they gained an unpaid army of QA-testers, and you gained an opportunity to try out the latest new game before anyone else got to see it, possibly even influencing the game’s development with your feedback. On the one hand, you got to try out all the cool rides in the theme-park for free before it opened. On the other hand you were expected to keep your crash-helmet handy at all times and not complain too loudly if you had to use it.

Real beta testing still has a place in the modern game development process, so I thought it might be helpful to start a (not entirely tongue-in-cheek) list to remind everyone what a real beta event looks like. Here’s the first ten things I came up with.

It might be a real beta if…

1. You were asked to sign an NDA.
2. You were asked what other games you’ve ever beta tested.
3. You were provided concise instructions about how to fill out a bug report.
4. You weren’t selected.
5. You didn’t pay money for access.
6. You were expected to provide feedback, and you were given explicit instructions how to do so.
7. You were denied access because you weren’t providing feedback.
8. Some of your feedback was actually implemented in the full release of the game.
9. All your characters were wiped prior to full release.
10. There were several beta test dates announced, each separated by at least two weeks of down-time.

Even “open beta” still serves a purpose when it’s done right. MMO’s are after all just ridiculously huge, hyper-complicated machines. There comes a point towards the end of development where several several of your project
milestones read something like, “this SHOULD work.” And you need to verify that the systems and processes you’ve tested with 30 players will still work correctly with 300 or 3000 players. A real open beta is used to stress test the game and get in some last minute fine-tuning before the game goes live. A game that had its release date pushed back because of the results in open beta is using the process correctly. A game that offers more than one open beta event with a couple weeks or more of down-time in between (to address the bugs) is using the process correctly. A game that announced a firm release date and allowed a select few players to play the game a week early is just offering early access, especially when the “select few” paid money for the privilege.

Reader
the_balance

Heh, I can actually hear Jeff Foxworthy rattling off a list like that in the form of his old, “You might be a recneck,” routine.

“Did you sign an NDA? Youuuuu might be playing a beta.”

It needs some work to flow a little better, but you get the idea!

Reader
Loyal Patron
Kickstarter Donor
Patreon Donor
kgptzac

Pretty neat write up. Unfortunately the exact definition of beta software doesn’t really improve their quality, and the real loss is for those players who seriously want to trade their leisure time to help a product to succeed.

Reader
rafterman

Seems like those days are gone. Developers have figuered out they can make more money before release than after, or at least as much, by making people pay to help them get their game working.

Reader
Kickstarter Donor
squid

But you don’t need to insult your own game by releasing it and saying, “Don’t judge it yet, just pay for it while we fix it!”

Or, in the case of games with add-ons, just pay for it while you fix [parts of] it.