Wisdom of Nym: Final Fantasy XIV’s lack of public testing is fine by me

    
11
Now fight for something!

When the Final Fantasy XIV media tour happens, it’ll be the first opportunity anyone outside of Square-Enix has gotten to look at anything related to Endwalker. This is a bit unusual. A lot of MMOs have public betas of some kind for their expansions, and even those without a full beta event for said expansion usually include some degree of opportunity to test out new content patches with actual players instead of just internal company representatives.

Back when I first started playing Final Fantasy XI, I had no real experience with MMOs (due to that being my first one and all) and thus had no real sense of public test servers being a thing. Over time, obviously, I grew accustomed to them being around… but then I started getting used to not having them around again for FFXIV. And quite frankly, looking at everything the game has accomplished over the past several years, I’m increasingly actually glad that we don’t have public testing outside of the occasional media event. I think it works out well.

First and foremost, I think it’s important to preface this particular analysis by noting that when you get right down to it, it’s pretty clear that FFXIV puts out some remarkably balanced and well-designed content without any kind of test server. While I think there are definitely some areas where there’s room to look at the setup and argue that the game could be richer with some public testing (which I disagree with, but we can put a pin in that), the fact remains that the simplest argument against public testing is that the game is working fine without it.

But let’s put that to one side for a moment. Let’s look at the advantages of lacking any sort of public testing, starting with the fact that new stuff is new for everyone at the same cadence.

There’s no impetus to test things, find optimal routes, and be ready to speedrun everything on the day of release because the first day of early access no one is going to know the fastest way to speedrun anything. Sure, you can make certain guesses, like how to level up as quickly as possible… but leveling in and of itself isn’t going to get you through the MSQ, so even that requires a little bit of re-evaluation. Your fastest way through the MSQ is to just do the quests as they come up, after all.

Similarly, new dungeons, raids of all varieties, trial bosses, beast tribes… all of this stuff is new for everyone at the same time. No one knows the best way to do things on the day of the launch. We’re all novices at the same time. That gives an equalizing effect and it prevents people from getting bored with the content before it’s even officially playable.

Slash!

Just as importantly, it keeps datamining to a minimum. While there is still stuff mined out around FFXIV (it’s discouraged but present), generally datamining is much more controlled and rarely involves figuring out things in advance of official notifications. Games with public test servers rarely enjoy the same grace period, with most players actively participating in the game’s news cycle knowing about changes, plot points, content mechanics, and so on long before anything new actually comes out.

It also means that tuning is based more consistently around expected average players rather than cutting-edge players, and tuning is further expected to be done based on the playerbase as a whole. This is an important and subtle distinction, but it ties to people not practicing or taking part in the content ahead of time.

Basically, in games wherein you’re using a public test server to collect data on how well players can clear things, you are naturally getting data based chiefly on the people who are devoted enough to start practicing on the content before it’s live. That means that if something is tuned fine for that particular group of players (who are usually quite dedicated to the game), it isn’t necessarily designed to be accessible to the majority of players. Getting your data without that imbalance means that everything has to be balanced around an expected average, and the adjustments that come later are based on observed reactions from the actual average playerbase.

Now, the issue of balance does actually come up in certain cases, like the launch state of Ninja in Shadowbringers. It’s an indisputable fact that Ninja was in a terrible place when the expansion started, and odds are with player feedback something could have been done about this ahead of time without having to make Ninja pretty useless for a while until the balance passes were done, right? After all, then the developers would have had information ahead of time.

Except, well… class balancing is hard. It’s hard to do it at the best of times, and having to translate a whole bunch of player feedback from multiple languages, sifting through all of it, and deciding which bits of feedback were accurate and which should be discarded doesn’t necessarily fix the problem. It’s definitely true that someone would have noticed that Ninja was a mess before launch; what’s not clear is whether or not people would rightly identify that just Ninja needed major tuning out of melee and whether or not the developers would be able to find the ultimate solutions with that feedback.

Based on how many people currently say Samurai is underpowered, my bet is “probably not.”

BEHOLD MY HUMAN LASER!

The other big thing that public testing can really do is serve as a means of hype, getting people excited for the next major bit of content ahead of release. However… even disregarding the fact that FFXIV has little trouble getting people hyped up for its next few releases well in advance (we’re all counting the days until Endwalker at this point), that hype is mostly confined to the people who are really active in the game’s community and with news, once again.

For that matter, the test server hype can work against the actual release. I talked before about getting bored with the patch before it had even come out, and that’s definitely an issue. Imagine if all of the patches for FFXIV were up for testing for a month before they launched, even if we pretend the MSQ was somehow disabled and it was just a matter of chain-running the new instance and raids and what-not on the test server. Wouldn’t you be kind of bored by the time it actually came out? Wouldn’t patch day feel less like a chance to get to see all of this in action and more of a sighing “finally” when all was said and done?

So, yeah… I don’t miss having public tests. Part of me does miss the energy of the game when it did have its beta events before the relaunch and wishes that something similar was going on for the next expansion… but another part of me is happy that by the time I’m really playing this content, I’m really playing it.

The media tour notwithstanding, I mean.

Feedback, as always, is welcome in the comments down below or via mail to eliot@massivelyop.com. Next week? You know we’re going to need to talk about that live letter, naturally.

The Nymian civilization hosted an immense amount of knowledge and learning, but so much of it has been lost to the people of Eorzea. That doesn’t stop Eliot Lefebvre from scrutinizing Final Fantasy XIV each week in Wisdom of Nym, hosting guides, discussion, and opinions without so much as a trace of rancor.
Advertisement

No posts to display

11
LEAVE A COMMENT

Please Login to comment
  Subscribe  
newest oldest most liked
Subscribe to:
Reader
Geo Kavu

Guess having a QA team that is not served the pizza leftovers of the development team while having drunk leads cubicle crawl and grope them really helps the quality of their work.. Who’d have thought..

Reader
Jo Watt

How else can a certain company pay it’s CEO 100’s of millions and top executives so much money if they actually have to hire and pay people to properly test and balance their products. You think these companies are making games so everyone can have fun?

Reader
wjowski

Whatever XIV is doing, it seems to be working well for them.

Reader
Jeremy Barnes

I don’t think public test realms themselves are a problem in and of themselves, but they almost always exist for bad reasons. Whether it’s lack of funds for proper testing, laziness, etc, etc.. they do a disservice to the game as a whole.

I think public test realms should not be used the way they are currently. I think they should be more “R&D” focused. Try out new, big bold changes to see how players react so the developers can use that information for a better product.

Kinda how ESO tested out healing changes on their live servers. Those changes didn’t really seem to work out for them, but that is the type of stuff I think a test server should be used for and it never should be up 24/7

Reader
L B

Player testing is fine…if you’re in beta. Outside of that? No.

Players should not be given any insight into what or how the game will develop into. Doing so just allows for the top players to gain an additional advantage. As we see with a certain MMORPG, Player testing creates a bad community.

Reader
Erika Do

As someone who used to do QA at an MMO company, I’ve never used a public test realm, because I was never interested in doing my job for free for another company no matter how much I loved the game. I’ve often wished they weren’t there and am happy they don’t exist in FFXIV for all the reasons you mentioned above. Especially in FFXIV where spoilers are a big deal and I love not knowing what I’m going to get until I’m in the released content seeing it for myself.

I used to play WoW, and constantly seeing in-depth articles on Wowhead about all the upcoming content and various changes to this and that going on on the PTR completely removed any sense of discovery once the patch actually hit. We already knew everything about any new dungeons or raids, the looks and stats of every drop, and had our route planned out for us ahead of time. FFXIV on the other hand doesn’t even tell us most of that information in the patch notes, we have to actually go into the game and see it for ourselves. I wouldn’t have it any other way.

Reader
somber_bliss

I agree with everything you said and have a question. Do you see these alpha and beta ptr type stuff as a threat to job security? My buddy is in qa and always, mostly jokingly says peoplr are taking his jerb.

My stance is, you can’t replace the quality of work a salaried employee can and will do compared to someone online for free. I’m sure there are super stars that exist in the world that do help out but i imagine most people’s ‘contributions’ are more of server capacity.

Reader
Erika Do

Not really. QA people get GM codes that let us spawn everything, teleport around, give ourselves EXP to get to max level and such to check everything extensively going down massive checklists, make sure none of the new content crashes the game due to one of the devs falling asleep on their keyboard and adding a few thousand extra blank spaces to a quest dialogue field. (Something that actually happened in a game I tested once!)

PTR is a different kind of testing, to give feedback from the perspective of real players going through real content. We rarely had time for that kind of exploratory testing because there was so much “does every item name in the game fit into this new UI window we’re adding” kind of stuff. There are so many combinations of items/spells/quests players can have in an MMO that it’s unrealistic to check it all.

Reader
kelathos .

Unlike a company that shall remain nameless, Square actually pays for a fully staffed QA, and their results speak for themselves. In this way, they can present a story in-game and not have it spoiled 3 months ahead of time. It’s pretty good. Others should learn from Square’s example.

Reader
IronSalamander8 .

I’m mixed on test realms. I don’t use them often, but was big into a test for Tankers back in CoH days as we were going through so many changes and needed to give feedback for my favorite AT being changed so much, but that was all mechanism testing, not content. I do not want to have story or other related things spoiled, when I do test, it’s for bugs, changes or to see if I want to buy the game or not. So in FF14’s case, I don’t really need a public test as I already like it, and haven’t heard any major changes for my favorite classes.

Also as Vincent Clark points out earlier, they have their own testers for bugs and balance, we shouldn’t be needed for that in the first place.

Reader
Vincent Clark

I’m definitely in the “no public testing” camp.

I find the companies that do public testing lack either the resources or will to properly promote their game so they use the public testing as a stand-in. And in too many cases, that backfires.

There is nothing worse than a company enticing you to test their game, you spend hours doing just that and provide feedback on several critical issues…only to see that feedback ignored/discarded and the update released buggy anyway (LotRO and WoW immediately come to mind).

Lastly, I’m paying to play a game. I expect the people making it to know what they are doing and have the resources to test/QA it before it’s released. They are the ones getting paid to do so, not me.

FFXIV promotes the hell out of their game. The last 2 months before the first week of early access is flooded with dare I say too much information about the expansion, so there is never a lack of hype going into that first week.