Guild Wars 2 gives a full technical accounting of last year’s server rollbacks

More transparency like this, please

    
7

Guild Wars 2 studio ArenaNet has been the subject of many side-eyes over the last few years, amassing a reputation as a studio with a messy and opaque technical side and a messy and opaque leadership side – and the mass layoffs in 2019 didn’t help. Earlier this month, however, the company finally publicly admitted who’s been running the studio and Guild Wars 2 since Mike O’Brien left and Mike Zadorojny stealth-departed at the end of 2019. And now, we’ve gotten an unusually detailed technical dev blog from Platform Team Robert Neckorcuk, giving us even more hope that ArenaNet’s revitalization includes a reset for its public image and transparency.

Neckorcuk gives a full accounting of a particularly dark day in the game’s history for players back in May 2020. You might recall that was the month that the EU megaserver was rolled back and cost players a significant amount of cash and gameplay time, which for Guild Wars 2 was an exceedingly rare incident. ArenaNet then moved to compensate players, but that was a bit of a bungle, as players complained about the compensation gifts and the mixed incentives.

Neckorcuk digs much more deeply into what happened behind the scenes over those 20 hours, starting with the update the week before, a rogue database issue, and (no kidding) the drivers that all helped cause and compound the cascading disaster. He also details the whole process of how the studio identifies problems, solves them, gets the hamsters running again, and prevents the issues in the future. And you might have noticed, that kind of downtime hasn’t happened again.

“The most impactful change for our databases was to increase alerting on key database metrics, not just system metrics like CPU or hard drive space. For our live operations, we added a number of alerts into a third-party tool to improve our response time for future issues. And for general operations, we’ve improved the record-keeping of our AWS infrastructure, now tracking more than just the instance type. Our reports now include instance types, generation, drivers, and storage types. We built a common package to install on all new servers that includes specific driver versions. Any future migration plans will update this common package, ensuring that we don’t repeat this issue again. We have completed the migration for all the remaining database instances and more, providing higher performance for improved service. In the last fourteen months, we’ve recorded an uptime of 99.98%, with only five minor service interruptions impacting user log-ins.”

If this is the new ArenaNet… we like it.

Advertisement

No posts to display

7
LEAVE A COMMENT

Please Login to comment
  Subscribe  
newest oldest most liked
Subscribe to:
Reader
Hikari Kenzaki

That was an interesting and fun (and a bit of ‘I feel that pain’) read. I’ve certainly been in those situations in the heat of the moment you’re trying every solution you can think of, hoping one of them sticks. Usually, the solution is more people, trying more things until you figure out that one detail you’ve overlooked.

Technically speaking, the GW2 server system is a marvel. The fact that they can deploy patches and run two versions of the client side-by-side seamlessly is no minor achievement. We were actually talking about this the other day.

Reader
Aelfy Carcini

We have tons of hope. There’s a better GW2 in the horizon :D

Reader
styopa

Wow, that was exhaustive, detailed, honest, and damned interesting for me.

What a great message from ANet- both the superficial blog itself, and the strongly-positive, customer-forward ‘feeling’ such a deep-dive communicates to the players.

Nice job to him for writing it, nice job to him and his team at coping with the issue (honestly, the idea of a system running without fail since 2017 crapping out is even more terrifying than usual).

Thanks Anet. Good job all around.

Reader
Kickstarter Donor
Greaterdivinity

If this is the new ArenaNet… we like it.

If this is the start of them actually starting to open up to the community and communicate more, and especially more honestly/frankly, I’m super here for this new Anet.

Genuinely hope that this is a new chapter for them and they get a clear direction, clear leadership, clear communication, and some sort of consistent identity for themselves in and the game. It still blows my mind how uneven the studio still is despite 20 years of existence and a sequel that’s almost a decade old.

Get your groove back, Stella.

Reader
Bruno Brito

It’s amazing how Colin seems to have learned from his mistakes….i guess.

We shall see.

Reader
Kickstarter Donor
Greaterdivinity

I don’t know if we can say “who” is behind this change in attitude, but it’s welcome nonetheless.

Reader
Bruno Brito

I mean, the timing fits, so i’m giving him credit. He needs it :P

But anything that makes GW2 better is good in my book.