Path of Exile apologizes for last night’s outage and server rollback – here’s why it happened

    
15

Yesterday’s Path of Exile patch didn’t quite go to plan. The 3.7.4 update was intended to add the promised stackable Fossils feature as well as multiple bug fixes and tweaks. But it didn’t take long for Grinding Gear Games to bring it all to a screeching halt to roll back the database.

In its message to players overnight, GGG quashes the initial rumors that the downtime and subsequent rollback had anything to do with an exploit. In fact, it had to do with a database migration problem that wound up wiping Korean details from the database – very much not intended. At first, GGG rolled back just the account database, but then it became clear that caused problems with folks who were already busily stashing stacked fossils, so downtime and a full database became necessary.

“In total this incident led to 123 minutes of effective downtime including the 29 minutes of gameplay that was lost,” writes GGG, whose lead programmer Jonathan Rogers took responsibility for the original database error, failure of QA processes to catch it, and his own error in misjudging the impact on fossils.

“In the future, we will make sure to understand all of the repercussions of a rollback before making this call. We would still have had some downtime, but we would have prevented any player gameplay loss. This incident is not up to the standard of service that we want to provide to you, and I want to personally apologise for how it was handled. You should expect better from Grinding Gear Games and we will be doing everything we can to avoid this type of thing in the future.”

And this is why GGG keeps winning awards.

Source: Official site. Thanks, BalsBigBrother!
newest oldest most liked
Subscribe to:
ejester
Reader
ejester

break the entire game and ROLLback the entire database and offer no compensation for this? … doesn’t seem like an award winning company to me despite the authors fanboing of GGG.

was a laundry list of fuck ups that caused this and all we get is a sorry?

I understand that it’s impossible to prevent terrible shit from happening all the time, but when it DOES happen, you should be going out of your way to make it up to your community and not just shrugging and going about your business as usual.

Reader
Kickstarter Donor
Patreon Donor
Loyal Patron
BalsBigBrother

Oh come on it was what just over a couple of hours and they were back up again. That is not even close to the worst service for an online game and you think that deserves compensation, sheesh!

Reader
kalech

I would’ve expected compensation if the game had a monthly subscription, but not since its free. It was only 2 hours of downtime and 29 minutes of rollback. Sucky but hardly the end of the world. :|

Reader
Bryan Correll

offer no compensation

So what kind of compensation do people playing for free deserve?

Reader
Kickstarter Donor
Patreon Donor
Loyal Patron
Ashfyn Ninegold

I wouldn’t call an apology from the lead programmer taking personal responsibility a “shrug”.

ejester
Reader
ejester

lol listen, your comments are comical, but where to start…

1. The downtime isn’t the main issue, it’s the rollback. Rollbacks can be potentially devastating to a player. What if someone, during that time that was rolled back, got their first mirror drop, or maybe 5 mirrors from a card turn in (or heck even 5ex for that matter), what if they just beat the the atlas for the first time, unlocked the last something they needed, etc, etc, etc. The list can go on and on.

The point is, rollbacks have serious consequences for the player and they should always be compensated. always. This is a PERMANENT loss of player data. They are effectively wiping your save data for a period of time.

2. You keep implying because the game is “free to play”, that no one deserves anything. I mean, I don’t know about you, but I’m positive nothing is free to play. It ALWAYS costs SOMEONE, maybe not YOU, but I know I’ve definitely invested money in POE over the years and still do. Just because you’re cheap and can’t afford in game purchases to support development, doesn’t mean everyone else is.

3. Anyone playing, that was affected by the rollback, deserves compensation equally. I don’t know why I need to state this, but apparently I do.

4. Apologies are meaningless when its just words. Put your money where your mouth is and show your community you are sorry. These are virtual goods, they cost nothing to manufacture, store, or ship. It literally costs the company NOTHING to go “here’s some bonus double xp, increased loot chances, free mystery box, etc” whatever they decide, that coincides with the aforementioned apology for breaking the shit in the first place.

Games, especially free 2 play games, are notorious for how they abuse their playerbases. So why is it a terrible thing to hold devs responsible for their product?

They break stuff, they then cause a permanent loss of player data and you think they shouldn’t compensate people for this? why?

I rest my case.

Reader
Kickstarter Donor
Patreon Donor
Loyal Patron
BalsBigBrother

The roll back lost 29 minutes of actual game play and yes you are correct it is possible that you could have lost something in that time and that would suck for that person if that was the case.

However odds are really small and I think its fair to say that most folks just lost a bit of xp that they will have already recovered. To put in perspective I have been playing PoE since beta and never seen a mirror drop

To ask for compensation, yeah lets have a full 29 minutes of double xp or increased loot to make up for it, that is just as comical as you seem to think us folks who think this is no big deal.

Covynant001
Reader
Covynant001

Two things, updates in the middle of the US evening prime time suck, even more so when the devs screw the pooch as they did last night.

Equally annoying is their web site goes offline during updates so I sat for a couple of hours wondering what the heck was going on. (Caught up on some Netflix shows, one bonus)

Google searches brought up nothing, probably reported on some stupid social media site but …yeah…fk that.

So kudos for great communications? I’d rather get some free stuff.

;)

Reader
Castagere Shaikura

Didn’t this happen before with these guys?

Reader
Kickstarter Donor
Patreon Donor
Loyal Patron
BalsBigBrother

If only you had taken the time to look at their website then you would have seen their tweets stating what was happening. Yes the website is down but they show any posts they make on twitter about ongoing maintenance on the landing page that does stay up.

Like this one for example:

Maintenance happens when it happens someone will always be affected I guess it was the USA’s turn.

Reader
kalech

Stupid social media sites… You mean the sites they use to communicate with players? If you refuse to check the sites that the devs use to communicate downtime, then that’s on you. The info was readily available.

Reader
Tanek

Keeping players informed like this seems like it would be one of the most basic things to do when there are problems. I’m constantly amazed at how many games opt for silence or dismissive responses instead.

Reader
Kickstarter Donor
Patreon Donor
Loyal Patron
BalsBigBrother

Considering the number of rumours that were flying around even in the short time this all kicked off its nice that we can pin it down to a factual answer.

Reader
Tanek

Exactly. If the company does not provide the information, people will come up with their own explanations and most of those will probably be much worse than what actually happened. Leave those rumors to fly around too long and they become the playerbase’s accepted reality no matter what is said later.