But what happens if the cheating is unintentional? What happens when the bug is so ingrained into the system that even casual, lapsed players accidentally took advantage of it just by returning to the game? How would you react if, shortly after resubscribing to a game, you had items or experience points taken and had your account suspended or banned? These are the things CCP Games’ Senior Project Lead of Player Experience David Einarsson had to deal with when tackling the ghost training bug.
How the bug came to be
While Einarsson spent a good chunk of his time during his GDC talk trying to explain EVE to those unfamiliar with the title, I’m hoping I can skip most of that for sitegoers. Essentially, the bug started when the game moved to a free to play model. The labor system (offline skilling) changed depending on whether or not the player was a free-to-play person or a subscriber. The system checked your status when you logged out, and checked it again when you logged in. No big deal.
Except for the fact that, between that time, your subscription could lapse. You could go six months without playing or subscribing, decide you want to come back, subscribe, log in, and get skill gains equivalent to what a six-month subscriber would have. Now add in the potential abuse when creating multiple free to play accounts, subscribing for a month, leaving the account dormant for awhile, and then paying a subscription fee before logging in to get all the benefits at a fraction of the cost.
In most games, already you’ve created a great way to sell pre-leveled accounts. Except that, well, EVE isn’t most games. EVE has “skill extractors,” devices that remove skill points in certain denominations that let you take your earned skills and sell them on the open market for ISK. Again, remember that ISK can be traded for game time (PLEX(, which is how we get estimates on its real-world value. This bug, when combined with “skill farms” – dummy characters to draw skills out of to sell for a profit on the open market – could make a character very rich and/or very powerful.
While skill farms could already be a thing with the free-to-play option, the progress is much more time-intensive without the bug, and potentially more costly if you’re going to subscribe to the game. As EVE‘s economy is complex enough that CCP needed an actual economist on the team, the ghost training bug posed a very real threat to the game.
While the devs had gotten reports of it as early as November 2016, they didn’t know how easily it could be replicated. However, in May 2017, the method was revealed on social media. Players were furious. It created a serious PR issue. The team tried to fix it in June 2017, but actually made the situation worse. The way the bug previously worked was this:
- Subscribe
- Queue the skills to train
- Stop playing (the game records your status as a subscriber)
- Stop subscribing and don’t log in for a few months
- Subscribe
- Log in with bonuses you should only have gotten as a full time subscriber (the game sees the subscriber status, never noticed the subscription cancellation, and assumes it’s been there the whole time).
What CCP did was automatically set the system to forcibly pause the skill training for potential. It didn’t fix when the game detected the subscriber status. However, the “fix” affected many more accounts than the studio anticipated. That means legit players may have had their accounts affected. Worse, it could lead to further accidental exploitation by giving people a reason to come back (subscriber or not) and possible once again trigger the bug. Once the team realized the situation, Einarsson knew CCP had to tackle the problem a different way.
Dealing with cheaters and Plan A
Again, the world of EVE is pretty cutthroat. I’d hazard to guess the outside world would assume tons of players would have abused the system. CCP assumed the same, and Einarsson constructed a plan based on this assumption.
The first part of the plan was to not simply dump the problem onto AI. The devs had already screwed up once by automatically trying to fix the bug, so they’d use the AI to gather the data and go through everything else by hand. While some companies may have been willing to gamble, Einarsson was not. He felt responsible since the team had already let down players. He wanted to protect the integrity of the game and felt CCP’s role within their universe was as space janitors there to keep things running and clean up the messes.
So Einarsson outlined an investigation. First, there would be data collection. The data would be used to create a list of who abused the bug. CCP knew thousands of accounts affected had been affected and assumed that people did it on purpose, so the company would take punitive action.
However, just in case, they would look for intentional abuse and then follow the money. This could be done by checking skill points extraction logs and assuming innocents wouldn’t be using them. If it seemed there was an unintentional use of the bug, or the character hadn’t used an extractor, the policy was to “observe and report.” Einarsson didn’t want to compound the PR problem. Leniency would be given to those who turned themselves in, and CCP announced this opportunity to the playerbase.
From there, offending farm accounts would perma-banned. Main accounts would be suspended based on their past history of abuse. ISK and assets would be confiscated from the offender’s main account based on abuse profits. Einarsson admits that they had no plan for unintentional abuse.
During the first few days after rolling out the plan, CCP took no action. Again, it was observe and report. And observe they did. The findings were far from what the team was expecting. In fact, Einarsson said that their expectations were flat-out wrong: The majority of the affected players had benefitted unintentionally. Worse, more abuse had actually happened because of the original fix. CCP wasn’t seeing abuse from huge skill farms but from regular players.
The story couldn’t end here, though. The benefit of the bug was far too great to ignore. Plan A had to be tossed out the window. Einarsson and the team had to do something different.
Plan B: Work with the players
As I previously mentioned, CCP had warned players who intentionally abused the bug that it had a list, but would be lenient to those who turned themselves in. One such player drastically helped CCP in determining how it wanted to tackle the problem. Einarsson didn’t share all the sordid details but did say the player had at least initially used the bug unintentionally. CCP and the player emailed each other back and forth, talking about how they could best fix the situation, with the player requesting a look at the data used to determine is ill-gotten gains.
While CCP initially had considered taking assets or ISK to fix the situation, the player had a unique idea: If CCP would give the player free extractors, the player could extract an amount of skill points about equal to what they’d unfairly gained, and CCP would take it and remove it from the economy. CCP learned so much from this player that it was used as a basis for dealing with the rest of the playerbase.
Again, most players were unaware that they’d been exploiting at all, so CCP decided to email those who unfairly benefitted from the bug. The were told how the bug worked, how that player benefitted, and value of their benefit. They aimed for an apologetic but informative tone.
From there, the player was given several options to help resolve the situation. First, the player could hand over CCP’s estimated ISK value or an equivalent in assets. They could also use skill extractors CCP would provide to remove the bugged abuse gains, just as the first player had. Finally, the player could mix and match the options (ISK, assets, and removing skills via skill extractors) to help fix the economic issue that had been caused.
Taking a note from the player the team’d initially spoken to, CCP offered data to any users who asked for it. To sweeten the deal, any extra skill points that wouldn’t fit in the extractors could be kept. For example, since the extractors can hold only 500,000 skill points, if you gained 510,000 from the bug, CCP would let you keep 10,000 skill points. This came out to about a 5-10% skill gain still on average.
Remember, this was for innocent players and/or those who turned themselves in. Bans were only for serious abusers.
The Results
So what? Maybe you’re thinking this is all a puff piece to praise CCP. Maybe you think it’s something that could happen only in EVE. However, for someone who just can’t get into EVE as a concept (too much mathcraft for me), hearing a studio do this is inspiring.
Let’s look at some of the numbers provided by Einarsson. First, most people were understanding. Eighty percent of the customer support surveys were positive, with 21% of the players receiving them responding. That means that not only were people willing to fill out the form after having to give something up, but they actually had a pleasurable customer experience while giving something up. I can’t even imagine how this would work in, say, a bank where you were accidentally given too much money and suddenly were asked to return it.
Giving players options to resolve the issue helped CCP, not just economically but PR-wise. People saw that the company wouldn’t just “shoot first,” and actually empowered their customers by allowing them to choose how to help solve the issue. In fact, some people even thanked CCP for their approach with personal letters.
In all, 6,036 characters owned by 119 players were reviewed. Of those, 796 accounts owned by 23 players were perma-suspended. That means about 80% of the players affected were innocent and/or willing to work with CCP to resolve the issue. Seven months later, CCP checked back those players. 57.85% of the investigated players were still playing EVE, and many were still subscribers as well.
Yes, clearly businesses should try to be transparent, honest, and make a good first impression when approaching a customer about a problem. We already know in our gut that it’s best to be less reactionary and more exploratory – to make human contact and not to just outsource the work to machines.
But this isn’t the norm. This was a company that went out of the way to do things right. Clearly, for the industry people in our audience, this might hopefully inspire them. But most of us are players. When you have a problem in the future – and it will happen and you will experience terrible customer service – in your “I’m taking my business elsewhere!” letter, maybe drop a link to CCP’s approach here. Show them the value of good customer service.
