EVE Evolved: How mass surveillance is improving EVE Online

When developers get on stage at an event and start talking about things like big data, artificial intelligence, and cloud solutions, it often sounds like bullshit designed to attract investors who think technology is magic. But on a frozen rock in the middle of the north sea, Icelandic game studio CCP Games seems to be genuinely using these technologies to make huge improvements to the way it develops EVE Online. This year the studio took an interesting new data-driven approach to validating game design ideas, and we saw the results in the Onslaught expansion.

EVE is without a doubt one of the most complex MMOs on the planet, a virtual universe inhabited by several hundred thousand players engaged in territorial conflict and economic behaviour that often mirrors the real world in startling detail. Analysing that universe and the behaviour of players is no small task, and developer CCP Games relies on feedback from players and groups such as the democratically elected Council of Stellar Management to help figure out problems that need to be tackled. Now the developers have added deep data analysis tools and even neural networks to their arsenal, with some interesting results.

In this edition of EVE Evolved, I look at how CCP Games has been using data collection to aid development.

Getting it right

One of the big problems with balancing a complex game like EVE Online is that it’s often impossible to tell if player complaints are real and representative of a common case. Players have been known to lobby developers for changes that would help their alliance, and they’ve created organised campaigns against anything that threatens their way of life. Some players really don’t know where the line between EVE and real life is and will try to draw devs into their propaganda metagame.

The player-elected Council of Stellar Management helps cut through that to some extent by advising CCP on issues that players think are important, and user stories from exit surveys and other sources help too, but validating those views with facts can be difficult. Even if virtually all players would agree on an issue such as the war declaration system being broken, it can still be difficult to conclusively prove the negative consequences of wars and even harder to judge how different changes to the system would affect the game. What’s been missing from this conversation over the years is cold, hard facts.

“We’ve been really good with opinions and feelings, but now facts are creeping in in a really good way. What you see there on Abyssal Deadspace and Wardecs, and what we did on jump fatigue earlier in the year, these are elements where we’ve been able to make a much clearer picture of what the problem is and then tackle it more deliberately than before.

A combination of interviewing people on an individual basis who are involved in particular areas, getting overall survey consensus of where the larger group is, and then getting those concrete facts gives you a much better picture than trying to overly bias any one of those inputs.” – Hilmar Veigar Pétursson at EVE Vegas 2018

Mass surveillance

Part of the reason these problems have been so hard to solve throughout EVE‘s lifetime is the lack of data available on many of the actions players take in-game. When you have over 35,000 people online at the same time in a single shardless star cluster, that’s a lot of actions being taken every second. Logging literally everything that happens in-game would have ruined server performance, so historically EVE has discarded most of that data and developers have been very selective with what information they keep.

This is where CCP’s new granular data collection tools come in, which were added to the EVE Online server back-end with the Into the Abyss expansion. CCP now has a low-impact way to push information on a huge range of actions taken in EVE out to an elastic cloud service, providing an insane level of detail on everything players do. Before she left for a job at Unity Technologies, former Executive Producer Andie ‘CCP Seagull’ Nordgren referred to this jokingly as “mass surveillance,” but it does solve that lack of data problem the studio has always had.

“This data has proven to be super super useful, way more useful than we originally thought. That plus we did a lot of player surveys and interviews, and it gave us a lot of confidence that these are the right next steps. Also it helped us when we started play-testing co-op Abyssal Deadspace.” – CCP Burger

Interrogating the data

Collecting granular data is only one part of the puzzle for CCP though, as getting any useful insights from such a huge and unwieldy dataset is a legitimate challenge and a target of ongoing research. Interrogating the stored player information for meaningful answers has become a big data problem, and the tools for solving those problems include everything from standard statistical methods to self-learning neural networks. A new team involving two students from the University of Reykjavik was given carte blanche to work on this data, and is doing some impressive work.

CCP Rise gave some hints as to how this granular data can be used to influence design decisions in his presentation on ships and balance at EVE Vegas 2018. In the EVE Keynote (below) he also showed a series of detailed graphs on Abyssal Deadspace completion rates that painted a clear picture of the range of ship fittings that this PvE mode works for. This went on to inform design changes to Abyssal Deadspace, including the new PvP arena and co-op frigate sites. CCP Rise also described how developers discovered that combining metrics such as number of jumps in a particular ship turned out to serve as good surrogate for a ship’s relative power in PvP next to its peers, which helped them validate that assault frigates needed to be rebalanced.

What does it mean for EVE?

While the average EVE player might not be interested in how CCP collects data behind the scenes or what wacky technologies and methods are being used to analyse it, I think all players will feel the results of this new approach. This year I wrote several development speculation articles on topics such as ways that abyssal deadspace might be expanded on and how to fix the war declaration system. Practically every item in my development wishlist was confirmed at Vegas, and perhaps that’s no coincidence.

When I make a prediction about what’s coming for EVE or write about what I’d like to see in the future, I’m making a case for the next logical step that I see as a player or for a change I think is needed. This doesn’t always align well with CCP’s own design decisions, and I think the fact that it did this time is a good omen. CCP’s new data tools may be helping developers to more quickly and conclusively validate potential plans with live data, and if that’s the case then we can hopefully look forward to more common sense iterations in future.

Fans didn’t seem too interested when CCP first discussed its new granular data collection tools, but it may turn out to be one of the most important features of the past several years. By performing detailed mass surveillance on the complex world of EVE Online, CCP is now better able to validate its game design decisions and separate fact from fiction in user stories and interviews.

When I asked CCP Burger at Vegas about the ways they’re using this new data, he said “It’s amazing, you wouldn’t believe.” But after seeing CCP Rise’s abyssal deadspace graphs and seeing how the Onslaught expansion was developed on the back of that — Trust me, I believe.