Just before the news about the state litigation against Activision-Blizzard broke last month, Senior Software Engineer Joseph Cochron and Principal Software Engineer Kurtis McCathern gave a talk at GDC 2021 called War Stories from Azeroth, discussing some of the technical issues the team’s struggled with from The Gates of Ahn’Qiraji event to Warlords of Draenor. While some of it was tech-heavy, a few of the problems and solutions discussed were fairly accessible for gamers who aren’t software engineers. Many of the topics involved well-known events from World of Warcraft’s heyday, and some readers may have had firsthand experience with some of the more blunt “solutions.”
Let’s take a walk down memory lane as we rip off those nostalgia goggles and take a good hard look at the road many of us carelessly tread.
The gates of lag
For those who don’t remember, the opening of the Gates of Ahn’Qiraj were kind of a big deal in the original WoW. It was a one-time event per server, asking both factions to put together massive amounts of materials and embark on an epic quest that would grant one person a special black bug mount and the title of Scarab Lord. It also awakened a few bosses in the zone before allowing the server access to the Ahn’Qiraj raids.
The problem was that the servers couldn’t handle it. Remember, the event took place before the popularization of streaming sites like Twitch, and even YouTube videos at the time were less than common, further driving up the desire to be at the actual event. As thousands of players would go to the zone, and because coalescing, multiple world copies, and the ability to put limits on the Area of Interest distance didn’t exist at the time, the servers crashed – which only made it so people wanted to go more.
This led to “tourists,” players from other servers who would go to a server they had heard had been opening the gates, who would make their way to the gates. With few options, gamemasters logged in and would teleport players, mostly the tourists, away from the area in order to help fight the lag, but it wasn’t enough. Players themselves would try to schedule their server’s opening for off-hours so as not to crash the server for everyone, but even our own readers have commented on how this caused drama.
Fast forward to 2020. The gate event hits test servers and is still super laggy. The devs learn a few tricks. This time, the studios had the servers stop tracking player movement and direction facing updates, among other technical changes. While it’s still laggy, you can visually see how much things improved from the crowd sizes. As McCathern noted, the 2006 pictures tend to have about 60-120 players able to participate in the event, while the 2020 crowd has “almost the entire population that’s logged into the realm.”
Going forward, Blizzard worked at dispersing the crowds. Events spread out more. The Zombie Plague prior to Wrath’s launch may have made people unhappy due to its design, but spreading the plague also spread players around. WoW vets also probably remember that Wrath of the Lich King and Cataclsym had players going into one of two introductory zones of their choosing, but it wasn’t good enough. Pandaria tried pushing players on opposite sides of the continent based on faction and slowly moving them together, but there were still issues.
Quest blockers, paradoxes, and achievement floods
When Warlords of Draenor came out, Blizzard decided to go back to a single intro area but came equipped with a few changes. This time, the aim was to move players out of the area rather than try to have them bump into each other for organic meetings. The game also had multiple zone copies. However, once through the area and back into the single-copy-zones, there were still “quest blockers” – objects or mobs that people needed to interact with/kill before progressing forward, which created progression chokepoints. The virtual line to interact with these targets slowed down progression, which allowed other players to catch up and then also clog things up.
As McCathern says, it was clear that the team was trying to fix an engineering problem with design, and it wasn’t working. Copies of the entire world seemed the best fix, even if it might create paradoxes, such as seeing an NPC both dead and alive in the same spot. The team went ahead, and it’s clearly made things easier on everyone since then. While the company was trying to make the servers smaller and asking players to simply wait while less patient players played through, issues remained. Engineers had work to do, but not all problems are about player crowds.
Cochron noted that the early tests for the game’s Achievement system was equally brutal. In fact, pre-Wrath players can actually go back and check early achievement dates, such as their level 10, 20, 30, etc. ones. Some may notice that despite being level capped when the expansion released, they received retroactive achievements sometimes days later. This was what the solutions the team came up with to put less pressure on the system. As you can see from the picture, the theoretical deluge of achievements players could have received on day one could have made things much worse, and I’m sure many of us already recall the expansion being fairly laggy.
At launch, WoW had been a read-heavy game, meaning there’s a lot of data fetching as opposed to storing. Achievements suddenly made the game write-heavy, putting a sudden strain on the servers. But the feature was deemed important. It opened up new goals for all players, additional ways to play, and also provided data on player actions and behaviors. The problem also wasn’t one they could simply throw money at, hence the staggered achievements. The team also changed the way data was being saved. Rather than everything being saved at once as it happened, the team switched to collecting batches of data in “heartbeats.” This was a bit of a problem when there was a bug causing each beat to be recorded three times, but once Cochran helped clear that up, it helped things move forward.
The team’s expanded since then. One thing is that devs have their own “Dark Realm” test server, where they can move their live server character over. Yes, you can test things out by dropping in a new character and beefing them up to be level/gear appropriate for the content you’re testing, but using a pre-existing character can help devs see or feel out when something is amiss when current content collides with upcoming content. Though the game is seeing some struggles right now, the team notes that each expansion strives to add something new instead of “more of the same.” Ideally, the company will be able to make use of the lessons it’s learned on the server tech side of things when launching a future expansion that gets players excited about visiting Azeroth again – whenever that might be.
More coverage from GDC’s digital event earlier this summer:
