The Lab Rats: Your Role In Open Betas Explained

Activision’s multiplayer behemoth Call of Duty: Modern Warfare 2 was a smash hit right out of the gate, selling nearly five million copies at launch. This translates to five million users jumping into multiplayer lobbies on day one hoping to quickly find a match and start getting their money’s worth. Unfortunately for these early adopters, multiplayer sessions in Modern Warfare 2 went from a good time to a glitch fest in a few short weeks. First there was the javelin glitch. Then came the overpowered akimbo shotgun, care package knife deaths, wall and elevator glitches, unintended infinite ammo, and several more problems that plagued the otherwise impressive game.

While it’s tough to predict what will happen when a multiplayer title goes live, many of these bugs can be minimized, if not eliminated ahead of time, through proper testing procedures – a process Modern Warfare 2 surprisingly did without. With the increasing prevalence of open betas, gamers can play an active role in polishing the final product to avoid disappointment at launch.

Every step walked, shot fired, vehicle destroyed, power up chosen, and path taken during beta phases is tracked by developers to test for glitches and address balancing. Even if you refuse to participate in forums or lack the motivation to submit a 5,000-word essay expressing your concerns, your behaviors alone provide the data needed to make necessary tweaks. An open beta is not a feature-rich demo with a few blemishes; it’s an invitation to help create a better game.

“I’ll say it up front: If this is a demo of our game, this is the worst demo ever,” says Blizzard design director Dustin Browder in regards to the StarCraft II beta. “It’s not really meant to be a demo. It’s really meant for us to test our servers, test software stability, and of course, test the balance and gameplay experience.” This isn’t just true for StarCraft II, but for multiplayer titles all across the board that go through the beta process.

Top image: A heat map from MAG’s Absheron Refinery during one 256-player Domination round. Heat map images show the movement of players and vehicles. Areas shown in yellow and green are the most lightly traveled, areas in red are the heaviest trafficked on maps.

Next: How developers like Zipper Interactive (MAG) and Bungie (Halo: Reach) prep for betas and stress tests

Preparing for the onslaught
According to MAG producer Alan Van Slyke, there are two essential elements in preparing for a beta release. On the development side, coders must make sure the game is stable enough for proper public testing. Then there is the technical process, which must ensure the beta is ready for download, the patching is functional, the servers are available, and escalation procedures are in place in case of a major crash.

To prepare for the onslaught of everyday gamers when the beta opens, multiplayer titles undergo internal testing during alpha phases where developers bring in teams from other departments to play the games and provide feedback. In the case of Halo: Reach, Microsoft and Bungie employees tested the game through internal alpha and beta phases to check for major balance issues and address server stability prior to releasing a build to the public.

“The preparations began months in advance,” explains Bungie’s community director Brian Jarrard. “As much as this is a work-in-progress snapshot of the game, we recognize we’re putting this game out there and it’s going to be critiqued. People, to some degree, can be forgiving in acknowledging it’s a beta, but they also have expectations and they’re going to voice opinions about the game based on their experience.”

Even with the amount of testing prior to their release, public betas are far from perfect. If you’ve been part of a beta test before, you’ve probably seen your fair share of glitches and ­server ­crashes.

“Releasing a beta is like another release of the game except our standards are a little bit lower,” Blizzard’s Browder says. “We will knowingly go into a beta with things that are a little bit broken, but not much. Still, we want to hold our standards pretty high because it’s this trial run that helps us get the real release correct.” However, these precautionary measures aren’t ­always ­bulletproof.

The stress test
By simply jumping into a multiplayer beta, you’ve instantly become part of the stress testing that can contribute to a smooth, crash-free launch when the game releases at retail. If a game becomes a hit, the servers need to be able to handle millions of gamers jumping into the multiplayer lobbies at once. The only way to test if you’re up to snuff is to subject your system to a real-world deluge of players.

“There is no other way of working out how your game is going to react when millions of users are making decisions you can’t quite anticipate,” observes Ged Talbot, lead designer for Blur. “The larger the group of people, the more chance, you’ll find problems wherever they may lie.”

When the Halo: Reach beta opened to the public last May, Bungie expected several million users. The team worked to prepare server clusters for this volume using the infrastructure already established with Halo 3. Despite preparations, the servers were unable to handle the jump from zero to one million users in a short period of time, which caused the beta to crash.

“That was exactly the kind of test we could only accomplish in a real beta of this scale: Do as much as we can to emulate what would happen during a real retail release and get a huge number of data and simultaneous connections that really stress all the servers,” Jarrard explains. Bungie figured out the snafu and got the Reach beta up and running again fairly quickly, eventually reaching 2.7 million unique beta players in just over two weeks.

For MAG, the first-person shooter centered on 256-player online battles, stress testing was invaluable to the beta process. During the alpha phase, the team at Zipper Interactive built a system that simulated real players to ensure server stability, and confirmed enough AI bots were able to log in to reach the desired size and scale at the beta’s launch. However, Van Slyke says this is merely an engineering solution.

“We can’t predict how players are going to play,” he remarks. “That’s where the open beta comes into play.”

Top image: MAG's killmap images show the locations of players when killed and location of the killer as noted by a connecting line. Red dots represent dead players, blue dots represent killers. These images identify spawn camping issues and deadlines of certain chokepoints, objective areas, bunkers, etc. This could also suggest sightline problems on maps.

Next: You're being watched. Find out how you play a development role in open betas

You’re being watched
During online betas, servers collect tons of information for each unique user. This intel helps developers assess and address ­gameplay ­imbalances.

“It’s not so much like Uncle Sam watching everything you do,” says Kyle Zundel, producer on ModNation Racers. “It’s more observing output from errors, and a lot of these errors can only be caused under real-world conditions with hundreds of people doing different things at the ­same ­time.”

During the heavy traffic periods, developers watch a wide range of player behaviors to detect balance issues. In the case of shooters, the weapons chosen, number of shots fired, and kills per weapon are often tracked to check for imbalances in weapon power. If one weapon rises as the predominant choice for most users and is producing an overwhelming number of kills, this suggests that the weapon may be overpowered and might require tweaking.

In racing titles, dev teams track player locations and car pathing to produce collision maps that show where players crash most often. A clump of red data points in a given section on a map may be indicative of a flaw in track design.

In competitive multiplayer titles across the board, developers also track total matches played, win-loss percentages, and the performance of matchmaking systems. Internet connection types are examined to confirm the systems are pairing similar connections together for an optimal experience. Bungie even tracks how often players use in-game muting to help users find like-minded teammates.

All told, developers collect terabytes of data to produce stats, graphs, and heat maps. With such a high volume of data flowing to the servers, each developer has its own unique way of funneling the data to the design teams. Some developers build servers that place all the gathered information into a database that designers can search for specific scenarios. If a designer needs a heat map for a specific location where players die the most, they’re only a query away from receiving one. Data is constantly collected to provide up-to-date reports that can be used by teams to fuel the baseline discussion on which bugs need to be addressed. This data is further complemented by direct ­user ­feedback.

Your voice matters
During multiplayer betas, development teams also scan forums and message boards for user feedback on perceived bugs and other gameplay grievances. Community coordinators comb through and aggregate the information to ensure these suggestions go through the proper channels and reach the right people.

“[The beta process] is really the first time we can get a heartbeat on our potential consumer base,” explains Ramone Russell, community manager for ModNation Racers. “Consumers play the game differently than we do. It’s invaluable from a developer’s standpoint because that’s the only way we can get tons of raw data and make changes based on the things our beta ­testers ­find.”

When working on ModNation Racers, Russell took each user’s bug report and collaborated with the quality assurance team to check the game logs and see if the complaints were valid. Bungie does the same with Halo. If users are causing a ruckus in the forum about headshots not registering, Bungie works with quality assurance to confirm that these user claims are more than just complaints filed about intentional ­design ­decisions.

“There’s tons of little things like that that started out as anecdotal feedback, and we were like, ‘let’s just give them a few days to get used to it.’ After there was more noise about it and it became more consistent, we would eventually discover there’s actually a real issue,” ­Jarrard ­explains.
“At the end of the day it’s [the players’] game,” Russell says. “This is a time where your voice can be heard and changes that we make can be based on the feedback that you ­give ­us.”

Change you can believe in
Changes made to a game during a beta are primarily based on your feedback and behaviors, so next time you have access to a beta key, get chosen out of a pool of potential testers, or unlock a beta with another game you already own, you can do your part to help make a better gameplay experience.

“We want to know what you think about the game. We want to know the good, the bad, and the ugly. Most importantly we want to know what you didn’t like, what bugs you found so we can make those changes, and make the product better,” Russell says.

“Everybody wants a better game. When we run our betas properly and are able to gather the correct information and make those changes that improve the product, everybody wins.”

Top image: In this Collision Map, collision images represent where player vehicles hit other objects in Blur. This is used by designers to find out if more turn markers or lights need to be added to tracks.

Next: Betas by the numbers

Halo: Reach

Hours logged by beta testers: 16 million

Number of screenshots and videos uploaded: 1.4 million

Amount of upload data for said screenshots and videos: 1.3 terabytes

Number of kills during public beta: 1.1 billion

The most time spent by a single player in the beta: 198 hours

Approximate hours a day said player spent playing the game: 12

Number of kills with magnum: 142,657,966

Number of kills with DMR: 127,822,409

Number of kills with Needle Rifle: 49,341,380

Number of kills with the Flag: 8,927,112

Total forum posts created in the Halo: Reach beta forum on Bungie.net: 360,245

ModNation Racers

Total number of players: 122,000

Total number of creations: 78,000

Total number of games played: 226,000

Total number of transactions: 128,000,000