Disclaimer: I'm going to brag a bit. You know, just to get it out of my system. And then I promise to move on to what I'm really trying to say. So if you don't like bragging farts, stop reading here and feel free to skip down to the next section if it pleases you. :) So I've been playing MMOs for about 12 years now. I do not have enough fingers on my hands to count the number of closed betas I've been in. I consider myself lucky to be invited to test so many MMOs. Today, I don't do much beta testing. I sometimes miss it, at other times I'm glad I have the free time to do other things.
Cannith Crafting: A Live BETA
So when Cannith Crafting was put on the live servers in a BETA format, it was the first time I've heard of a MMO subsystem being tested in such a fashion. It's highly unorthodox to release sub-systems into the live arena, but in the case of the Cannith crafting system I can understand why, what they had to gain (and potentially lose) from doing it. There are also some interesting unexpected results from using this method of "live" beta testing.
The Reason: Not Enough Testers
In huge games like WoW (nearly 12 million Players) or even Aion (3 million players), the response to a beta-testing call-to-arms is astounding. Typically a percent of the player base are also closed beta testers and able to see content months in advance (but only after rigorous NDA agreements are signed). If we assume that only one half of a percent (0.5%) MMORPG's population are close beta testers that would be nearly 60,000 beta players for a game like WoW. (
1)
Last year, MMOData.net reported a total of around 110,000 active DDO players. If we make an assumption that of that population less than 1% are beta testers; that amounts to around 600 to 800 closed beta players. In my opinion, that's hardly enough hands to test a complicated sub-system such as new crafting system
There is a saying in the game business: "No game survives contact with the players, but that's how you find out what kind of game you've got." Today there are Open Beta servers; even WoW had one for Cataclysm. DDO has its own, called Lamannia. Open Beta testing tries to solve the "not enough" people issue. But in most cases, I feel it's just a PR stunt to get players hyped on the new stuff. In the case of DDO and Lamannia its been proven to be more than a PR gimmick, it works and Lamannia testers have really proven their worth and even stopped a bad update from being pushed.
Open Beta servers work great in practice with bug fixes, and even with testing new content like quests and adventure areas. But with new sub-systems with their own mechanics that may affect different areas of the game such as the loot system or the in-game economy, I think that it's a different ball game.
And here is where releasing a beta sub-system to a live environment comes in. My best educated guess (and opinion) is that Turbine took a pretty huge risk with Cannith Crafting by putting out a Beta out on live. Why? So that they could get more players into it and to see what happens. It's not an every-day-thing to allow EVERYONE a chance to influence a game in such a risky and public manner.
Reward and Risk
Live Beta's can turn out to be a wonderful thing, and I've seen this type of mechanics testing done in not in MMORPGs, but in regular table-top RPGs (namely Pathfinder) where new character classes, spells, and feats are given in an open beta format for players to test. Paizo was able to produce an awesome supplement to their game because they had awesome constructive feedback from their fans. The reward was an award-winning game both beneficial for players and designers. There are also more people buying the game and new players being introduced to Pathfinder.
I can also see how Live Betas can burn a game to the ground. DDO players invest a lot of time into a game by questing, killing monsters, exploring, and now crafting. By putting a beta crafting system on a live server, you run the risk of wasting a player's time. If the system changes or is bugged in some fashion that effectively ruins their hard work, then the risk is a player not playing the game anymore.
Worse than losing a player from your game, is that you run the risk of losing the player's trust for any game you run. Players expect consistency, so that they can learn, adapt, and master the game. What do you think would happen if a Dungeon Master kept on changing the rules of their D&D campaign game every other encounter? Players would get angry and leave the gaming table, because they can't trust the DM to be fair. Even if the DM said "Wait a minute! That was only a practice run!" Players still would be reluctant to trust the DM, and the chances of them playing in another game with that DM goes down.
Unsettling Effects: Useless Feedback
Back in the day when a closed beta invite meant something, betas also had (and still have) feedback forums. These beta forums were used for remarking on things that weren't necessarily bugs, but on things that might not be as good in practice as they seemed to be in design. At worst, you were working as a real tester by helping to squash bugs without the pay. At best, you'd have an influence on shaping the game itself.
I have been observing a frightening trend with open beta testers: we now have a "testing" community that believes there's nothing more to beta-testing than downloading the client and playing for a bit. After playing on the open beta server, they then spend most of their time on the forums or in the global channels complaining about how a game in development still has bugs, and all done without filing a bug report. This kind of attitude sends shivers down my spine.
In a live beta, this effect (understandably) seems to be amplified. I see a lot more useless commentary. To be fair, I can't tell how many crafting bugs were reported since it was pushed to live; my only hope that it's more than what we would see in Lamannia or in the DDO's closed beta, Mournlands. Perhaps, there needs to be more advertising on how to find bugs and how to report bugs - it's not like players are given a list of what to test for. Or perhaps there needs to be more incentive than just personal enjoyment of experiencing a live beta? I would love to see an open beta player spotlight given to the person who found the best bug.
Summary
I want to see this kind of testing system work, but I have to admit that I wouldn't want to see certain kinds of changes nor a beta version of a new kind of combat mechanic on live. The Cannith Crafting Beta on live is okay, only because people can opt out of the majority of it by not crafting (with minor intrusions found in the game economy, and the tide of available players). (
2)
The reasons why a beta of Cannith Crafting was put on the live servers is understandable: the more people you have testing the more bug reports, and feedback on overall game effects you get, and sooner. Turbine played a very interesting game of risk, and it's yet to be seen if the rewards sought will be got: Will turbine get more players? Or even retain the current player base? I think we'll only see after it's been a while from when Cannith Crafting is finished. And though I don't like the trends of how beta testing and feedback is performed on the community's part, there isn't much that can be done when open betas, preview servers, and "live" betas are becoming common way of testing games. My only hope is that this method of "live" beta testing works in beneficial ways, such as getting new players, or retaining players that would have left without even a hint of crafting.
1 "Subscription and Active Accounts with a Peak between 1m and 12m."
MMOData.net. http://www.mmodata.net/
2 Regardless of it being Live Beta or not, Cannith Crafting has and still will affect non-participants in surprisingly direct ways, such as change in economy and the buying and selling of goods either from Auction House or Brokers. It's been interesting to watch both money supply and the supply of goods go down. And then watching both money supply and supply of available goes go up when crafting was shut down.
Cannith Crafting has some indirect ways influencing the player base: tide of available players for questing. The tide is in when there is a pool of players ready and available to join groups for questing. The tide goes out when players are crafting; simply put they cannot quest when they are crafting. In my guild, the flow of when players are available to quest has changed, typically after 10 pm (pacific) there are less players willing to quest because they are crafting.
