Is User Feedback as use as Big Data Predictions?

[SavinDwarf]

An interesting article appeared on Gamasutra discussing whether gaming companies would rely more on big data.

It has the advantage of ignoring the vocal minority and clinically focusing on what people are willing to pay for -- Big data doesn't even need to know why people might want something.

http://www.gamasutra.com/blogs/NickL...hy_ask_why.php

Big Data approaches to predicting behavior/needs/wants etc. is being used across many industries. Its definitely being used by some gaming companies. I don't know if Turbine uses Big Data approaches to predicting user responses to changes/content/etc. But whether they do or do not, do you think its a better approach that listening to the vocal minority on forums?

Or maybe a healthy mix of forums, focus groups, Big Data and user groups is a more balanced approach?

There is a great book on the subject:
http://big-data-book.com/

here is the book overview:
[quote]A revelatory exploration of the hottest trend in technology and the dramatic impact it will have on the economy, science, and society at large.

Which paint color is most likely to tell you that a used car is in good shape? How can officials identify the most dangerous New York City manholes before they explode? And how did Google searches predict the spread of the H1N1 flu outbreak?

The key to answering these questions, and many more, is big data. “Big data” refers to our burgeoning ability to crunch vast collections of information, analyze it instantly, and draw sometimes profoundly surprising conclusions from it. This emerging science can translate myriad phenomena—from the price of airline tickets to the text of millions of books—into searchable form, and uses our increasing computing power to unearth epiphanies that we never could have seen before. A revolution on par with the Internet or perhaps even the printing press, big data will change the way we think about business, health, politics, education, and innovation in the years to come. It also poses fresh threats, from the inevitable end of privacy as we know it to the prospect of being penalized for things we haven’t even done yet, based on big data’s ability to predict our future behavior.

[\quote]

[Sapience]

Or maybe a healthy mix of forums, focus groups, Big Data and user groups is a more balanced approach?

Not really much question here. We've always been very open and forthcoming that we take information from multiple sources including data, player feedback(this comes in many forms, not just forum posts), focus groups, and surveys.

[Yula_the_Mighty]

"Big Data" is the flavor of the month name for this process. Online game development houses have been using the "Big Data" concept for over 10 years. The first time I recall it being talked about was with Sony for EverQuest that launched in 1999. Due to the design of EQ, it is largely a forced grouping to make progress at reasonable speed. Sony was amazed at percentage time that the player base spent soloing in spite of the difficulties. Or standing around using the game as a Chat Room. There is a smaller amount of time spent "In a group killing stuff on a Raid" than Sony expected.

This usage of MMORPG game space is the reason games like Lotro focus heavily on solo. That is what people do. People play the game like they do real life. Most of the time we solo in real life. When we group we tend to group with friends and family (we call these folks kin in Lotro).

It believe most online game companies operate Turbine does. They use multiple sources to get the information that they need for their decision processes.

Online games have a rich supply of customer data since everything we do can be recorded on the game operators servers. All they have to do is keep the data. Use various programs to mine and look at the data. It will not tell you the Why? They can easily find out the What?

[EmyrSelyf]

[QUOTE=SavinDwarf;6833674]

Big data doesn't even need to know why people might want something.

QUOTE]

Then big data is making a terrible mistake..... In business you have to be able to plan ahead (or at least you try to). Data can predict something only if there is a trend already in progress. If you don't know the "why"s then you can't predict a response by the consumer to a shift in the nature of the business. If you can't predict their response, then you can't protect yourself or make plans to increase market share from the resulting chaos.

There's certainly a renewed push by business types to use metrics and data as the end-all decision making tool showing up recently in the media. If it makes customers click on an ad more often, why not use a horrible design? Things like that, however, ignore the longer term effects of the decisions. They are concerned with short term, immediate goals and they run the risk of destroying a company in the long run if there are no checks on their excesses.

Businesses have to have an understanding of the customer and the industry that goes beyond data.

[SavinDwarf]

Then big data is making a terrible mistake..... In business you have to be able to plan ahead (or at least you try to). Data can predict something only if there is a trend already in progress. If you don't know the "why"s then you can't predict a response by the consumer to a shift in the nature of the business. If you can't predict their response, then you can't protect yourself or make plans to increase market share from the resulting chaos.

.

I have little experience in the gaming industry, but I have lots of experience in other industries and have been Product Manager for products with user bases in excess of 5 million. Assuming the gaming industry follows similar patterns to the rest of consumer industry sectors:

1) The vocal minority is less than 1% of the user base. The other 99% vote with their check book.

2) The demographics of the vocal minority don't even come close to matching those of the 100% -- so their input is only useful when it comes to creative ideas and out of the box thinking. But you can't use it as a source of what to build/improve.

3) Random polls, unlike politics, tend to get somewhere between a 10x and 20x response rate from people in the vocal minority when compared to the overall 100%.

4) Random polls that exclude the known vocal minority still fail because it is estimated that the sub set that responds is still incorrectly weight by a factor of 3-5 to 1. Which makes the results useless and unpredictable.

5) Analyzing buying patterns gave pretty good results. Because of this the best way to find out of a new idea was worth going into mass production was to offer it to sub sets of the users and see how buying patterns changed.

We were doing this 30 years ago in any case where our customer base was over 10,000 to 25,000. If the gaming industry has come to the same conclusion then they use the boards for creative ideas and not for whether an idea/enhancement will be seen as an advantage to the total user community.

In the case of LOTRO we have 100,000+ players, and how many post on the board as a %, how many read as a %. I bet the industry standard metrics are not far off.

You can get the % to climb if people feel they are contributing -- such as a WIKI. But most MMORPG forums have the opposite effect. You need to be thick skinned to post an idea to these forums. It may be only a few that are the problem, and they may not even realize they are the problem. But for the most part player suggestions immediately get trashed by someone -- and not in a supportive and helpful way. Most people won't dream of making that second suggestion.

In my opinion, Turbine would be crazy to rely too much on the feedback from these boards, its unlikely t be representative of what the overall player base really wants.

[Yula_the_Mighty]

Then big data is making a terrible mistake..... In business you have to be able to plan ahead (or at least you try to). Data can predict something only if there is a trend already in progress. If you don't know the "why"s then you can't predict a response by the consumer to a shift in the nature of the business. If you can't predict their response, then you can't protect yourself or make plans to increase market share from the resulting chaos.

The concept of what we call data mining is to get accurate data on the what. You want to know what the customer is really doing. Not what a vocal 1% says the customer base is doing.

You use other methods to determine the
Why are they doing this?"
"Should we try to extinguish this behavior?"
"Should would change our product to make it easier for the customer to do what they are doing?"
and plenty of other questions.

Some companies like ours will tweak the product in what we call a test market without telling anyone. Monitor the situation to see what happens. We will conduct polls and surveys using techniques to get answers from the entire customer base. All kinds of different methods. Hopefully all of them come up with the same results.

[SavinDwarf]

The concept of what we call data mining is to get accurate data on the what. You want to know what the customer is really doing. Not what a vocal 1% says the customer base is doing.

You use other methods to determine the
Why are they doing this?"
"Should we try to extinguish this behavior?"
"Should would change our product to make it easier for the customer to do what they are doing?"
and plenty of other questions.

Some companies like ours will tweak the product in what we call a test market without telling anyone. Monitor the situation to see what happens. We will conduct polls and surveys using techniques to get answers from the entire customer base. All kinds of different methods. Hopefully all of them come up with the same results.

That is exactly what we do in other industries. We also used to hire professional organizations to build survey questions using all the normal techniques to try and avoid bias in the questions and asking the same questions in different ways, and including additional demographic question where our data was deficient or missing.

My experience with the few surveys I have received from the game industry don't seem to have been professionally constructed ?? (but it based on about 4 of them, so it may be bad luck and not an overall indication of the industry).

the gaming industry has other challenges such as the owner of the credit card may not be the player etc.

To your knowledge does the gaming industry use independent surveys to get demographic data on the player base?
My experience was with an industry were directly asking age/sex info would yield poor results. So we had a shell organization to question on fashion, dating, sports etc. Then we would spot check the results of a few known cases against the results. We regularly hit 98%+ on sex and 96% plus on age range without ever asking sex of date of birth.