The Pocket Cyclone

Today’s post for my idevblogaday.com entry is describing some ideas about survey’s that I am really passionate about. I will soon write a followup post about a project I am doing on the side with my summer intern, creating a mini-survey for a crowd of beta game testers.

If done correctly, surveys can be great tools. However, in my career, I have witnessed so many surveys going wrong, I could write a full blog about those absurdities. I have seen large corporations creating strategies to meet objectives derived from completely meaningless survey results. It is simple. If a question is not clear, the results are misleading.

Example for such a survey question:

How do you like Apple’s strategy on a scale from 1-5?

Well. Sounds that answer from a bunch of people could give insightful results.

However, try to answer it yourself and chances are you will ask: From an investors perspective? From a customers perspective? From a developers perspective?

You could assume that if you ask that question to a specific group, like employees, it might be clearer, but what I frequently found when working with individuals through survey results is that too many folks have a great capability to put different hats on. So when you ask that innocent question to say Apple employees, a significant portion will put their investors hats on and answer the question from that perspective. Which instantly renders your results worthless.

Another example I stumbled upon in an internal survey once:

How do you rate the performance of Senior Management?

Straightforward, at least I thought so. Talking to individuals afterwards to try to put their ratings (which were spread all over the board) into perspective, I found that the one half interpreted the term “Senior Management” different than the other half! One thought it means their direct reporting line boss, the other half thought it means the highest management level, like the CEO and the Board of Directors. And of course that triggered very different answers. And rendered that survey section useless.

To solve this dilemma, you either make the question more detailed or you work with sub-questions.

However, there is another unwritten law of making surveys successful: Make them as short as possible. People are not willing to sift through pages and pages of questions. Especially if you do a web based survey, the attention span you can count on is very short.

The problem is, those two philosophies are very contradicting. And the results are often poor, just look at any random survey that you find in your inbox these days.

So for my upcoming exercise of creating a survey for game beta testers, here are my Top Five rules to avoid that conflict and hopefully create a meaningful survey:

1. Stay clear from general questions

“Do you like this game?” is pointless. Instead, try to break it down into digestible components and also approach it from different, tangible angles. “Would you recommend it to a friend?” and “Will you play it again?” will give you small pointers which in their sum will give you the answer you are looking for.

2. Don’t seek answers to Life, Universe and Everything

True, you captured an innocent victim who will answer your questions. False, you can not exploit them, they will not finish or give you overall low quality

Instead, focus on one or two areas that are really important to you. Ten questions maximum!

3. Stay clear of free-form text answers

Those are a huge turn-off for anybody doing a survey. You run a high-risk of many people not completing it. Plus you lose comparability of answers.

Instead, try to either work with multiple-choice answers or scale answers (“on a scale from 1 to 5″). Make sure you make clear which number stands for “best”. And then add one question with an open text field, inviting thoughts and comments.

4. Don’t be nice

You don’t want your survey participants to be nice. You want them to be honest.

Honesty is something that needs to be encouraged. Simply asking for it can be enough and dramatically increase the benefit you get from the answers. Leading by example, using frank language in small doses is a more daring approach that I saw working: “Would you recommend this game to a friend (1-5, 5 means “hell yes”, 1 means “nope, it sucks”)?”

Caveat: You can find bad things in any game, so if you challenge your participants, they will be more than brutally honest.

5. Don’t focus solely on the survey subject

While it is a great practice to focus the questions around certain aspects of the game, you need to leave some room for general questions. It is as important as the answers to your questions to understand more about the person who gave you those very answer.

I recommend asking a few general questions regarding the person (sex, age, video game playing habits, etc) but also the environment that he/she took the survey.

Summary

When you manage to create a survey that is meaningful but small, then you can do something really great: You can get the same person to answer the survey for different games! For my own approach I definitely try to get there, I want them to rate another game, either a famous game like Doodle Jump or Angry Birds or a randomly selected Indie Game with the same set of questions. I’d expect this to give me some good insights on my target group.

And even if you make the survey anonymous, try to talk to some friends who took it. Try to find out where the weak spots are and eliminate them. Creating a meaningful survey is a lot of work.

Let me know what you found working for your own projects, if you plan to use surveys for pre- or post-release evaluation of your apps and also which survey websites you prefer.