Get the big, easy questions out of the way early.
Screener questions have the power to ensure you get feedback from the people that matter to your business. That's because they can identify unsuitable candidates early. Think of the process as a funnel. You’re refining your pool of potential participants, and refining them further until your left with the people you want to see and hear from. Or, think of it like weeding an over-grown garden. The biggest, tallest, most obvious weeds come out first, simply because they’re the easiest to grab. Then comes the hands-and-knees part where you pull out the smaller ones.
If you know that you absolutely want experienced gamers to be part of your Concept testing, ask some specific questions about how users interact with games right out of the gate. Users who answer appropriately move on, users who don’t get dismissed (and maybe saved for future studies for newbie gamers.)
Avoid asking leading questions
There’s a conversational tendency some folks employ by adding a “right?” at the end of every sentence, so that you have no choice but to nod or shrug in agreement. Right?
This is an example of “leading.”
A leading question will indirectly nudge a user toward answering a question in a certain way. You want to avoid this in your studies and especially in your screener surveys. This isn’t the time to cajole people into answering how you want them to answer. You’ll end up with study participants who aren’t what you’re looking for. A good way to identify whether a question might be leading is if it includes a hint or excludes possible answers.
Examples of a leading question:
Leading: On a scale of 0-100, how much do you hate pistachio ice cream?
Not leading: On a scale from 1-10 where 1 is gross and 10 is delicious, rank how you regard pistachio ice cream?
Another way to avoid leading questions is to provide a series of unrelated options as answers. For example, if you want to screen users who have a high level of concern around internet privacy issues, rather than diving right into questions about internet privacy, you can create a question like this to get to the people who really care:
Likewise, avoid yes or no questions which tend to be leading. Users might answer in the way they believe will entitle them to participate in the study.
Make like Law & Order, and when you spot a leading question in your own screener survey, object!
Provide clear and distinct answers that don’t overlap
We’ve all taken a survey where, on a certain question, we find ourselves searching for that “all of the above” option. It’s frustrating when we’re asked to choose just one answer when the real answer is “a bit of this, a bit of that.” If you force your respondents into choosing just one that doesn’t really hit the nail on the head, you’re not getting the most helpful insight into your product. The solution to this is making sure your answers have clear borders between them without any overlap.
Be sure not to overlap ranges.
For example, when asking for ages or sizes or anything with numerical value, make sure your values are mutually exclusive:
Correct: 0-3, 4-7, 8-12
Incorrect: 0-3, 3-7, 7-12
Likewise, for answers less definitive or answers that can’t be made into mutually exclusive options, perhaps offer the multi-select option rather than the single select option.
Ask precise questions
Asking good questions is just as important as offering clear and distinct answer choices. A mushy question where users are required to interpret what it is you are looking for will yield weird mushy answers and land you with survey participants who might not be right.
Try asking about a specific aspect of the user experience: