User feedback for your game is one of the most important things you can do to determine if your game is fun or not. Note that user testing is not the same as running a focus group on your game to squeeze out the last ounce of your indie personality. No! Think of user testing as similar to when comedians try out their new routines on fresh audiences at tiny clubs. They do this to hear which jokes hit and which jokes bomb before they perform them at bigger venues.
Many indies go to conventions and shows to get feedback on their games. there are several reasons why I don’t think this is the best place to do this
- One, it is super expensive. You gotta pay booth space, decorations for your booth, airplane tickets, hotels, employee time, not to mention potential post-show sick time if they get convention plague.
- Two you can’t target your playtesters. If you make a casual game you might have a hard time finding testers because convention-attendees are pretty hardcore. Also, any random passerby might not be fans of your game’s genre. They might have just wandered into your booth because it is close to the bathroom and they are waiting for their friend to get back.
- The feedback will be biased because you are inviting them to evaluate your game at your booth surrounded by your game’s art, and schwag, and stickers, while you wear a T-shirt with your game on it. That is a lot of pressure to put on someone if you want their honest feedback. Most people don’t want to be mean to your face because they see you are clearly invested in this game.
- Conventions have a party atmosphere which means the show’s zeitgeist prefers party games. That is why 4-player local coop games show so well at conventions yet typically underperform when released. People are hanging with their friends and party games fit that feeling. Deep strategy games show terribly because who has the patience among the din of the convention to learn all the game’s complex rules while they would be happy to do it in the quiet of their own room.
- You can’t record the session or take detailed notes because part of your brain is also in booth management mode. It is as difficult as trying to run a high school science experiment while also hosting a cocktail party.
Here is An easier and cheaper way to get feedback.
If you are just starting out the best way to get feedback is to recruit players from forums and subreddits and then watch them play your game through remote screen share on discord or google hangouts. Here is how to do that
Find players
I like to recruit players on subreddits that match my game’s genre or on subreddits for the most popular game in my genre. I also search out people who left steam reviews on games in my genre. This is very similar to how I recruit for my 1×1 interviews. See here for how to find customers to talk to.
I send potential players a direct message and ask them the following “Hey I saw your comment on <that-other-game> subreddit. You seem to know your way around these types of games. I am trying to get feedback on my game and would really like the feedback of someone who knows as much as you. Would you mind trying my game out on a screen share so I can get your feedback? I know it sounds weird but for the inconvenience I am willing to give you <a dollar amount you can afford> for just 30 min of your time.“ Side note, if you can’t afford paying them, you can alternatively offer to give them a free copy of the game.
If they agree, I then forward them to a calend.ly invite that I set up where they can pick a time. Note I love calendly you can do a lot on the free plan.
I also forward over this disclosure form (Side note I know that site looks sketchy but I swear it is legit.) This just lets them know what I am doing and that I will not be using what they say as marketing of the game. This isn’t necessarily required but I always like to play it safe and be upfront with my testers.
The test
I use Discord to host the screenshare because everyone on Steam pretty much uses it. But, I defer to them if they have a different preference such as skype or google hangouts.
I send them over a build of the game well before the call but ask them not to install or play it because I want to get their first impression.
At the start of the call tell them what you are doing. Here is what I usually say “I am working on a game and I am trying to find out how people understand it and play it.”
Tell them that you want them to play it and say aloud what they are thinking. Tell them to think of it as if they are doing a let’s play of this game. Also, tell them you are not going to talk or answer their questions because you want to simulate how they would play it in their own home. Tell them to please be as honest as they can and that you are not offended if they get confused, or bored, or frustrated. In fact, you welcome their honesty.
Also be sure to emphasize that you are interested in watching how they play it and to pretend you are not even there. Sometimes people think these user tests are actually critique sessions and instead of actually playing the game they just nitpick everything they see rather than actually playing the game in any natural way. Try your best to get them out of this mode.
If you are going to record the session, ask them if they are comfortable with this. If they say no, don’t record but continue with the test.
Introducing your game
Now I used to be confused with how much to tell test participants about my game before they get their hands on it. Do I tell them the genre? Do I tell them how to play it? The goal? The mission? Or do I just dump them in the deep end and say nothing and have them start the game?
But since my Steam user research, I like to use this as an opportunity to test my marketing too.
What I do now is before we start I send them 4 screenshots of my game. Then I send them the short description that I am considering putting on my Steam page and I send them the top 5 tags I am considering. These 3 bits of data are the key things people use to figure out the genre of the game.
Then I ask them “based on what I just send you, what type of game do you think this is and what do you think you will do while playing it?” Note that if what they say is wildly different than what your game is it means you need to fix your marketing materials.
Carry on no matter what they say.
The user test
The hardest part of running a user test of your game is shutting up. Don’t tell them what to do. Don’t tell them what to expect. Just shut up and let them experience, stumble, get lost, and stumble some more.
It is hard for me to shut up even though I have been running user tests for years. The trick I use to keep my mouth shut is I tell myself that for the next 30 minutes the only thing I am going to say to the user are going to be in the form of questions.
For example….
A common thing you will experience is players asking semi-rhetorical questions like “Hmmm, should I go through this door?” Don’t answer “Yes obviously!” or “No, that is the last place I want you to go. My level design clearly is pointing you towards the mountain right in front of you.” Instead, I would say “What makes you think you should go through the door?” or “What do you think you will find through the door?”
These questions let you understand what the player is thinking. It helps you know if your player is actually stuck for the right reasons.
Now there are times where the player gets totally lost or finds some glitch and ends up through the floor. In those cases, I have a 3-strikes rule. If they try 3 things unsuccessfully and yet are still stuck without making any progress, and their answers to my questions indicate that they are truly lost, I put a pause on the test, put them back on the right path and then go back to my “no talk, only questions” way of testing. Just be sure that you took detailed notes of what got them in this state so you can fix it later.
After the time that you reserved with them is up, stop the test and thank them for their feedback.
At this point don’t ask them questions like “so what do you think?” or “would you buy this game?” I will get to how to ask that in a minute because those are some pretty loaded questions. However, if at this point they say things like, “can I keep this game?” or you have to kick them off the call or “When is the full version coming out?” that is a pretty good sign.
After the test
After the call immediately send them whatever honorarium you agreed upon. Note if you are paying in Steam cards you must be friends with them for at least 4 days before you can send it through Steam. Probably has to do with money laundering or something.
I also send them a brief follow-up survey to get their assessment. This is where I try to get their feedback about how interested they are in the game post play. Here is what I ask them
- “What games have you played reminded you of <your games name>?” This question helps with marketing so you know your target market.
- “What is the last game they bought that was like <your games name> and how much did they pay for it. Was it on sale? Part of a bundle? or did they pay full price for it the day it launched. – Past behavior is a better indicator of what they actually do.
- I also ask them the net promoter score question. Basically you ask them on a scale of 0-10 how likely are you to recommend this game to a friend?
- I ask them why they gave the score they did.
Compile notes right away
I retype my notes or replay the recording while the test is still fresh in my mind. If i can’t do it right away I definitely do it before my next test. When you do two tests without reviewing your notes, your observations will start to mix together. Don’t waste all that great work you did by forgetting what you saw.
How many people should I test?
For any single build of the game, test 6-12 unique people. After about 6 people you start to see the same things over and over again. That is just a waste of your time. At that point, I stop my regular testing, review my notes, make my changes, and then set up another test.
I also resist the urge to make a bunch of changes after just testing 1 person. Sometimes 1 person is just an outlier and there are bigger problems that are only borne out by having more than 1 person test.
But if 3 test participants reveal that my game is extremely unplayable I don’t do any more tests until I fix what I saw.
Other user sources
Now there are other ways to do testing than this remote screen share. You can go to local game developer meetups like IGDA and give it to your fellow club members. Although, watch out because game developers have more opinions and are not representative of the average players. You will not be testing the usual Steam audience.
You should also check to see if your local game or comic book store has a demo night or would be willing to let you set up at the corner of the store.
I have not tried it yet but barcades are incredibly popular right now. Check with the bar manager if they would let you setup at the corner of the bar with a laptop and controller. Offer 1 free drink for any playtest.
Testing and boundaries
After my last article about doing 1×1 customer interviews, I had a reader bring up that it is a much different situation if you are a woman trying to interview internet randos. The tendency for (most likely male) fans to try and push boundaries can be a NO GO for this type of research. I cannot provide an appropriate answer because I have the privilege of being immune to this behavior because of who I am. However, I put out a couple of calls to women I know in the UX field to see if they have any tips for this.
One thought I did have was to do this research with a buddy. So it would be two researchers and 1 participant. I would hope that a problematic person would be less likely to offend if there were 2 people on the call.
But if you have any suggestions for ways to solve this please let me know.
Photo by Daria Nepriakhina on Unsplash