The following is the methodology I used to run my user test of Steam.

Goal

I am an independent game developer who is not employed by Valve. I do have 1 pixel art platformer game (soon to be a second) on the Steam store and have earned money by selling my game on their platform (they receive a 30% cut of my sales).

I conducted this test because every time I have to market my game, or write the store page text I am at a loss for what I should write. I constantly ask myself “What do my customers need to see? What do they want?”

So I decided to ask them. I worked as a UX designer for 7 years and applied the techniques I learned on the job. Here is how I did this.

Observation technique

For this user feedback experiment I employed a user experience (UX) technique called a “Contextual Inquiry.” This methodology works by having a proctor (me) and the person I want to study (referred to here as “participant”) have a 1-on-1 open-ended interview mixed in with direct observation of the participant completing a series of tasks. I picked this technique because I didn’t know what I didn’t know. There is very little public qualitative data on Steam user buying habits (there are, however, more than enough untested theories by game developers). An open ended interview allows me to go in without any assumptions and let the participant tell me how they buy games. I could also then transition into having them show me how they use Steam. When using any interface or tool most people aren’t conscious of their habits or actions they take. By doing a direct observation I can pick up on things that they didn’t think were important for me.

If I just did an interview, people have a tendency to say things that they don’t do just to make me happy or to impress me. For example if you ask a person how much they exercise you will get wildly different answer than if you actually go out and watch them work out.

For this study, all participant interviews were conducted remotely over a Discord screen share. I used OBS to record the audio and video. Participants used their own computer. Some did mention that they sometimes browse Steam on their mobile device but for this test everyone used their gaming PC to browse Steam. I did not know any of the participants before the test and never met them face to face. (see “recruiting” to see how I selected the participants).

I asked all participants the following questions:

Question 1) I started the test by asking the participants how they access steam and why they check it and how often.

Question 2) I then asked participants what they do when they first arrived on Steam. What do they check first? What are they looking for? This question was somewhat unstructured because I wanted to be open to the different browsing styles that each participant had. If they went to the discovery queue or any of the other algorithmically sorted lists I transitioned to my next question.

Some participants said they check their wishlist first. In those cases I asked them to show me their wishlist and to show me how they manage it. After we discussed their wishlist, I asked them how they decide what to add to their wishlist. This question led them back to the Steam front page and to my next question.

Question 3) When they got to one of the Steam lists, I told them I was interested in seeing how they find and decide which games they are interested in. I told them to wishlist two games that they have not previous wishlisted and are genuinely interested in. I reminded them to search as if I were not there watching them. Note that I specifically asked them to find a game they were “interested in” and not find something “new.” This question was worded this way because I did not want to bias them against wishlisting games that they heard about on other platforms (such as Twitter, Youtube, or Twitch) but never wishlisted. I wondered if some people only wishlisted games they heard about through external platforms such as the games press and completely ignored Steam recommendations. That turned out not to be the case but I wanted to ensure that I did not influence the participant. I also didn’t want to say “new” so as not to make them think I wanted them to wishlist a newly-released game.

When they clicked on a game from Steam to investigate, I let them browse at their own pace but would step in to ask them to clarify some comments they made or to remind them to “talk aloud.” If they did click the “wishlist” button I asked them to give a brief explanation of what aspects of the game or store page influenced their decision.

Question 4) After participants wishlisted 2 games or if they made it through their discovery queue without any selections, I steered them to try out the new Steam lab features. When doing this, I first asked if they were aware of the “Steam Labs.” I wanted to test brand recognition of these features. I also wanted to know if they had used it. If they answered “yes” I asked them to show me. This was a secondary sub-test to see if they could remember how to navigate to it. If they answered no to either question I would direct them how to get to the Steam Labs page. Once there, I did not tell them which of the 3 Steam lab “experiments” to use because I wanted to see which one was most interesting to them. After they had wishlisted a game using the Steam Lab tools or they were within 2 minutes of our 30 minute time limit I told them it was the end of the feedback session. Note that not all users I interviewed had time to test the Steam Labs features. This was because we spent more time exploring the queue or looking at game pages.

You might notice that I never gave the task “please buy a game.” As a game developer I am well aware that most gamers wishlist before they buy. Therefore I didn’t want to force them to do something they would not normally do. Also part of my research is to figure out HOW they decide to buy vs just wishlist. I am aware that most people only buy games during seasonal sales. (Note I will be doing future contextual inquiries during a seasonal sale). Also the ethics of forcing someone to buy a luxury good seems highly problematic.

At one point I was considering giving participants $25 and telling them to spend it however they wanted on Steam. However, a games researcher I talked to pointed out that a person’s behavior changes when they are given a gift. She pointed out that if she received a pro-bono Starbucks gift card, her beverage choice would change quite a bit. Good point!

Recruiting

For this study I wanted participants who would theoretically buy indie games because, well, I make indie games..

In the article “Your Target Audience Doesn’t Exist” by Sergey Galyonkin (the Steamspy guy) describes those people who buy indie games.

“1% of Steam gamers own 33% of all copies of games on Steam. 20% of Steam gamers own 88% of games…. So, to be a member of the “1% group” of Steam gamers you have to own 107 games or more.”

https://galyonk.in/your-target-audience-doesn-t-exist-999b78aa77ae

Per Sergey’s research, the person who only buys/plays Fortnight, Skyrim, or Call Of Duty does not go out and buy boutique pixel art platformers (the games I make). For this study I tried to find gamers in that 1% group.

I also wanted to ensure that within the “1%” group I sampled users from a diverse background of ages, ethnicities, genders, countries, and life circumstances.

I did require that participants be at least 18 years old. Mainly because the average PC gamer is around 38 ( See this NPD survey: https://www.pcgamesn.com/pc-gamer-statistics-reveal-equal-gender-split-and-average-age-38 )

Also because I required all participants to sign a consent-release form so that I could share videos of our interviews. If the participant were a minor, I didn’t want to have to go through the extra steps of getting their parent’s to sign the form too.

To find these folks, I approached communities where Steam 1%-ers hang out. I went trawling in the subreddits of /r/steam /r/indiegames and /r/gamergirls. I approached the mods in all of these subreddits to ask them if I could make a post asking for participants. They all declined. So I searched the subreddits for posts about Steam and PC gaming and took the step to Direct Message (DMed) anyone who left a comment.

I also used Steam directly to find users. I looked at the review sections of popular indie games and looked for users who had left reviews and had large game libraries. I then direct messaged them.

I also posted in the Arizona Streamers discord channel asking for participants (I live in Arizona.)

When I reached out to the potential participant I introduced myself and said that I was conducting a research project on Steam buying habits and would like their input. I then asked them if they would fill out a Google survey. The survey had the following questions

How do you spend your money on Steam?
What is your gender? (M/F/Prefer not to answer/a blank to specify their own gender. This question was to ensure that I got a more diverse representation in the study.)
Are you over 18? (See my comment about the average PC-gamer age of 38 / consent forms)
What are your buying habits for Seasonal Steam Sales?
What country do you live in? (Again to ensure participant diversity)
What is your Steam User Name (This is so I could see how many games they had in their library)
Would you be willing to do a 1 on 1 screen share (through Discord, or Google hang outs) where I observe you browsing the Steam store and you tell me why a game interests you? Time of the screen share would be about 20-30 minutes and you would receive a $25 gift card for your time. If YES, leave your email address in the space provided. If NO, leave blank. (Note there is no guarantee you will be selected for this interview, it will depend on my schedule and the types of gamers I am looking to study)

If the participant indicated they were open for a 1-on-1 interview, I would look at their survey answers to see if they fit the demographics I was looking for. Then I would setup a time for us to meet using calend.ly.

Participants were offered a $25 honorarium for 30 minutes of their time. An honorarium is standard research practice and does not negatively influence participant behavior.

When scheduling with the participant I never mentioned that I was a developer or had a game on Steam. I did this because I was afraid I would bias them into thinking I wanted them to review my game or that their feedback could be used in some form of marketing for my games.

I sent users the following consent form from usability.gov. This was to ensure that I could share my findings and that I would not use their quotes in any form of marketing. https://www.usability.gov/how-to-and-tools/resources/templates/consent-form-remote-usability-test-adult.html

Data synthesis

During the interview I took very limited notes so that I could focus on observing the participant and responding to their comments. After the interview I watched my screen recording and transcribed relevant points.

When compiling these observations I took an inductive reasoning approach. The main reason I undertook this project was because I really had no idea what Steam users were looking for and how they decided which game to wishlist and which to buy. As noted above, I used the contextual inquiry interview technique because I had no idea really what I would find. Also since this was my first qualitative study of Steam buying habits I didn’t have too many prior observations to reference.

For more information about inductive vs deductive reasoning see this

https://socialresearchmethods.net/kb/dedind.php

I grouped all my observations into categories I saw emerging such as The Discovery Queue, Reviews, Wishlisting Behavior, Screenshots, etc. I then combined the observations from each person into those categories. Those categories became the sections in the body of my article.

Methodology for a qualitative study of Steam

Goal

Observation technique

Recruiting

Data synthesis

Further reading

Leave a Reply