0
0
A couple weeks ago I presented a very high level look at Early access. What I found was that for some games it was actually better to release in Early Access (and go full 1.0) that it was to just launch in full you can read the full analysis here.
After looking at the numbers and writing that blog, my new perspective on Early access is that it is higher risk, but actually higher reward. The risk is that if you launch and it doesn’t go well, then you are stuck with a lead weight around your neck and you will have a bunch of fans asking for constant updates and you might be wondering if your company can afford to finish this game with the poor sales you had. Do you tough it out, spending money that you can never make back, just to make fans hate you, or do you just cut bait and abandon the game.
In today’s blog I want to try to address this question:
“Based on the EA launch, what can I expect with my full launch?”
Is my game a lead weight? Is it successful? What is normal? Is there any hope?
Methodology
For this analysis I used VGInsight’s database of scraped game data via their fantastic API. The team at VGInsights were very nice to give me access and they are good people. This is not a paid endorsement. I just really like it.
For every game that had done EA and a 1.0, I sampled the number of reviews they had 1 month after EA launch and 1 month after 1.0. There is nothing magic that happens 1 month after, it is just a standard date to judge by. By one month a games launch discount is over, any Discovery Queue featuring is over, and the game is out of the New & Trending widget (available only for 1.0 games).
To see this visually, here is a review history for a random game I picked (Revita). Notice how you can see how the spike of reviews usually level off after the first month (indicated by the gray bars).
I also created a survey and got 13 responses from developers who had games that had done a full 1.0 launch. While this isn’t statistically significant, looking at the variety of games helped me get some perspective on how reviews relate to sales (more on that in a minute).
For this survey I started with all EA games and filtered them by a bunch of criteria.
- I started with all games since 2014 that have at least launched to EA (8910 games)
- Then I filtered it down to just games that released 1.0 (3857 games remained)
- Then I filtered out games that had 0 reviews during the EA window and fewer than 10 reviews by full release and at least. (2313 games remained)
In the end I looked at 2313 games.
From VGinisghts, I exported all the data to a CSV and then used the help of ChatGPT to parse the data and draw graphs. NOTE I know ChatGPT can hallucinate but I was not using it to generate data, rather I was using the code parsing tool to quickly filter rows of the table by different criteria, generate queries, calculate things like correlation coefficients, and draw graphs. I always double checked numbers using google sheets to make sure the numbers were correct.
Boxleiter problems for EA
To estimate sales for most games, you take the number of reviews and multiply it by 30. It usually works pretty well to at least give you an idea of how much a game sells.
But when I looked at the 13 games from developers who gave me their sales numbers I noticed that a lot of gamers wait to review the game until the 1.0 launch.
For most of the games I surveyed, they got a lot of sales in the first month of EA, but relatively not a lot of reviews. Then when the game launched to 1.0, I saw fewer sales but more reviews than at EA. On average, the units sold per review ratio was 10.3 points LOWER during the 1.0 than during the EA launch.
A significant portion of EA buyers don’t review the game until it has its 1.0 launch.
This discrepancy, makes it much murkier to calculate revenue at EA and 1.0. Therefore in this blog I will only be reporting review counts.
For future revenue estimates maybe we should see if there is a pre-1.0 boxlieter number and a post-1.0 boxlieter. I didn’t do that in this survey but may do this in the future.
Correlation between Early Access Launch and Full 1.0 Launch
Before I started this, I had a hunch that you can get a pretty good sense of how your full launch will go by looking at your EA launch. After scraping the data from VGInsights, I did the calculation.
Looking at the number of reviews earned in the first month of EA and the number of reviews earned at full launch, the correlational coefficient is 0.75.
The Coefficient is based on a scale of 1.0 meaning a perfect cause and effect and 0.0 meaning it is just random noise. A 0.75 is a very strong correlation.
Here is a plot showing that relationship. Each X is a game.
How much better will your game do when you do your 1.0 launch?
This is the question everyone asks me after their Early access launch. So a good way to predict what you are doing when you launch. Basically multiply your EA launch review count by some constant and maybe predict how 1.0 will go.
So here is how many more times your 1.0 full release review numbers will be from your Early Access review numbers
Average: 3.78 times
30th Percentile: 0.76 times
Median: 1.4 times
70th Percentile: 2.5 times
Here is a graph that shows the distribution of how many more times a game will earn on its early access release.
Only 109 games earned more than a 10X increase in reviews.
Only 5 games earned greater than a 40X increase. Here they are
- Super Animal Royale with approximately 49.09 times increase.
- Retrowave with approximately 42.15
- Animal Revolt Battle Simulator with approximately 44.34
- Tower Tactics: Liberation with approximately 49.67
- Shadows of Forbidden Gods with approximately 44 times increase.
Our EA launch was awful, can our game ever succeed?
My general marker for success on Steam is 1000 reviews. I have looked at a lot of games and I find that to get up to 1000 reviews (even for free games), you need the Steam algorithm to boost you by showing your game in the various recommendation widgets. Games that have 1000 or more reviews have been “blessed” by the algorithm and are shown everywhere.
After looking at all the games, only 1 game managed to EA launch with fewer than 10 reviews and then earn more than 1000 reviews by 1.0 launch. ONLY 1! That game is Prop And Seek it’s a Free to Play multiplayer game which had 4,606 reviews as of today.
On Average, the games that got at least 1000 reviews by 1.0 launch had 407 reviews within the first month.
IMPORTANT: there is nothing in the algorithm that magically kicks in when review_count == 1000. I just use 1000 as a bellwether indicating the algorithm likes what it is seeing with this game and showing it all over.
Are there any big improvement stories?
When the Steam algorithm sees that a game will do well it throws so much support behind it that the sales (and therefore reviews) look like a hockey-stick that hides smaller success stories. For instance Mount & Blade II went from 100,636 reviews at EA launch to 203,621 at 1.0 Launch. It is huge increase between the two launches, but the game was a big hit from the start.
What about minor successes? I dug through the data and these are the top 4 games with the biggest improvement in review count relative to their EA month-1-numbers.
| Game | # reviews one-month after EA | # reviews one-month after 1.0 | Days in EA |
| King of Retail | 11 | 2674 | 1268 (~3.5 years) |
| Good Pizza, Great Pizza | 11 | 1423 | 1676 (~4.6 years) |
| From The Depths | 63 | 7312 | 2283 (~6.25 years) |
| ΔV: Rings of Saturn | 17 | 1739 | 1439 (~4 years) |
One thing you might notice is how long the games are in Early access. Could this be the answer?
Let’s take a look
Can I just grind it out? AKA: What if we work really hard and spend a long long time in early access development before we launch?
This is secretly the most important question for EA. This is the opportunity cost question. This is essentially the computer science halting problem question.
Is it better to just keep updating your game forever or is it better to wrap it up and start working on a second game that might do better?
The basic question is, is there a correlation between the number of days that the game is in Early Access to the increase in reviews from EA launch to the 1.0 Launch?
The correlation coefficient between time in EA and review increase on a scale of (1.0 to 0.0) is 0.141 meaning a very very slight correlation. Here is a splatter chart of all the games and their increase in reviews plotted against the number of days in EA.
As we know, correlation does not imply causation so my hunch is the different causal variable is positive sales. Games that are having a steady increase in reviews (and for that matter sales) are able to sustain development longer. If developers aren’t getting sales, they are just going to fully release the game and move on.
My recommendation based on this: extending development really doesn’t change the course of action. There doesn’t seem to be a tipping point where all of a sudden things turn around. So if patching and improving your game isn’t improving sales, it is probably time to move on. If you enjoy making the game and it is bringing in enough money to sustain you by all means keep the game rolling.
My theory is that you can know very quickly whether your game has “the magic” to do well on Steam.
SIDE NOTE: curious what game is that little “x” on the very far right of the graph? GearCity! It had its 1.0 release in January 2022 after 7.6 YEARS in EA. During that period it gained 1041 reviews over that period or 136 reviews per year.
Do review scores at EA matter?
One of the primary reason developers go into EA is they say “We want to get feedback on our game and improve it with the community.” (Side note, I don’t think this is a reason to do early access but more on that in a future post).
So let’s look at percentage positive reviews and see if that correlates with the number of reviews in the 1 month after 1.0.
Correlation: 0.0659.
That is a very very weak correlation. Basically how positive a game is has almost no effect on the number of reviews (and therefore sales). That sounds crazy! I have a few theories why that is but again this post is going on way too long and will have to save that for a later post but my general theory is that for most games, review positivity doesn’t affect people looking to buy the game.
Editorial: What does this all mean?
So after looking at a bunch of factors and all this data here is what my gut says….
BUT FIRST A CAVEAT: this is just my opinion and I am making some logical leaps that are not 100% proven in the data. But, we are all risk takers embarking on this entrepreneurial and artistic journey so there will never be a perfect decision that can be 100% proven with data. So yes, yell at me that correlation does not prove causation and that statistics cannot tell me what I am about to say. I know that. At some point we have to leave the comfortable nest of hard data and try to fly. So here goes:
Basically there are 3 scenarios based on what happens to your game after your first month of Early Access:
Scenario 1: The game is a miss
In this scenario, the game gets fewer than 10 reviews in the first month of EA. Your choice in this scenario is easy: this game didn’t connect and you should drastically reduce the amount of time and money you spend on it.
Polish the game up a bit. Put out 1 or 2 more updates in the next couple months. Try your best to be a “good dev” and don’t yell at your customers, then ship your 1.0 as quickly as you can without too much additional investment and move on. Be honest with your fans that the game didn’t connect and move on. I have found that developers who abandon games without taking them 1.0 can get a reputation for not finishing and can raise the ire of the Steam customer base, which will hurt your chances with future releases. Don’t abandon your game but also don’t spend too much time trying to get out of the EA hole.
I have also seen a lot of developers not understand what EA is. They assumed that EA is a sort of beta test. They throw a quick prototype up on Steam Early Access with 0 marketing to “get feedback” and decide whether they should develop the game fully and release a 1.0 version and maybe get a publisher. This strategy is not how EA works. Your EA launch is your launch. You need to do a full marketing push. You need to get 7000 wishlists before you launch. You need to make a trailer, get into festivals. If you want to sign your game with a publisher, and your game is available for sale, they will not touch you. They want to be there for the launch of your game at EA because that is when you will make most of your money.
Treat EA like it is a full launch of a nearly final game. If you want to “get feedback” to see whether you should invest fully in a game, put a free demo out on itch.io, or do a playtest on Steam while it is in a “coming soon” state. I really wish Valve would get this message out louder. I have seen far too many devs think EA is just “testing the waters”
I want to be clear, if you flub your EA launch and get fewer than 10 reviews, your game is done, you cannot climb out of the hole.
Scenario 2: The game is a hit
The game got over 200 reviews in the first month of EA. In my opinion, this game is on a good trajectory to get into “Real Steam” and get support from the Steam algorithm further increasing your sales.
Based on the data, 80% of all games that got to the magic 1000 reviews by their 1.0 launch, had at least 200 reviews in their first month of EA. That isn’t a guarantee but it is a good sign.
If we look at it from the other direction, 285 games earned 200 or more reviews in their first month of their Early Access launch and 209 of those games went on to earn 1000 or more reviews by their 1.0 launch.
So my recommendation to games that got 200 or more reviews at EA launch is to keep updating the game, building your community, and (depending on your burn rate) living in Early Access for a long time. The Steam algorithm will most likely support you and show your game through out Steam. This scenario is the best scenario.
Scenario 3: The game is stuck in the valley of sadness
In this scenario the game got more than 10 reviews in the first month, BUT it never got over that 110 review threshold that I have seen seems to be the turning point..
This scenario is when you really have to make some hard decisions and take on some big risks. It is very hard at this point to reach orbit and get into real steam. For instance There were 151 games that got 65 to 110 reviews but only 14 of them made it to “Real Steam.” This is why I call it the valley of sadness. The game isn’t a total failure, but it also isn’t a hit.
Games in this scenario are often on their own when it comes to generating sales. At this stage, Steam will not be showing your game in that many places. You can’t rely on the visibility algorithm yet.
So, your choices depend on your financial situation: IIf your burn-rate is low enough and it doesn’t cost you much to keep developing your game, and your marketing team can find and bring in enough customers to keep the studio alive, keep trucking along.
However, if you are holding out for things to change when you go to full 1.0 launch, I will be honest, it most likely will not happen. It is quite rare for a game in this position to radically change course and become a hit.
So similar to scenario #1, I would look at ways of cutting features, launching your game while you are still in the good graces of your community. I would not extend development. I would not risk spending additional resources hoping that you are going to fill and inside straight and suddenly become a hit. Look up the sunk cost fallacy. Do not make bad decisions trying to spend your way out of this problem. Do not spend good money on bad (as they say.)
SUCCESS MIGHT HAPPEN, but it is unlikely to. Please be careful.
Here is a graph with a bunch of games grouped by the number of reviews earned at EA launch.
So why did I pick 200 ? I don’t know, it is just my gut after looking at this chart. Games that earned in the range of 200-300 reviews seemed to kinda reach this plateau of predictability. Like we can kinda see you have a 30% chance of succeeding once you get up to 200ish reviews at EA Launch. Again, all of this is risky. There are no guarantees and 30% is still awful chances 🙁
Side note about this graph: why did the games with 250-290 EA reviews faire worse than games that earned 210-250? I think it is just a limited sample size. Since 2014, only 22 EA games have earned in the 250-290 review range. There is a lot of variation there.
GOODHEARTS LAW WARNING:
This is your regular reminder to observe Goodhart’s law. This is also my necessary acknowledgement so that the people in the Reddit comments section don’t say stuff like “Chris fell into the trap of Goodhart’s law” “Hasn’t this idiot heard of Goodhearts law?”
Read up on Goodharts Law. Here is his picture:
Remember “when a measure becomes a target, it ceases to be a good measure.”
SO MY WARNING TO YOU: There is nothing magic in the algorithm about 200 EA reviews. Don’t try and set a goal to get 200 reviews by your first month. Do not try to buy review bots or coerce your employees to leave 200 reviews. 200 is just a measure I kinda just plucked out at random. This number is just like your temperature; it just indicates you might be sick or healthy. Raising your temperature doesn’t make you sick, it’s just that when you are sick you usually have a high temperature. So, getting 200 reviews by employing shady and/or unethical tactics will not make your game a hit. Review count is just an indicator that you are on the right road.
