| sum_gamerounds | |
|---|---|
| count | 90189.0 |
| mean | 51.872 |
| std | 195.051 |
| min | 0.0 |
| 25% | 5.0 |
| 50% | 16.0 |
| 75% | 51.0 |
| max | 49854.0 |
A/B Test for Cookie Cats Starting Gate
python
AB test
Analysis of AB test to determine the effect of changing the game’s first starting gate.
A more technical write-up can be found here
Introduction
This is the result of an analysis from an AB test for a puzzle game called ‘Cookie Cats’. The test was meant to examine whether there are any changes when moving the starting gate from level 30 to level 40 . The “gate” is a roadblock for the game meant to stop player progressing through the game. It forces you to wait, or to ask a fellow player to pass through. This waiting time is hypothesized to extend player’s retention by reducing the risk of burning out.
The dataset is taken from kaggle but also known to be taken from DataCamp. But I cannot find it from DataCamp. It contains userid, version (gate 30 or gate 40), sum_gamerounds (total rounds played for 14 days after first playing), retention_1 (whether player came back on the day after first playing), and retention_7 (whether player came back on the 7th day after first playing). There are no missing data, however, one player was removed due to unachievable total rounds played (~50000 sum_gamerounds which equates to around 34 days assuming player spent 1 minute per round).
Analysis and Results
Based on our objective of finding significant difference between the two starting gate, we will be mainly focusing on retention_7 since it is the strongest metric that signifies player ‘stickiness’ and having sum_gamerounds as the supporting metric.
From Table 1, we can see that the mean lies in the 75th percentile, meaning that the distribution of total rounds played is right skewed. We can also see that the standard deviation is around 195, which suggests that we have quite the long tail. Because of that, we used mean as a measure of center and checked its robustness with median and trimmed mean (10%). We found no significant evidence between the two starting gate on total rounds played from our test. Not to mention, the three measure of center agrees that players from the two groups played about the same amount of rounds, whichever way we measured it.
sum_gamerounds description
It is important to note that at least, two-thirds of players from each group did not reach their intended gate within the first 14 days. This might reduce the power of our tests, which may decrease our reliability to detect effect. Note that this is assuming rounds is equal to finished levels at least until their intended gate.
While it is not clearly showed in Figure 1, there is a significant evidence that the split between groups are uneven (total of gate_30: 44,699 vs total of gate_40: 45,489). It is important to keep in mind that this might introduce some bias to our results.
Our tests found no significant evidence that there is a difference in retention on the first day between the two groups. Note that the players are not reaching their intended gate on the first 14 days, which implies even more players did not reach their intended gate on the first day after playing. The experiment can only reliably detect 0.93 percent points (pp) of difference reliably while the test only observed 0.59pp which was made worse by the samples dilution.
However, there is a significant evidence that increasing the gate level decreases the retention on the 7th day. The tests results for both retentions are visualized in Figure 2. Recall that there might be bias introduced by the experiment already and might invalidate these results.
Conclusion and Recommendation
To sum it all up, the test managed to find significant evidence that increasing the starting gate level from level 30 to level 40 decreases retention on the 7th day (19.02% vs 18.20%). But, we found no significant difference in total rounds played and our test was underpowered to detect changes in retention on the 1st day. In one sentence, players who got the later gate were less likely to return after a week.
Hence, we would suggest the company to not change the starting level. Though, the experiment design itself is flawed due to the uneven split between the two groups and may undermine the results of this analysis. For a stronger recommendation, we would recommend to redo the experiment and make sure that the bias introduced by the design is minimized and then redo the analysis.
For a more technical page, do click here