Very insightful post. Now for the most important questions – what is the best way to come up with the most important business questions to ask…?
This post is about that: the important questions. What they are and how to get them. It’s about the thought process and a sneak peak on how analysts, scientists and statisticians translate business questions to quantitative questions that can be answered with data.
What defines an important question?
Depends on who you ask. I’ve had and I quote “very important questions” that I enquired “why do you want to know this?” and got the reply “I’m really curious about it.” Don’t let the overuse of quotation marks lead you think I have a problem with these questions. I have even less problems with curiosity! The amount of stuff we’ve uncovered because we were curious is overwhelming.
My point is that these questions are not important. What is important is that we act. Actionable insights is definitely an overstated expression but when applied properly it is true. However it is important to define why is it that we, the analysts, cherish actionability so much.
To explain let me copy here a thing I wrote a couple of days ago:
Every Game Analytics and Data Science Team Member is a data analyst. Our core responsibility as data analysts is to extract and report empirical evidence to the business questions that are brought to us. Our objective is to empower other Miniclipers’ decision making process.
I think it’s pretty straightforward but to be clear, an answer implies a decision.
Oh and by the way, this was taken directly from Game Analytics and Data Science Team Handbook. Just don’t tell anyone I told, ok? It’s our little secret!
Where do we find them?
Human beings are really good at spotting problems. Retention is dropping, revenue is dropping, etc is dropping. Spotting problems is part of our survival instinct. These are the easy ones. So this is your first source of questions.
This is where questions like “What caused X?” occur where X can be any problem you found. Let’s keep this question for later use.
On the other side the spectrum are the opportunities. Should be easy to understand the logical difference. However, opportunities are much more difficult to spot. Most of the times we think of opportunities because of something we saw or read or a game we played. That’s fine but let me invite you to go a bit deeper. The reason for that is that the questions are usually “Will this thing improve Y?” which is a tad limited. I won’t even keep it for later use.
The point I want to make is that I like to think about features as opportunities. Let’s say that you want to introduce new content. Ask yourself this: why? If you think of a feature – in this case content – as an improvement to the game, what are you expecting to improve? Which part of the player lifecycle are you expecting to affect? Let’s say that you want to introduce new content to increase long term retention of veteran players.
And there’s your question: “Does new content increase long term retention?” This one is a keeper too! Let’s just rephrase it a bit: “Does A affect Y?”
Dealing with “Does A affect Y?”
This is a pure A/B testing question. So I hope you see where this is leading us! When you think of a feature A where you want it to affect Y, hopefully in a positive way, you want to test it. Your mindset should be that all your A’s are experiments and their impact on Y is what will determine if they’ll go into the game definitely or not. Technically speaking we are forming an hypothesis that A will improve Y.
The reason why we, the data dudes, want all this to be an experiment is quite simple: if we test an hypothesis we can imply causality. We can say, with more or less certainty, that A was the cause of the effect on Y. We can even recommend if A should go or not in the game. Ain’t that neat?!
Dealing with “What caused X?”
But not all game development is about new shiny features. Solving problems is fair game too. Keeping in mind that X can be any problem we find, answering what caused X can be particularly difficult because it is very difficult to demonstrate a causal relationship.
Most of the times we can only demonstrate a correlation and eventually tie it to some unintended consequence of a previous decision but we can’t say for sure the root cause of X.
Naturally there will be some theories and therefor some hypothesis drawn from data. From this moment on it is very similar to the opportunities section. For instance, let’s say you have a retention drop and the question is “What caused the retention drop?” Analysis shows that the retention drop started after the introduction of a new monetisation mechanic. You’ll probably have two hypothesis:
- Removing the monetisation mechanic increases retention
- Adding a new retention mechanic increases retention
Obviously you’ll worry about the drop of monetisation of hypothesis 1 and the effectiveness of hypothesis 2. Time to setup an A/B test.
In case you didn’t notice this creates a cycle:
- Data shows opportunities and problems.
- Opportunities and problems define hypothesis.
- Test the hypothesis
- Go to 1
And now to the most awesome conclusion in this blog until this day: the way to come up to important business questions is through the cycle of data informed design.