Guide to product research

Recently we held a meetup where Dashly and other IT companies talked about internal researches. We invited Alice, a user researcher at Skyeng, who told us about how Skyeng’s research team manages to combine user research and data analysis. She also shared some research organizing experiences. Other representatives who joined her to talk about their companies’ research methods were Kate, product designer at Miro, and Max, head of web & mobile development at ER-Telecom. The one shouting questions at them was Dimitrii Ive, the CEO & founder of Dashly.

Why product companies like researches even more than colleges do

While conducting qualitative researches on the development stage lets you save time and resources you would otherwise waste implementing useless features, later stage researches may let you see how to increase your key metrics and simplify the main processes of your company. 

The key of a product research is in it taking as shortest cycle as possible — the sooner your idea meets harsh reality (with your team receiving valuable feedback on the way) the less losses in time, money and other resources your company will eventually suffer.

How do you know if you need a research

The goal of your research must be clear. Don’t research general issues. For example, questions like “why aren’t people buying our product?” is not a good research goal. Ask more in-depth questions like “How do users go from reading our blog to registering in our service?”. Questions like that may serve as a good research starter. 

With that in mind, it’s important not to go researching every bit you can possibly reach by discarding useless and unsound researches.

How do you find out if a task is faulty? We know a task is set correctly when we have a clear understanding of what we need the information we may uncover for and what we’ll do with it. A research task is faulty when don’t know what we’ll get from doing the research.

Get the guide on product research

Got it! Check your email?

Who must do researches

It depends on what tasks you set for your researches and how big your company is. In larger companies, researches are usually held by a separate team. Skyeng, for example, has a research team that fully focuses on gathering insights for their product teams, and each product team has researchers assigned to them. Researchers are lead by product managers and are a part of the research team which is run by a horizontal manager. Product managers are responsible for business in this chain of command, and a horizontal manager is responsible for staff development. ER-Telecom, too, has a research team that leads large researches for the product and marketing team.

Another way is to assemble a cross-functional team for each research. In Miro, researches are lead by teams that consist of product managers, designers, and developers who conduct research as an additional task. Miro also has a data team that actively takes part in qualitative research and conducts its own quantitative researches. 

In Dashly, researches are made by different teams depending on the task of a research. Dashly’s growth hacking team constantly verifies hypotheses and does its own researches too. 

What skills does a researcher need

• curiosity;
• critical thinking;
• analytical skills;
• well developed empathy;
• communication skills;
• product knowledge;
• UX analytic skills and basics;
• counting ability;
• non-perfectionism;
• sociological imagination (an ability to interpret reality);
• an ability to abstract from your product, an ability to treat it critically (seeking out its flaws will help you improve your product);
• active listening techniques;

Don’t panic just looking at this list: nobody expects you to have all of these skills at the very start, but doing product research will give you a chance to develop them. 

Choosing methods and tools

Product companies use and combine different methods:

When it comes to product researches, qualitative methods are used more often as this method is more convenient to use for short iterations. However, Skyeng and other companies have proven that sometimes it can be useful to complement the data you’ve gotten during qualitative research with the knowledge you can get with quantitative research. For example, you can take some in-depth interviews and analyze the qualitative data on the segment you included in your interview at the same time. It also works the other way around, quantitative data can help you find your users’ pain point and qualitative data can help you cover it. 

Sometimes you can do with using just one method, like in-depth interviews or feedback gathering. But the thing is, you can combine them to reach an even better result. You can complement usability tests with short interviews and feedback gathering — with desk research. Choosing the right methods depends on the goals of your research and the capabilities of your team. 

Sometimes your team is in one hemisphere and your users are in the other one. In this case, you have to use remote tools that will serve you the same function. Fundamentally, the tests you’ll be having will be the same as corridor testing, except done using a video chat app, or phone calls, in the case with interviews. That’s how these processes are usually done in product companies. 

How to get ready for a research

You’ll need to set the following:

• problem,
• goal, 
• tasks,
• hypotheses.

You’ll also need to choose the segment you’ll be testing these hypotheses on and make a guideline (a list of questions) that will encompass all of the research tasks. 

Tip: don’t ask your respondent about what they would do, ask them about their past experience in a similar situation. This way, they won’t misinform you and you won’t end up developing your product in the wrong direction.

How to find respondents

You can invite your users to take a part in your interview or a survey by messaging them or sending them an email campaign. Combined methods will come in handy here as well: you can pick your internal respondents at the start of your research based on the quantitative data you have.

Want to hear the good news? Unhappy users are always happy to share their negative feedback. Make use of it! 

It’s hard to find people who have never used your product before and fit certain criteria at the same time. Still, how do you find them? In most cases, you’ll need to look for respondents of your desired segment individually. You can do it via official communities in Slack, Telegram, or Facebook, for example. You can also ask your users for a way to reach out to their co-workers. In any case, just don’t forget to take time looking for respondents into account when planning research. 

What questions to ask and how to ask them 

Write an introduction and think of some introductory questions. Tell people about what you do. Tell them about how important it is for you to make your product the best you can for your users. Make sure being a part of this research is a positive experience for your users. 

Basic rules: 

1. Ask open-ended questions. 
2. Avoid flattery.
3. Don’t let your respondents lie to you.
4. Don’t ask for opinions, ask about experiences. 
5. Talk less, listen more.
6. Be happy to receive critique.  

Don’t forget to thank the respondents who took part in your research! After your conversation (a must-do) and in the process, if that fits.

We highly recommend reading “The Mom Test” by Rob Fitzpatrick at least once and getting more practice interviewing if you want to learn asking new questions, answers to which will not misdirect you and your product.

How to make sure your statistical sampling is enough

It depends on the nature of your research. For quantitative research, it is based on the saturation concept: the sampling must be large enough for you to seek each possible meaning and every experience option important for your research. But remember not to make it too large as that will make your research counterproductive. The principle is simple: stop once the information you’re receiving starts to repeat.

Qualitative research are aimed at reconstructing the sense of your interviews, not a statistical generalization. 

How to filter insights out of the pile of gathered data

The sequence of methods and the framework set for analyzing the data you’ve gathered have to be picked based on your preferences, there is not a universal formula to it.

The earlier you start making your own research — the sooner you’ll learn to pick and combine frameworks that suit your needs. 

How to measure research effectiveness

It’s not easy to measure research effectiveness. Calculating if research results are worth the resources you spend on them is even harder. When do you consider your research successful? 

• You can look at the insights you’ve gathered. 

If all of your questions had been answered but getting them answered brought you no insights, it means your questions were defined incorrectly. That’s why it’s so important to filter out faulty material out of your research at the very beginning. 

In the best-case scenario, you’ll get a list of recommendations i.e. certain actions you’ll have to complete in order to improve your product. Your researchers will then pass the list to your product team, product managers will work that list by making additions and prioritizing it, then the list gets passed to the devs. That’s how it works in Skyeng. 

• The result of this research was Skyeng launching a new feature and getting a boost in metrics.

How to implement research results

Your researchers’ aim is to give your product team not some purposeless manuscript, but working conclusions and recommendations. Skyeng has developed a report template — a set of questions that their research team has to find answers to by the end of the research. After that, the product team is given a structured list of insights (with quotations and references to source materials — to keep the product team safe from the temptation of misinterpreting the data) that confirm or disprove the hypotheses.

If you’re doing many types of research and learn a lot of stuff from them, think of a way to share your discoveries with your team. Guys at Miro do it by holding regular Product Insights meetups, and Dashly has Fish Thursdays. Inner meetups like that let each team keep other teams informed about what interesting stuff they have discovered. In Skyeng, research results are discussed at stand-up meetings and presented to other teams’ clients. 

How to score research results

It’s important to figure out the most convenient way to score your results so that it’s always easy for you to come back to them and analyze the data from a new perspective, or even correlate these results with your new insights. 

Research may be repeated. Your product is evolving, consequences are changing, new problems arise, and sometimes you have to repeat your research on the same aspects of your product, this time — with new goals and new tasks. You may decide to make some of your research regular. 

If you research often, you’ll need to make a research catalog. Our colleagues at Skyeng have one as well. Theirs is a table organized in a way that allows to conveniently look for research on different topics. They keep notes and conclusions on each research in Notion.

Dashly uses Favro to prioritize hypotheses and score data related to them. You may choose any method you like. 

What are the risks?

• Getting low-quality data. 

You may forget to ask something due to the lack of experience, feeling stressed during the interview if it’s the first one you’re taking, or your respondent being too silent or too talkative. Making a list of questions may prevent it. 

• Finding out what you’ve already known. 

To avoid this, you have to know your product and inner processes well.

• Finding something that will bring you no benefit or something you can’t implement in your product.

Remember that your resources are limited and that it’s the user you’re making your product for, not your team.

• Wasting too much time and energy.

Before starting a research, compare what would be easier to do: research or an A/B test. If you don’t have enough data or insights, do research to add some new hypotheses to your hypothesis jar.

Checklist

• Look at your analytics. Does looking at all the data make you question “why”? Make a qualitative research on it! Or the other way around, say, you held a custdev session and found something new. Test this insight on the quantitative data you have. Find its weak spots and decide on the research methods. 
• Choose a user segment. 
• If you can do a desk research — do it. If you can make do with it — make do with it.
• Define a research goal. Divide the goal into tasks. If it’s still unclear to you what you want to get from your research — don’t even start it. 
• Make sure that the task of your research is not faulty (i.e. you know for sure how you can use the information you’ll get as a result of this research).
• Think of some hypotheses for every task.
• Discuss your hypotheses with your team. 
• Think of questions, answers to which will let you confirm or disprove each hypothesis.
• Don’t let your research take too long. It must have the shortest cycle possible.
• Don’t ask users about their theoretical experience, ask them about their real-life experience — not what they would have done, but how they are solving a problem now or how they were solving it before.  
• Talk less, listen more. 
• Don’t listen to flattery, aim to get more critique. 
• Make a list of questions: from general to specific, from easiest to hardest.
• Don’t be the only source of truth about your customers. Discuss research results with your team. 
• Compose cross-functional teams and synchronize your experiments so they don’t affect one another. 
• Think of a convenient way to score results of your researches. 

And remember what guys at Skyeng told us, and researches are their specialty: combined methods are an awesome thing, operational processes are your growth point, and internal researches are your source of insights. Amen to that. 

We thank Alice, Max, Kate, and Dimitri Ive for sharing their knowledge and experiences?