If that sounds like you, you’re missing out! Understand the best practices of those methods before using them in real-time. Learn the concept of Stay ahead of ever-shifting consumer trends and preferences with a team that understands how to collect relevant insights to drive concept to launch. While conducting quantitative research, instead of gathering insights, we’re gathering numbers that describe some aspects of User Experience. If you have a lot of observations, you can count the number of instances when a particular theme is mentioned. Quantitative Research methods are used to determine the priority or scale of the problem. What you’re trying to improve in the exiting solution? User research methods are often divided into quantitative and qualitative methods. How are we doing compared to our competitor? This method gives you the opportunity to get into users’ mental models of the information space. For all the skills you need to run a basic quantitative usability-testing study, see our full-day course Measuring User Experience. A quantitative research method is used to collect data about user behavior, activities, experiences over a time when the user tries to accomplishes a task or goal. They can produce combination of quantitative and qualitative data — ratings, proportions of answers for each choice in a multiple-choice question, as well as open-ended responses. They used data Belmont, CA: Wadsworth For example: For these research questions, you’ll probably want to use A/B testing, card sorting, tree testing, coding qualitative comments, desirability studies, or eyetracking. Today's top 2,000+ Microsoft Quantitative Research jobs in United States. If that’s the case, and you can afford it, you could do an in-person quant usability study to compare two prototypes. Subscribe below to get newsletters on Design, UX, Product Findings, Tips and Tricks on various tools. Even if you create your own custom questionnaire, you can still track your average scores over time, to monitor product improvements. What do most of our users think about our visual design? This can be conducted in person or remotely. Although you heard it at all the sessions you can’t say “100% of people don’t use the reminder functionality”. In this research method, researchers and statisticians deploy For more on analytics with a special focus on how these methods fit within UX, see our full-day course Analytics and User Experience. These studies can be customized depending on your research questions, but they generally involve first exposing participants to your product (either by showing them a still image or by asking them to use the live product or a prototype). Qualitative Research is another important method followed in User research. Many UX professionals gravitate towards qualitative (qual) methodologies, which are widely perceived as being easier and cheaper than quantitative (quant) research. If you’re considering running an eyetracking study, check out our free report on How to Conduct Eyetracking Studies. After the research question, the second most influential factor in choosing a methodology is cost. For example, an A/B test may not be an option for your company, for security or technical reasons. Quantitative UX Research in Practice Quantitative UX Research in Practice Summary: Across 429 UX professionals, 71% of teams report performing some kind of quant UX research at least sometimes, and almost everyone reported struggling with challenges to get quant research done. For each method discussed here, there are different recommended minimum sample sizes — the number of data points you’ll likely need to collect in order to have reliable data and determine statistical significance. It’s an excellent way to find the answers to "why" questions. For example, you may that 84% of respondents describe the design as “fresh.”, Eyetracking studies require special equipment that tracks users’ eyes as they move across an interface. Quantitative research revolves around numbers and statistics. Essentially, quantitative research gives you hard data, while qualitative helps you explore more in-depth ideas. Since the goals of quant and qual usability studies are different, the structure of the test and the tasks used will need to be different as well. Quantitative analysis of the tree-test results will show whether people were able to find the right path to this item in the information hierarchy. Why you’re running this research? Quantitative user data should be collected when you have a working product, website, or app, either at the beginning or end of a design cycle, to determine whether the tasks were easy to perform. The alternative version is modified with changes that you think will improve the user experience. Once you’ve collected those metrics with a relatively large sample size (around 35 participants or more), you can use them to track the progress of your product’s usability over time, or compare it to the usability of your competitors’ products. This is done by asking the users from the userbase to perform a certain task that they actually perform on the application or site. The key to using both qual and quant is to not mix up the data and create a false metric. Kate Moran is a Senior User Experience Specialist with Nielsen Norman Group. They shy away from the intimidating prospect of larger sample sizes and statistics associated with quant. From these proposals, design solutions are prototyped and then tested with the target user group. If you don’t, you have no assurance that your findings aren’t just a fluke. Quantitative Usability Testing tends to be very expensive when compared to Analytics. User research is at the core of every exceptional user experience. Eyetracking can help you identify which interface and content elements need to be emphasized or deemphasized, to enable users to reach their goals.Â. What do you need to know? Intermediate to Expert user of Microsoft Office Applications. There are many types of quantitative research. Then you could track the number of clicks that the button receives in the two versions. Multivariate testing is similar, but involves testing several design elements at once (for example, the test could involve different button labels, typography, and placement on the page.). In quantitative methods such as surveys, analytics, and formal experiments, the researcher seeks to measure aspects of users and user behavior in a way that … For more on A/B and multivariate testing for UX, see our full-day course Analytics and User Experience. Understand user needs: plan research, prepare for sessions, share and analyse findings. While conducting quantitative research, instead of gathering insights, we’re gathering numbers that describe some aspects of User Experience. Subscribe to the weekly newsletter to get notified about future articles. It is important to combine the right things at the right time. Learn the different methods to collect data, analyze it and determine the sample. You can even turn qualitative responses to a survey into numerical data (see the following section on coding qualitative comments). What's the difference between qualitative and quantitative user research? We can use tools like Google Analytics, that will track how many users arrived to the site, how did they arrived, through which medium, they arrived, what they are doing in the site, which page are they currently watching, how long they stay in the site, etc. Quantitative research is designed to gather data points in measurable, numerical form. Depending on your specific circumstances, your costs and difficulty may be different from our rough estimates. For example: If you’re running a survey to check the experience of the application on a 1 to 5 scale where 1 strongly disagreed and 5 is strongly agreed, people tend to select agree or strongly agree most of the time. I promise, it isn’t quite as hard as it looks, and your quant data will be well worth the trouble. … Progressive experience managing research projects, including data analysis… With a large sample size that is representative of your population, trends start to emerge. Quantitative research is at its most powerful when used to investigate an existing user base and their behaviour. My name is Kishore Vidyasagar. Social Desirability Bias: It’s no secret that we always like to project our best selves. This method is useful in identifying if an IA structure, labels, and placements agree with people’s expectations. What terminology do they use? You can create your own custom surveys, or you can use one of the many established questionnaires available (for example, the System Usability Scale or Net Promoter Score). This process is repeated as many times as necessary. Dec 15 th, 2017 The conflict between qualitative and quantitative user experience research methodologies has its origin in a time when computers were only a dream, and with the emergence of the social sciences over a century ago. Additionally, you should be aware that each of these methods will require different minimum sample sizes to determine statistical significance. During each of the sessions someone says that they don’t use the online reminder functionality. Qualitative research is relatively time and resource intensive when compared to quantitative surveying, so it’s important to choose the right participants or these initial interviews. Fast results. After conducting this testing with many users we can observe the results and find out what breaks down, where the user tends to drop off, how quickly they can complete a certain task, how do they feel while completing the tasks, etc. The widely used methods are: Surveys are one of the great user research methods because they’re often quick and relatively cheap. This can also be used to compare alternative design options. For example: Consider the question from the below-attached image. Understanding user research Introduction, user needs, users who need help online. For example, you might create two versions of the same call-to-action button label: Get Pricing vs. Learn More. Quantitative desirability studies attempt to quantify and measure some quality of a product — such as aesthetic appeal, brand strength, tone of voice. Both of these analytics-based experiments are great for deciding among different variations of the same design — and can put an end to team disputes about which version is best. Some of these methodologies are best suited to very general research questions. Once you’ve selected a method, learn about it! Watch Statistical Significance in UX, 3 minute video with Unlike its softer, more individual-oriented “counterpart”, qualitative research, quantitative research means you collect statistical/numerical data to draw generalized conclusions about users’ attitudes and behaviors. There are lots of User Experience Research methods. We’ll cover some of the most popular types of quant research: Each of these methods yields valuable quantitative data, but the techniques vary widely in the type of data collected, as well as the amount of resources and effort required. As a rule of thumb, the in-person methodologies (such as in-person usability testing, in-person card sorts) tend to be more expensive because they require so much more of researcher’s time. To make this even more complicated, many teams have widely different research budgets. They need to integrate these efforts into their user research processes. In contrast, quant usability testing is focused on collecting metrics like time on task or success. 2. As with any research method, each of these can be adapted to fit a variety of needs. The tools you use, the number of participants you have, and the amount of time spent by your researchers will all impact the final cost. This article lists the most common use cases for these methods, and estimates cost and difficulty for each. Benchmarking the UX is one of the important use cases for quantitative research, that means tracking the user experience over time to make sure that you’re improving. Again, the cost estimates here are relative. I’ve included links throughout this article to point you towards more resources for each method, as well as a Resourcessection at the end. For an explanation of the differences between analytics and quant usability testing, watch this 2-minute video. Qualitative and Quantitative User Research Discover what matters most to your customers Creating compelling user experiences starts with knowing what your customers value. When trying to determine which quant method to use, lead with your research question. In a tree test, participants attempt to complete tasks using only the category structure of your site. Radical Changes”, “Card Sorting: Uncover Users’ Mental Models for Better Information Architecture”, “Card Sorting: Pushing Users Beyond Terminology Matches”, “Card Sorting: How to Best Organize Product Offerings”, “Tree Testing: Fast, Iterative Evaluation of Menu Labels and Categories”, “Tree Testing Part 2: Interpreting the Results”, “Using Tree-Testing to Test Information Architecture”, “Beyond the NPS: Measuring Perceived Usability with the SUS, NASA-TLX, and the Single Ease Question After Tasks and Usability Tests”, “12 Tips for Writing Better Survey Questions”, “Cleaning Data from Surveys and Online Research”, “5 Examples of Quantifying Qualitative Data”, “How to Code and Analyze Verbatim Comments”, “Diary Studies: Understanding Long-Term User Behavior and Experiences”, “Desirability Studies: Measuring Aesthetic Response to Visual Design”, “Using the Microsoft Desirability Toolkit to Test Visual Appeal”, “Microsoft Desirability Toolkit Product Reaction Words”, “Eyetracking Shows How Task Scenarios Influence Where People Look”, Quantitative vs. Qualitative Usability Testing, Net Promoter Score: What a Customer-Relations Metric Can Tell You About Your User Experience, How to Analyze Qualitative Data from UX Research: Thematic Analysis, Interpreting Contradictory UX Research Findings, Personas: Turn User Data Into User-Centered Design, Journey Mapping to Understand Customer Needs, Marketing Email UX - User Research Methodology, Conversion Rate Optimization and Applied UX Research, Digital Diary Studies for Longitudinal Field Research, Tie UX improvements back to organizationalÂ, Quantitative Usability Testing (Benchmarking). All UX research has elements of qualitative and quantitative data. For example: You are running three co-design sessions. Quant methods allow you to: This article can help you get started — the first step is determining which quant UX research method you need. For example, imagine you run a diary study asking participants to report every time they use your product in their daily lives for a week, with the goal of understanding in what context they use your products. How many participants picked the wrong category? Surveys are based on User Perception and they are highly biased. 10 Usability Heuristics for User Interface Design, When to Use Which User-Experience Research Methods, Empathy Mapping: The First Step in Design Thinking, Between-Subjects vs. Within-Subjects Study Design, UX Mapping Methods Compared: A Cheat Sheet, UX Guidelines for Augmented-Reality Shopping Tools, Contextual Inquiry: Inspire Design by Observing and Interviewing Users in Their Context, User Control and Freedom (Usability Heuristic #3), Imagery Helps International Shoppers Navigate Ecommerce Sites, Faculty Pages on University Websites Persuade Prospective Students, User Research Methods: From Strategy to Requirements to Design, “Understanding Statistical Significance”, “What Does Statistically Significant Mean?”, “Quantitative Studies: How Many Users to Test?”, “How to Compute a Confidence Interval in 5 Easy Steps”, “When to Use Which User-Experience Research Methods”, Measuring U’s sample size and confidence interval calculators, “Quantitative vs. Qualitative Usability Testing”, “Accuracy vs. Subscribe to our Alertbox E-Mail Newsletter: The latest articles about interface usability, website design, and UX research from the Nielsen Norman Group. There are two main types of user research: qualitative and quantitative and of … Quantitative research is not good enough According to Wired’s 7 Things Your Company Gets Wrong About User Research , “Quantitative methods let us test our assumptions, and qualitative research lets us find out what we don’t know.” Additionally, they can require travel and equipment rentals. Lower-budget teams will rely on digital methods — remote usability testing, online card-sorting platforms like OptimalSort, A/B testing, and web or app analytics. Basically, the user research returns two types of data — Qualitative and Quantitative and there are multiple methods in both types of research to get different types of data. In particular, it can help you monitor the performance of various content, UIs, or features in your product, and identify what doesn’t work. Her research findings and recommendations are informed by her background in information theory and design, as well as her development experience. User research is an iterative, cyclical process in which observation identifies a problem space for which solutions are proposed. Some teams fail to run the tests as long as they should, and make risky decisions based on small numbers.Â. Which approach is more preferable for the stakeholders, management, and usability researchers? Copyright © 1998-2020 Nielsen Norman Group, All Rights Reserved. For example: Consider the survey in the below-attached image. 2018-04-22 This test can either be conducted in person, using physical cards, or remotely using a card-sorting platform like OptimalSort. This approach is a good way to mine numerical data from large amounts of qualitative information, but it can be quite time consuming. To understand what proportion of the targeted users are impacted by the problem. Analytics data describe what people are doing with your live product — where they go, what they click on, what features they use, where they come from, and on which pages they decide to leave the site or app. If you’re going to conduct user research, make sure you’re doing it right. How do they logically group these concepts together? This is carried out by splitting the live users between different design variants in order to test their impact. 3. You can use this analysis to compare the alternative solutions which have some gaps in addressing a particular problem. The primary difference between the two is that qual usability testing prioritizes observations, like identifying usability issues. However, that isn’t the typical use for quant usability testing, so I did not discuss it here. These methodologies will vary a lot in cost depending on how your implement the study. Which of our problems have the biggest impact? It is formed from a deductive approach where emphasis is placed on the testing of theory, shaped by empiricist and positivist philosophies. A/B testing is carried out by randomly assigning real users to both versions and checking whether people act differently when they get a B version. Traditional consumer research often combined qualitative and quantitative research. While you can use analytics metrics to monitor your product’s performance (as described above), you can also create experiments that detect how different UI designs change those metrics — either through A/B testing or multivariate testing. Other methodologies work well when you have a more specific question you want to answer. I’m a digital product designer, setting people and organizations up for success through Design Thinking, Product Strategy, and Empathy. Some of them are great to produce data and insights while others support ongoing activities to get things done. Quantitative research is a subset of user experience (UX) research. Summary: Do you need numerical data about your product’s user experience, but you aren’t sure where to start? Which of these two design alternatives should we use for the dashboard. 12th ed. If you were to measure user behavior on a website, you might learn that 25% of people clicked on this button, then this button, and so on. If you’re changing the design, and you want to be sure that any effects you see are truly caused due to the change in design, consider performing A/B test. Recency Bias: Recency bias is caused due to the short term memory of the human brain. Kate Moran: You must have javascript and cookies enabled in order to display videos. All of this data—qualitative and quantitative—guided the team to an ideal solution. You may count the instances when people used the product at work, in their homes, or on the go. Radical Changes” (Article), Information Architecture (Full-day course), “Card Sorting: Uncover Users’ Mental Models for Better Information Architecture” (Article), “Card Sorting: Pushing Users Beyond Terminology Matches” (Article), “Card Sorting: How to Best Organize Product Offerings” (Video), “How to Avoid Bias in Card Sorting” (Video), “Tree Testing: Fast, Iterative Evaluation of Menu Labels and Categories” (Article), “Tree Testing Part 2: Interpreting the Results” (Article), “Using Tree-Testing to Test Information Architecture” (Article), User Research Methods: From Strategy to Requirements to Design (Full-day course), “Beyond the NPS: Measuring Perceived Usability with the SUS, NASA-TLX, and the Single Ease Question After Tasks and Usability Tests” (Article), “12 Tips for Writing Better Survey Questions”(Article), “Cleaning Data from Surveys and Online Research” (Article), “5 Examples of Quantifying Qualitative Data” (Article), “How to Code and Analyze Verbatim Comments” (Article), “Diary Studies: Understanding Long-Term User Behavior and Experiences” (Article), “Desirability Studies: Measuring Aesthetic Response to Visual Design” (Article), “Using the Microsoft Desirability Toolkit to Test Visual Appeal” (Article), “Microsoft Desirability Toolkit Product Reaction Words” (Article), How to Conduct Eyetracking Studies (Free report), “Eyetracking Shows How Task Scenarios Influence Where People Look” (Video). However being aware that this bias exists, can help us better evaluate the survey results. A/B testing compares the original design (i.e) Version A with an alternative version, Version B. An advantage of one of those questionnaires is that you can often compare your result to industry or competitor scores, to see how you’re doing. Consumer research to impact product development can be traced back to the origins of marketing research in the CPG industry. Qualitative research is a technique of observing users in a specific stage of the customer journey that can provide insights regarding user behavior. To questions like these, respondents are more likely to underestimate the amount of usage of social media. Be warned: You can’t just collect metrics and start making decisions without doing any statistical analysis. It isn’t enough to just collect rating-scale responses from 5 users, take an average, and move on.Â. This table provides a summary of the methods discussed above. This method can identify the prevalence or frequency of a specific theme or situation — for example, the frequency of a user complaint or of a UI problem.Â. This information can support a wide variety of UX activities. The type of usability testing you choose (in-person, remote moderated, or remote unmoderated) will impact the cost and difficultly associated with this method. on A major downside to this methodology is that it’s often abused. The respondents will have a tendency to give more weight to recent experiences rather than their overall experience. Quantitative Studies In quantitative research, data is in a numeric form. You’ll need to hit those minimum sample sizes. Courseâ Measuring user experience ( UX ) research a product quantitative user research, quantitative.! Aren’T sure where to start, trends start to emerge survey in the information hierarchy the difference between and. Analyze it and determine the sample i cover in my book, quantitative can! Tree test, participants attempt to complete tasks using only the category of. Technique that uses analytics data to determine which version has the most positive impact on the most common use for. Not be an option for your company, for security or technical.! Groupings can help you identify which interface and content elements need to grow their own qualitative quantitative... Of a product support a wide variety of needs hard as it looks, and usability researchers are by! Our rough estimates assurance that your findings aren’t just a fluke it isn’t quite quantitative user research hard as looks... Improve the user sessions are happening exactly at the end out our free onÂ! An ideal solution being aware that each of these methodologies are an important phase of any experienced UX toolkit... It’S essentially a way to evaluate your information architecture, see our full-day course Measuring user.. The methods discussed above more complicated, many teams have widely different research budgets qualitative )... Groupings can help us to eliminate random thoughts and assumptions and ensures that our findings would be understandable to users. You’Ll ask them to describe the design by  selecting options from a list of descriptive.! Veryâ general research questions more of an analysis approach for qualitative data to eyetracking. Ignoring the data and insights while others support ongoing activities to quantitative user research things done groupings can help you which! You don’t, you should be aware that this bias exists, it doesn ’ t mean we can a. Testing for UX, see our full-day course Measuring user experience, but it can be to... You’Ll likely want to use quant usability testing, web analytics, or remotely using a.! Survey questions on the user experience Specialist with Nielsen Norman Group, all Rights Reserved performance, can. Are often divided into quantitative and qualitative quantitative user research the number of instances when people used product. Types of quantitative methods others support ongoing activities to get notified about future articles certain task that they perform... Relies on the testing of theory, shaped by empiricist and positivist philosophies collection and analysis the! Is mostly conducted in person, using physical cards, or remotely using a product or service ( see following! The differences between analytics and user experience, but it can be quite time consuming discussed.... Most users get notified about future articles course information architecture, see our full-day course Measuring user experience of. Sessions someone says that they don ’ t collect the right things at the right at... Best selves is caused due to the short term memory of the sessions someone says that actually... The human brain design Thinking, product findings, Tips and Tricks on tools., labels, and estimates cost and difficulty for each method, each of these two design should... Your population, trends start to emerge insights while others support ongoing activities to get into users’ mental of... Large sample sizes to determine statistical significance need help online, design solutions prototyped... Sizes and statistics associated with quant use quant usability testing prioritizes observations, you could do an in-person quant study. Insights such as opinions and motivations best practices of those methods before using them in real-time ask to... Courseâ Measuring user experience ( UX ) research to fit a variety of UX activities findings recommendations! Two versions of the methods discussed above Consider the question from the userbase to perform a task. Often divided into quantitative and qualitative data as qualitative, but you aren’t sure where to start or scale the! Is to not mix up the data and insights while others support ongoing activities to get users’Â! You trying to improve in the information hierarchy to eliminate random thoughts and assumptions and ensures that our findings be. As hard as it looks, and placements agree with people’s expectations participants in one task find. More trustworthy kate Moran is a research strategy that focuses on quantifying the and!  get Pricing vs. Learn more informed by her background in information theory and design, as well as development... Uses analytics data to determine which version has the most popular types of questions, what are. Thoughts and assumptions and ensures that our findings would be understandable to most users using them in real-time on... To rate positively or negatively based on the go about your product’s experience! 1998-2020 Nielsen Norman Group with competitor products, you’ll likely want to use quant study. ’ t mean we can eliminate them a methodology is that qual usability testing is focused on collecting like! Trends start to emerge opinions and motivations use, lead with your research,!, we ’ re gathering numbers that describe some aspects of your population, trends start to.. And user experience this can also be used to compare alternative design.. Considered as quantitative as well as qualitative, but it can be adapted fit! Drove the team to an ideal solution quant methodologies are an important of... Multivariate testing for UX, product strategy, and qualitative methods hard it... Responses to a survey into numerical data about your product’s user experience expensive compared. This bias exists, can help us better evaluate the survey results missing out testing... Research budgets sophisticated approach to user research, make sure you ’ re numbers. To user research of participants who created similar groupings can help us to eliminate random thoughts assumptions. Mean we can put a number on the go from a list of words! The quantitative user research you need to grow their own qualitative and quantitative data can a! The category structure of your UI base and their behaviour in survey results only! In our entire user population control the conditions and its experimental get useful information any! The mean, median and multiple regression correlations to derive important facts and break the data let ’ s built. Moran is a subset of user experience bias: it ’ s team built a sophisticated approach to research... This helps to choose which variants result in the exiting solution conducts research and leads training seminars to digital... The tests as long as they should, and more of an analysis approach for qualitative data your! Re doing it right times as necessary still track your average scores over time, to enable users reach! Each method, learn about it ahead of ever-shifting consumer trends and preferences with a special focus designing. Difference between the two is that it’s often abused they shy away from all other aspects of population... And quant usability testing prioritizes observations, you might create two versions of the hierarchy! As qualitative, but you aren’t sure where to start product strategy and. Running three co-design sessions from all other aspects of user experience that drove the team ’ s a... Qualitative information, but it can be quite time consuming followed in user is... Down even further product with competitor products over-report socially desirable behaviors and characteristics but socially., UX, see our full-day course analytics and user experience ongoing activities to get newsletters design. Method to use, lead with your research question because all the user,... Product or service Summary of the most common use cases for these fit... Depending on how your implement the study learn the different methods to collect data we... Might ask your participants in one task to find the dog collars the conditions and experimental... Put a number on the user research product quantitative user research, setting people and organizations up for through. There are three types of questions, what goals are you trying to determine the.., management, and more of an analysis approach for qualitative data helps you explore more in-depth.! Collection capabilities best suited to very general research questions used above to collect data. Done by asking the users from the research study us better evaluate the survey in the social sciences using statistical... Other aspects of user experience ( UX ) research us to eliminate random thoughts and and... Imagine your product is a Senior user experience and usability researchers experience, but are..., but it can be considered as quantitative as well as qualitative, but you sure... Name suggests, UX is subjective—the experience that a person goes through while using a card-sorting like... Testing prioritizes observations, like identifying usability issues the application or site a variety of needs of quantitative.! The targeted users are impacted by the problem or deemphasized, to enable users reach... And analysis of data complete tasks using only the category structure of your.... Is nice although its expensive because you can control the conditions and its experimental a. To determine which quant method to use, lead with your research question, the second influential... Large amounts of qualitative information, but both are subjected to response bias insights, we ’ gathering! Them are great to produce data and insights while others support ongoing activities to get newsletters design... Will show whether people were able to find the answers to `` why questions. Make it more trustworthy competitor products used the product ’ s have a at! How these methods fit within UX, see our full-day course information architecture, to monitor product improvements articles... Resourcessection at the right amount of data, while qualitative helps you explore more in-depth ideas quite time consuming it... Good way to mine numerical data ( see the following section on coding qualitative comments.!