Hackers Guide to Advanced Research Methodologies Derive the research insights that matter with the most powerful suite of research tools to help you make better decisions Author Dan Fleetwood Pres.Longitudinal research Researchers and insights teams now need to realize that there is now a lot of data collected on things that have now gone ‘tonedeaf’ as a result of economic certainty and conservative consumerism. While longitudinal (or tracking) studies have already played such a significant role in capturing insights on awareness and purchase consideration, many factors in 2020 could affect the overall demand for particular products. Regardless of how well and business or industry is thriving during these times, researchers will need to continue monitoring and tracking data to see and understand what is changing over time (and what is NOT) for businesses to better prepare for the future.
Hacker's Guide to Advanced Research Methodologies Derive the research insights that matter with the most powerful suite of research tools to help you make better decisions! Author: Dan Fleetwood - President, Research & Insights Platform, QuestionPro Hacker’s Guide to Advanced Research Methodologies Table of Contents ● Introduction ● Advanced research methodologies & techniques - survey platform ○ ○ ○ ○ ○ ● Advanced research question types ■ Conjoint analysis ■ MaxDiff analysis ■ Anchored MaxDiff scaling ■ Card sorting ■ Text highlighter ■ Van Westendorp ■ Semantic differential scale ■ Heatmap analysis ■ Hotspot testing Advanced research logic ■ Advanced question & answer randomization ■ Block randomization ■ Distributed logic quotas Advanced research report management ■ Weighting & balancing ■ Data quality Advanced research analysis ■ Sentiment analysis ■ TURF analysis ■ Gap analysis ■ Correlation analysis Additional advanced research software tools ■ Auto translate ■ Integrations Clients we have served Hacker’s Guide to Advanced Research Methodologies Introduction With no ability to foresee the COVID-19 pandemic and its effects on all of us in both our personal and work life before the new year, organizations have had to be agile to navigate, 2020 Despite the halt, perhaps even a ‘reshuffling of the deck’ of what we call our day-to-day life, businesses’ rules have not changed: stay up to speed on major trends and avoid the risk of being left behind That rule still strongly applies to market research, as we need to meet other organizations’ needs to understand and understand how to maximize growth for our customers and partners. Let’s look back on some of the new market research trends that have emerged *Note that while many of these trends have come as a direct response to COVID-19, many trends, before the new year, were predicted to thrive, both in 2020 and beyond These trends are despite COVID-19’s direct impact on the marketplace and based on certain market research patterns from the 2010s. Shift of market research towards new methods There has been a marked shift in market research to include newer research techniques to unlock insights Some of the major shifts in market research in 2020 and beyond, are: 1) Longitudinal research - Researchers and insights teams now need to realize that there is now a lot of data collected on things that have now gone ‘tone-deaf’ as a result of economic certainty and conservative consumerism While longitudinal (or tracking) studies have already played such a significant role in capturing insights on awareness and purchase consideration, many factors in 2020 could affect the overall demand for particular products. Regardless of how well and business or industry is thriving during these times, researchers will need to continue monitoring and tracking data to see and understand what is changing over time (and what is NOT) for businesses to better prepare for the future. 2) Measuring perception and sentiment - Certain brands and products will thrive due to these times, while others won’t This is a result of the sweeping changes in attitudes and behaviors among marketplace consumers Something vital to consider is that many of these changes won’t be temporary; purchase behaviors in the masses could change for good Since customers will be exposed to new ways of purchasing products, many of these consumers have the potential to find out that these new ways may be better suited for their day-to-day lives. Hacker’s Guide to Advanced Research Methodologies It will be essential for researchers to consider all of the factors that affect purchasing processes and brand perception and awareness to infer the next steps and directions for clients to thrive during these times. 3) Surge in online qualitative research & video focus groups - Over the last decade, online qualitative research has gone from a novelty or a ‘nice-to-have’ into an essential research methodology across the market research industry In general, traditional qualitative research is declining because it is so time-consuming and expensive to designate physical spaces to host these qualitative research activities. With added features such as being able to share video responses and recording participants’ interactions on the platform, market researchers can now reach out to their audiences in a shorter period and probe the ‘why’ about a brand or a product now faster than ever Plus, live explanations of thoughts and feelings are more natural, leading to even more feeling, sharing, and connecting with your audience. 4) Shorter but smarter surveys - Short surveys and quick polls, more than ever, are popping up on websites, phones, social media, and chatbots One reason for this is the growing fatigue for respondents to answer longer surveys and a continued increase in utilization in social media or communication apps on electronic devices, particularly mobile phones Apps for social media and communication account for at least 50% of all apps used worldwide With that, researchers can intercept and collect more data in real-time at a higher rate. 5) Incorporation of Artificial Intelligence in data collection - Researchers are pushing for new data collection methods to be more seamless and automated processes As mentioned, shorter surveys are a direct result of this But more than that, the demand for data delivery to go straight from laptops to high-level decision-makers is rising. Areas for data collection to become more seamless include analysis at open-ended text responses These capabilities will enhance time-efficiency for researchers and create more value for clients and break down the ‘why’ when purchasing behaviors Another example of this is in Sentiment Analysis – where marketers and researchers will be able to decipher positive or negative responses better. QuestionPro is at the forefront of facilitating research and has consistently stayed above the curve to bring simpler but powerful research to brands, organizations and researchers alike. Hacker’s Guide to Advanced Research Methodologies Advanced research methodologies & techniques - survey platform Using our research platform will provide you unmatched flexibility in how you collect actionable insights for your brand Leverage our DIY research tools, smart survey logic, and more, that solve your biggest research challenges and align with your brand and research goals. Our research platform solution matrix will help you with: ● Choice-based analysis - Simulate market conditions and track and monitor preferences by using conjoint analysis, max diff analysis, card sorting, etc. ● Market trends - Monitor market dynamics and stay insulated from shocks which will allow you to constantly be above the curve. Hacker’s Guide to Advanced Research Methodologies ● Market segmentation - Monitor behavior across various geographies and demographics Make informed decisions on positioning based on value. ● Competitive benchmarking - Identify gaps in relation to your biggest competitors and clamp down on your biggest differentiators. ● Purchase behavior - Identify purchasing behavior including various aspects such as price sensitivity and trade-offs to assign optimum pricing models. ● Product & service research - Listen and act on what your customers and audience want, to stay above the competition and reduce churn. ● Academic research - Capture academic insights as an academician or a student with unlimited flexibility that no other tool provides. ● Ad testing - Monitor how your brand’s messaging resonates with your audience and impacts your referencability. ● A/B tests - Test everything across multiple vectors to make the most informed decisions that are representative of the larger population. Advanced research question types The QuestionPro Research suite provides you the ability to use advanced research question types to be able to solve any research problems that you may have. Conjoint analysis Conjoint analysis is a choice modeling research method to understand how people make purchasing decisions In the real world, we often encounter situations when we have to make tough choices between various alternatives The conjoint analysis question helps us understand what is essential for your target audience It involves how they make trade-offs and what essential features they are not willing to let go. The conjoint survey question is an advanced question type that market researchers use to present many combinations of product attributes like features, cost, brand, etc Based on the respondents' answers, market researchers can find out the most liked features by customers and get an idea of pricing Many times a purchase involves evaluating several parameters that make it complicated In such a situation, running a conjoint analysis survey can help understand customer psychology. Hacker’s Guide to Advanced Research Methodologies Types of conjoint analysis commonly used in surveys Choice-based conjoint analysis: This type of analysis question asks respondents to imitate their purchasing behavior while answering the survey The respondents submit responses based on the actual products they would choose in real-life, given specific prices and features. Types of designs for the discrete choice model QuestionPro offers the below design types for conjoint analysis using the discrete choice model: ● Random: This design is a random sample of the possible attribute levels The survey software will create a unique combination of attributes for the number of tasks per respondent To know what choices will be presented when your survey is deployed, you can run a conjoint concept simulator. ● D-Optimal: While designing experiments for estimating statistical models, optimal designs estimate parameters without bias, and with minimum-variance D-optimal design runs a set of tests to optimize or investigate the subject under study The algorithm creates an optimal design for the tasks per respondent and sample size. ● Import design: T his design type allows designs in the SPSS format to be imported and used by the discrete choice module For instance, you can import fractional factorial orthogonal designs and use them in QuestionPro surveys. Adaptive conjoint analysis: This type of conjoint analysis is used in surveys when there are many product features Researchers generally use it to identify key features that should be included in the product and not the best choice for determining the price For instance, the surveyor asks respondents to select their relative preference from several attributes They assess each pair on a grade point scale. The choice-based conjoint analysis, also known as discrete-choice conjoint analysis, is the most commonly used type of conjoint analysis survey question. Data analysis of conjoint survey question With QuestionPro surveys, you can generate a conjoint analysis report and filter the survey data The statistical analysis report consists of the below tabs. 1) Attribute importance - This tab shows which attributes are more important than others and by what percentage. Hacker’s Guide to Advanced Research Methodologies Learn more about a ttribute importance. 2) Profiles - It is a set of attributes with different levels The conjoint analysis software shows respondents various combinations of product features, prototypes, mockups, or pictures created from a combination of levels Each example is similar enough to be close substitutes but different enough to be distinguishable. Learn more about c onjoint analysis profiles. 3) Market simulation - Using this feature, you can forecast the market share of new products that don't exist today You can also measure the gain or loss in the market share based on the existing products' changes The conjoint analytics tool simulates the market share of the products to establish a baseline Then, you can see how the market share changes depending on new products and configurations. Learn more about m arket segmentation simulators. Hacker’s Guide to Advanced Research Methodologies 4) Estimated brand premium - In many cases, customers are willing to pay extra for a product with the same features as others but with a different brand This report finds out how much premium a customer will pay for a brand. Learn more about e stimated brand premium. 5) Price elasticity - It is the proportional change in demand for a product for change in attributes and price To view this report, map attribute type to brand & price for each level. Learn more about f eature attribute type. Additional asset: T he Hacker's Guide to Conjoint Analysis MaxDiff analysis Maximum Difference Scaling is a very effective method of establishing the relative priority attached by an audience to a large set of items (up to 30) These items might be: Hacker’s Guide to Advanced Research Methodologies 1) Determine the features or benefits of a service - Before investing time and money into a new feature or rolling out a new service, ensure that the target market is likely to buy and use the new offerings. 2) Discern areas for potential investment of resources - Take stock of the market trends without spending excess money and time on an expensive market research expert—just ask the target consumers! 3) Learn about consumer interests and activity preferences - There’s no money in attempting to manufacture genuine interest; ensure the business offers products, services, and activities based on consumer attraction. 4) Test potential marketing messages for a new product - The organization’s messaging concepts should resonate with the entire market for maximum impact—test it out with a statistically significant sample size to see if the brand voice resonates with the target audience. 5) Know which products or services are used - Reinvest in the products and services that the organization’s customer base is already investing in for maximum ROI. Key terms and concepts MaxDiff scaling is a powerful choice-based modeling method To be able to harness the full potential of this analysis method, it is important to know what these key terms and concepts, mean: 1) MaxDiff Analysis - MaxDiff Analysis is shorthand for Maximum Differential Analysis, also known as Best-Worst scaling, wherein respondents choose the best value and worst value from a set of attributes (and the attributes repeat in intentional permutations based on the logit model). I.e a large Bank determining which major credit card company to partner with for their customer credit card offering. 2) Attributes - The options that need to be chosen from (features of a product, answer options). I.e Preference of use (most preferred to use / least preferred to use). 3) Maximum attributes - The number of attributes that will be tested in the survey. 4) Task - The current screen the respondent sees (you may have up to tasks per survey). 5) Attributes per task - the number of attributes that will be shown on the screen at a time. 6) Logit model - Utility estimation model used to calculate the share of preference (See more in How to set it up in QuestionPro). You can view the MaxDiff analysis (best-worst scaling) in the following format: Hacker’s Guide to Advanced Research Methodologies Learn more about how to use heatmap analysis in your surveys. Additional resource: W ebinar on how to conduct heatmap testing Hotspot testing A hotspot question is used in the surveys to get feedback on images It consists of an image, and respondents are asked whether they like or dislike a specific area of an image There are no specific correct or incorrect answers It is used just to know what part of the image is most popular among the respondents. This question in a survey shows one or more selected regions to respondents On clicking that area, respondents will be able to share their views both graphically and textually A pop-up box appears over the questionnaire with a 'thumbs up' and 'thumbs down' button for 'like' and 'dislike.' Respondents can also leave their comments in the text box It will give researchers more insights about what they think of the image Hotspot question type offers options to either force respondents to answer the question or request a response You can also customize the border color of the area selected, which needs participants' feedback. Hacker’s Guide to Advanced Research Methodologies Learn how to use hotspot testing in your research. Additional resource: W ebinar on how to conduct hotspot testing Advanced research logic QuestionPro's logic engine is the most advanced in the industry We can piping, extraction, branching, looping and logic based on criteria of questions and metadata like custom variables passed in from other sources All this using a point-and-click interface - without scripting or coding The research platform also allows you the ability to manage conditional block rotation & randomization, conditional looping, and conditional extraction. The logic in our survey research platform that makes us the choice for researchers all over the world, are: Advanced question & answer randomization A randomizer question type allows you to present a set of questions to your respondents randomly. Survey creators can use it to eliminate possible order bias from your respondent group. Hacker’s Guide to Advanced Research Methodologies Advanced randomization in surveys is a technique by which the answer options are presented to each respondent in a different order It is used to overcome order bias - a behavior in which respondents tend to select the first option To tackle this, survey creators present answer options such that they need to spend some time going through all the options and choose an honest answer. The platform allows you the flexibility to randomize questions and answers in multiple methods so that the survey data collection is as accurate as possible. Question randomization A randomizer question lets you display survey questions randomly to the respondents You can configure question randomization at two levels and alter question order: ● Randomly display one question ● Randomize the sequence of a list of questions Learn how to use question randomization in your surveys. Answer randomization You can randomize the answer options in your surveys by the following methods: ● Simple - Randomize the order of all options It is classified into three categories. ○ Default - Answer items are displayed in the same sequence as set by the survey creator. ○ Ascending - Answer options are sorted in the ascending order alphabetically. ○ Descending - Answer options are sorted in the descending order alphabetically. ● Random - Randomize the order of few options out of all the options. ● Advanced randomization - Create subgroups of answer choices and display the groups randomly. Learn how to use answer randomization in your surveys. Block randomization The quality of data depends on various factors Some of them are the type of questions asked, the depth and breadth, number, topics covered, research method, etc Another factor that affects data quality is the order of questions The sequence of items can impact the way a respondent thinks and makes choices Hence, there is a high possibility that the respondents may feel inclined to answer in a particular direction Thus, the survey creator unknowingly affects the way a respondent answers the Hacker’s Guide to Advanced Research Methodologies questions This effect is known as order bias and affects the results of the research Reports generated from the data inflicted with order bias generate inaccurate insights. A block randomizer lets you select a group of questions that must be asked to respondents in random order You can keep one or more blocks fixed and randomize the order of others Alternatively, you can also display all blocks of questions randomly You can also randomize questions within a survey block. Learn more about how to set up b lock randomization in your surveys. Distributed logic quotas The distributed logic quota is a combination of different types of quotas Quota control helps you limit the number of responses to your survey If the survey uses more than one quota control technique, it is considered as distributed quota control Using this method, you can set up quota based on answers to multiple questions, custom variables, number of responses for each option, geo-location, and more. There are two approaches to quota control - pessimistic and optimistic. Pessimistic quota control does not count an individual as passing quota until they have completed a survey So we can be sure that all quota cells are filled If a user sends out many survey invitations, there is a strong chance that there will be an over quota situation due to too many completions. Hacker’s Guide to Advanced Research Methodologies QuestionPro implements the pessimistic quota system You can apply a distributed logic quota on multiple questions. Optimistic quota control counts anyone who starts the survey but does not finish it It has a lesser chance of being over quota, but a higher risk of being under quota To avoid going over quota, the survey creator should manage the timing of survey distribution such that all respondents not receive the invitation at the same time It would also help researchers have better control over the responses. Types of advanced quota control in surveys Types of advanced quota control in surveys are: 1) Response quota - This type of quota control is used to limit the number of responses to your survey Once the limit for the total number of responses is achieved, respondents will not be able to submit the survey. 2) Complex quota control - This type of quota puts a limit on the number of responses to multiple questions and custom variables It is also used as weighted quota control You can set the percentage distribution of responses across the options of a specific question. Hacker’s Guide to Advanced Research Methodologies 3) Custom variable quota control - You can set up quota based on the responses to multiple custom variables When the custom variable is set with a specific value for a few times, respondents will not be able to answer the survey further. 4) Advanced quota control - Use this method to limit the number of responses based on the selection of options, custom variable, geo-location, email list, and device type. 5) Dynamic quota control - Using this method, you can program the survey to take one of the below actions once the quota limit is reached. Learn how to a dvanced quota control in your surveys. Advanced research report management Some other features that we have developed keeping researchers and research oriented surveys in mind, are: Weighting & balancing Weighting and balancing is a survey feature that allows you to eliminate sample bias in your online surveys You can adjust the captured data to represent the population accurately This question helps researchers eliminate bias that occurs when the data derived from the survey does not represent the target population accurately to make sound decisions. The primary motive of weighting and balancing is to yield accurate data-backed decisions This is achieved by eliminating data that does not add value to representing the population accurately You can use weighting and balancing to eliminate demographic biases for the following and more: ● Age bias ● Gender bias ● Location bias ● Educational level ● Marital status ● And more. Learn how to set up weighting and balancing in your research. Hacker’s Guide to Advanced Research Methodologies Data quality Data quality is an important tool that enables researchers the ability to maintain a tab on the quality of survey and research data This allows only high-quality data to be used for analysis and insights management Using inaccurate data can skew up the results for researchers causing time and cost overruns The data quality tool, when enabled before a study is deployed ensures that there is the ability to catch discrepancies in data at an early stage and this data doesn’t get considered for analysis. This also ensures that only high quality survey data is exported to external analysis tools or can be analyzed as is The data quality tool works on qualitative as well as quantitative survey data. Learn how to set up data quality and analyze the most accurate data. Advanced research analysis Analysis of survey data is important because that drives you from data to insights By leveraging advanced research analysis methods, you can derive insights that matter the most to your brand In the QuestionPro Research suite, you can use basic survey analysis as well as advanced survey analysis to Hacker’s Guide to Advanced Research Methodologies unlock mature insights Not just that, create powerful role-based access reports and share with the most relevant stakeholders. The advanced research analysis methods that QuestionPro supports are: Sentiment analysis Sentiment analysis uses advanced artificial intelligence technologies like Natural Language Processing (NLP), text analytics, and data science to identify, extract, and study subjective information In simpler terms, sentiment analysis classifies text as positive, negative, or neutral. Traditional metrics, like the number of views, clicks, likes, shares, comments, etc focus on quantity. Sentiment analysis goes beyond numbers and focuses on the quality of interactions between the audience and the organization. With sentiment analysis, businesses can find out the underlying sentiment from what their customers say about them Due to its ability to understand text using artificial intelligence and machine learning techniques, sentiment analysis is widely used in market research Many software gather “base data” from sources like social media, documents, surveys, etc and analyze the emotions Sentiment analysis tools offer a visual medium to understand the feelings and, thus, convert qualitative data into quantitative data. Sentiment analysis of survey responses is based on two factors: ● Subjectivity - Personal feelings, opinions, or experiences which are subject to change from person-to-person. ● Degree - The extent or range of emotions from positive to negative. NLP and rule-based text analysis algorithms process all the input data and outputs a visual chart, also known as a bubble graph, that classifies different sentiments It displays positive sentiments in green, neutral sentiments in yellow, and negative sentiments in red. The bubbles have data filters in the center, which make it easy for the survey creator to analyze the results Just by having a look, one can quickly identify if the respondents have a good or bad experience with their business The sentiment analysis knowledge graph also shows the percentage of the respondents along with the kind of their experience. Hacker’s Guide to Advanced Research Methodologies Learn how to use sentiment analysis in your surveys. TURF analysis Turf analysis stands for Total Unduplicated Reach and Frequency analysis It is a statistical method analyzing how many people you contacted, and how often you are in contact with the people you reach. TURF analysis is used by market researchers to assess market potential and devise optimal strategies for product position and communication Such analysis helps them estimate the chances of success given limited resources. The ultimate aim of TURF analysis is to identify the most efficient product portfolio It is achieved by determining the products that have the greatest reach in the market. It can answer questions like: ● Where should we place ads to reach the broadest possible audience? ● What kind of market-share will we gain if we add a new line to our model? It was initially devised for analysis of media campaigns and has been expanded to apply to a product, line, and distribution analysis. Hacker’s Guide to Advanced Research Methodologies With QuestionPro surveys, you can perform TURF analysis for any Multiple Choice/Multiple Answer question or Matrix questions. Learn how to set up and use TURF analysis with QuestionPro. Additional resource: H acker’s Guide to TURF Analysis Hacker’s Guide to Advanced Research Methodologies Gap analysis Gap analysis is a technique that helps you assess a business by measuring the difference between the expected performance and its actual performance A “gap” denotes the difference between the present state of the business and its desired state. Use gap analysis to understand what a product lacks and what it needs to excel in the market Don’t just find out what didn’t work but also understand why it didn’t and identify steps to improve the processes. Make data-backed decisions with the gap analysis tool Use QuestionPro’s side-by-side matrix in order to run a gap analysis. Learn how to use Gap analysis with QuestionPro. Hacker’s Guide to Advanced Research Methodologies Correlation analysis Correlation analysis in research is a statistical method used to measure the strength of the linear relationship between two variables and compute their association Simply put - correlation analysis calculates the level of change in one variable due to the change in the other A high correlation points to a strong relationship between the two variables, while a low correlation means that the variables are weakly related. When it comes to market research, researchers use correlation analysis to analyze quantitative data collected through research methods like surveys and live polls They try to identify the relationship, patterns, significant connections, and trends between two variables or datasets There is a positive correlation between two variables when an increase in one variable leads to the increase in the other. On the other hand, a negative correlation means that when one variable increases, the other decreases and vice-versa. Learn how to set up and manage c orrelation analysis with QuestionPro. Hacker’s Guide to Advanced Research Methodologies Additional advanced research software tools To ensure that our research platform allows you the ability to easily conduct all the research that you need, we also provide auto translation of surveys and integrations with multiple external platforms to make sure your insights collection is as seamless as possible. Auto translate surveys You may need to run surveys in local or regional languages depending on where your customers, potential clients, or employees reside Running surveys across the board with only one language option is not ideal You may lose out on respondents as they may not speak the same language Running surveys in multiple languages is always a better idea; it shows you care, genuinely want to hear from them, and improve survey responses. Learn how to set up auto-translation of your surveys with QuestionPro. Integrations You can integrate your QuestionPro survey platform with multiple integrations including existing CRM tools, data analysis and management tools, API and webhook integrations, etc Use your existing Hacker’s Guide to Advanced Research Methodologies platforms and technology stack to leverage the power of the QuestionPro Research platform Some of our commonly used integrations are: ● API ● Webhooks ● SMTP ● FTP ● Salesforce ● Google Analytics ● Hubspot ● Zapier ● Microsoft Dynamics ● Slack ● Caspio ● Power BI Learn more about i ntegrations with the QuestionPro research platform. Hacker’s Guide to Advanced Research Methodologies Clients we have served The world’s leading brands, organizations, Academic institutions and researchers use the QuestionPro Research platform to solve their biggest challenges Some of our notable customers include. ... powerful research to brands, organizations and researchers alike. Hacker’s Guide to Advanced Research Methodologies? ? Advanced research methodologies & techniques - survey platform Using our research. .. Hacker’s Guide to Advanced Research Methodologies? ? Additional advanced research software tools To ensure that our research platform allows you the ability to easily conduct all the research. .. Hacker’s Guide to Advanced Research Methodologies? ? Learn how to use hotspot testing in your research. Additional resource: W ebinar on how to conduct hotspot testing Advanced research