Your ultimate guide to concept testing

What is concept testing?

It’s a tough job to manage the launch of your company’s products and services. You’re already facing several statistics that paint a grim picture of the reality of product launches:

• Only 55% of all product launches take place on schedule, and 45% of product launches are delayed by at least one month, according to a 2019 product manager survey from Gartner.
• More than 25% of total revenue and profits come from launching new successful products across industries, according to a McKinsey research survey.
• Preparing is the biggest problem when launching new products, according to a Harvard Business School report.

Under this pressure, you want to make sure your product connects with your customer before it goes to market. By doing the right product research, you’re checking that your assumptions about your customer are correct.

Product concept testing is an early market research method that maximises the odds of you launching a product or service that people want to buy. You explore the viability of a product or service with its target audience early on and improve its development from the feedback.

What concept testing is not

Concept testing is not a replacement for the ideation phase of a development cycle. Instead, it tests the basic idea that comes out of the ideation phase – whether it is fully formed or not. In this way, your target audience might vary as the product idea develops.

It’s also different from brand testing, advertising, or marketing campaigns – these promote the product or service when it’s near-complete or fully developed.

Free eBook: Introduction to product concept testing

How concept testing fits into market research

Traditional market research requires audiences to choose from a company’s predefined criteria to provide their feedback. It is one-way and ends when the company gets the answers they want.

Concept testing is different. It’s based on listening to the audience’s views and exploring how viable the concept is for them – without bias. You get their direct feedback on what they want, which might shed light on your blind spots. It’s an open learning process that goes both ways – you can continue to engage with the same target audience to get their take on your product developments as they happen.

It’s also a way to show good Experience Management. At a product launch, your brand’s perceived value and the strength of your offering are directly linked to your profit and loss. Likewise, a disappointing loss can hurt internal staff morale, investor relationships, and future business ventures.

What are the benefits of concept testing?

“Poor test-and-learn execution has been known to hobble a company’s fortunes for years to come.”

– Bain & Company

Based on its uses in the market, there are two main benefits of concept testing:

1. Leaders have actionable insights to make the right decisions

Mistakes and do-overs are costly, so it’s important to utilise research and insights from your audience(s) to make the right decisions. Concept testing provides the granularity and depth researchers, managers and leaders need to make the right choices.

2. It helps you gain support for the concept at an early stage

If your plan for a product or service requires a lot of input from senior leaders or colleagues, providing them with a ‘first look’ at evidence can speed up the process and show the potential of the work they’ll be helping with. When a team is confident, there is more support and willingness to go the extra mile for you.

An essential tool for product managers and executives

Why is this an essential tool for product managers and senior executives?

1. Concept testing can be a cost-effective and flexible solution

You can set up a simple and quick survey if you want high-level feedback, or you can delve deeper to understand more detail.

Concept testing is also a great money-saving technique as it prevents launching a faulty concept. And in comparison with the cost of outsourcing to external market research programs, research for an internal concept testing project is cheaper.

2. Concept testing can help you optimise the product or service

The insights are evidence that can make you feel empowered to act with authority.

You can uncover or identify information that will have a true impact on your product development process decisions. A few examples are:

• How effective the concept’s brand equity, brand image, and benefits are to the customer.
• The customer’s perception of the concept’s price range, buying preference, and how they might use it.
• The concept’s status among its target market competition.
• The concept’s weaknesses (E.g. Is there unclear communication, low customer value, or an unmemorable product personality?)

You can also test multiple concepts at the same time to see which works best at solving your customer’s problem. If a product concept doesn’t work, invest in the ones that do make a difference.

3. Concept testing acts as a quality-assurance check over time

It’s common for there to be several rounds of concept testing, allowing product managers to check in with target audiences and optimise the offering.

Vary your audience to get different insights. You can use the same target audience for reflections on your improvements, or a new target market to get fresh insights on your design and developments.

Concept testing programs also lay the foundations for future benchmarks. Ideas that satisfy key criteria and land well with your audience can be used as barometers for new and/or similar concepts.

4. Research can help you build up good customer relationships

You can build brand and customer loyalty and increase your brand equity value, by including your potential customers in the concept’s design and development.

As a brand, you look transparent and open to innovation. It’s also a great way to show you value the opinions of your customers. In addition to this, you should include a broader sample to understand if and how your concept appeals to wider audiences — you don’t want to innovate for just your current customers.

5. Companies that product concept test can avoid costly consequences

In 1957, Ford spent the equivalent of $3.1 billion on the “car of the future,” the Ford Edsel. While Ford conducted consumer polls, they failed to implement feedback and many of the touted new features were unreliable.

Coors made a similar blunder when they attempted to enter the bottled water market in 1990. The packaging and branding of Coors Rocky Mountain Spring Water were similar to Coors beer, which frightened many of Coors’ target demographic.

Three examples of successful concept tests

Some brands have leveraged the testing process to make decisions that have tremendous payoffs. They can cover lots of areas, including:

• A product’s marketing strategy
• Design concepts
• Go to market strategy
• Positioning testing

A few examples include:

1. Emerging fashion brand Shinola used customer research to verify which watches to feature in their spring 2018 collection.
2. Yamaha used concept testing to make a product design decision about using a knob or a sliding fader in a new electronic keyboard. Understanding customer preferences has helped Yamaha maintain its top keyboard in the industry.
3. Chobani used product testing to adjust packaging in one of its new products, Chobani Oats and Ancient Grains. This helped the brand increase the user-friendliness of the product and maintain the top spot as Australia’s preferred yogurt brand.

How do you conduct a concept test?

We’ve put together a step-by-step guide to help you create the best concept tests for your products and services. We cover these topics:

• Choose the right concept testing survey methodology
• Choose the right survey components for your concept test
• Choose the right flow for your concept test
• Identify the most promising concept
• Concept testing common mistakes and how to rectify them

Or, you can also identify your most promising product concepts using our Concept Testing Tool. It’s an expert solution that tests and reveals what people think of your company’s concept ideas.

Step 1. Choose the right concept test survey methodology

For a successful concept test, in the early stages, consider which methodology will best fit your needs. Some of the most common survey methodologies include:

• Single Concept Evaluation (Monadic) – Respondents complete a full evaluation of a single concept.
• Multiple Concept Evaluation (Sequential Monadic) – Respondents complete full evaluations for multiple concepts.

Each product testing methodology comes with its own positive and negative trade-offs. Get a closer look at our pros and cons table below:

Step 2. Choose the right survey components for your concept test

Concept tests can easily be constructed once the researcher has determined the key components for the survey (which vary according to your purpose).

Carefully consider the objectives of your concept testing to make sure the measures used will successfully answer your team’s questions.

The major components needed for a concept test:

• Measurement for the overall concept reaction
• Concept need or relative improvement over the current method of doing things
• Overall reaction to the concept (acceptability, desirability, interest)
• Likelihood to purchase or use the concept

Components for detailed concept analysis evaluation

• Likes and dislikes about the concept
• Attribute list evaluation, e.g. quality, value for money, durability, tastes, and so on
• Whether the concept would be an addition or replacement
• Awareness of substitute and complementing products

Components for use situation evaluation

• Likelihood of use in specified situations
• Current use of similar / competing products

Components for value analysis

• Price sensitivity analysis
• Preferred method of purchase

Component for segmentation analysis

• Market segments of your target market that you’re most likely to use (order and prioritise)

Step 3. Choose the right flow for your concept test

The flow of your concept test survey ensures that audiences understand and buy into the survey they’re taking.

In the introduction section, make sure to introduce your survey in clear language. Qualify the respondent with a few suitability questions to make sure they fit your audience preference. Ask demographic questions to give the respondent initial easy-to-answer questions to get them started (you can use the results to also filter your results later).

Next: The main section. Here you can start to add in the core survey questions. For example:

Establish a baseline of respondent’s buying history

• What brands have you purchased (within the product class) in the past three months?
• Identify purchase dimensions: number of bottles, frequency of purchase, frequency of consumption in a day
• Is the respondent an “early adopter” in the category?

Introduce the concept

Evaluate the concept as a whole

• What is the perceived value of the product concept?
• How innovative is this concept?
• Is this relevant to you?
• What do you think about the packaging? (Though not typically included in concept testing)
• What do you want to be different?

Explore the concept’s attributes

• What do you think about this X attribute? (Continue for each attribute)
• What are the benefits? Further probing questions might be:
  • Is it good value for the money?
  • Would it help you to… (do something more easily or better)?
  • Would it improve your health?

Compare the concept

• What’s the concept’s power to replace the current brand you purchase?

Ask about respondent’s buying response now

• What is your likelihood of purchase?
• Where might you purchase from?
• Do you believe in the concept?

End the survey by thanking the respondent. You could also use this opportunity to remind them that their data will be stored securely, and it’s only accessed by the research team.

Check out one example of a concept testing survey below. You can access this template for free, and begin using it immediately here.

Step 4. Identify the most promising concept

When you carry out concept testing, you’ll receive a lot of data back. If you’re using a turn-key system like the Qualtrics Concept Testing Tool, you can pre-assemble data together quickly into reports. If you’re sorting the data by hand, use spreadsheet software like Microsoft Excel to begin to sort, filter, compare and pivot-chart your results. You can also try to pull information into a visual format with the in-built charting tools.

We recommend splitting your analysis into two sections:

• Overall results will give you a high-level view of which concepts performed the best.
• Individual results will let you dive into each concept and understand how they performed and why. Here you can use psychographic segmentation to uncover more from your results.

Where qualitative analysis for open-ended questions has been captured, try parsing out common themes, strengths, and weaknesses. Using a tool like Qualtrics Text iQ can make this analysis quicker.

Concept testing common mistakes and how to rectify them

Some common mistakes when creating your concept tests are:

Mistake #1: Running concept testing on an idea in isolation

You might choose to run a test to get your initial insights, but then you stop there. This might provide you with the answers you wanted, but you may be missing out on the full benefit of comparing research data over time.

Solution: In advance, think about what you’re measuring and how you’ll use that information when you get it. This might direct you to use a different methodology or suggest whether you need to repeat the exercise to see if your product’s development is on the right track.

Mistake #2: Over-stuffing lots of ideas into one concept test survey

You may want to use the tool to compare lots of ideas in one survey and ask a long list of questions for each, to save time. Even if you explain this in the introduction, it could make your target audience’s experience long, tiring, and confusing.

Solution: It’s important to respect the time and energy of the targeted audience. For one idea, use the Monadic approach, which goes into depth on a single idea. For multiple ideas, try using multiple surveys that explore one idea each, and think about how many survey questions are reasonable to ask. This way, your target audience is clear about the focus of the survey and more likely to finish them.

Mistake #3: If you don’t get the answers you wanted in the concept test, you automatically cancel the idea

You find out that there is no demand or interest for your product ideas with your target audience. That could be a reason to stop investing your time and energy into the product’s development. However, this may be short-sighted as there may be potential profit in changing it.

Solution: The purpose of the research is to see what target customers think and feel about the idea. If it’s negative, then it could mean that the target audience is not interested at this time, or they are not the right audience. You may find more by testing more widely or trying again after some time. A bad result doesn’t mean it’s a bad idea. This is also where an early adopter question is useful — especially if you feel your new concept is ahead of its time or the market.

Mistake #4: You assume the audience thinks like you do

You’ve got an in-depth understanding of the product idea, and you want to share this in the survey introduction. You may also want to share marketing and brand ideas now, as you’ve been thinking of them already.

Solution: Avoid using technical jargon or acronyms that won’t be immediately understood by your audience. They can be intimidating and confusing for survey participants.

Also, avoid making a strong marketing pitch. Adding this information now can suggest the idea is set in stone – remember that marketing and brand activities come in at a later stage when the idea is close to or fully formed and you don’t want to prevent getting honest feedback.

Mistake #5: You don’t adjust the survey to cater to all geographies

You may have an audience you haven’t considered. Trial your test in different countries to avoid missing out on expanding your product idea’s reach.

Solution: Where you want to widen your audience pool, consider translating the survey into the native language. In addition, you can use images as survey answers, displaying them together to see which is preferred.

Learn more about concept testing

As we’ve seen, concept testing is more than getting feedback using a Likert scale. It involves considering the methodology, components, and flow of your survey for maximum impact.

It can be tricky to do and lots of companies are looking for support. Here are a few customers that turned to Qualtrics with their concept testing issues, and here’s how we helped them:

• Under Armour: This sports clothing and accessories brand struggled to scale its product testing with its current technology. By implementing the centralised Qualtrics Experience Management solution, they were able to increase product testing from 100, to 10,000 testers. Read more about this story.
• Pinterest: This image sharing and social media platform was trying to find out what would make their platform number 1 with their customers. Feedback on potential product developments was the starting point for an entirely new feature set. Read more about this story.

When concept testing is done right, it saves a lot of time, money, and energy. Most importantly, this essential, money-saving tool lets you know you’re on the right track and gives you the insights you need to move forward with confidence.