MaxDiff questions

A MaxDiff question (best-worst scaling) asks respondents to compare a list of items a few at a time. In each set they pick the item that matters most and the one that matters least. Across all the sets, SurveyRock estimates each item’s share of preference: how often it would be chosen over the others. The shares add up to 100.

Use it when you have 6 to 30 comparable things and need to know which ones matter most: features to build next, benefits to lead with, messages to test. A rating scale lets people say everything is important; MaxDiff makes them choose, so the differences between items are clear. For 5 items or fewer, a ranking question is simpler.

MaxDiff is available on the Growth plan and above. On other plans the MaxDiff (Best-Worst) tile is shown with a Growth badge. If an account moves below Growth, a MaxDiff it already has keeps collecting responses and its results stay visible, but a new one can’t be added and a survey that contains one can’t be duplicated.

Adding a MaxDiff question

1. Open a survey and go to Design.
2. In the left sidebar’s Elements tab, drag MaxDiff (Best-Worst) onto the canvas, or double-click it.
3. Replace the six placeholder items with your own. Use Add item, or Bulk edit items to paste a list with one item per line. A MaxDiff needs 6 to 30 items.
4. Write the question as the thing the sets answer, for example “Which of these matters most to you when choosing a project management tool?”

Settings

Open the question’s settings panel (the pencil in its toolbar).

Design

• Items per set: how many items each set shows, 3 to 5.
• Times each item is shown: how many times each respondent sees each item, 1 to 3.

Both start on Recommended, which follows the number of items. Three exposures per item are recommended when that keeps a respondent to 12 sets or fewer, two otherwise, and one only for very long lists. Under the two settings, a line gives the number of sets each respondent answers and an estimate of the time it takes, at about 12 seconds per set. More than 15 sets is long for a respondent: consider fewer items, larger sets or fewer exposures.

Wording

• “Most” column label and “Least” column label: the two picks, “Most important” and “Least important” by default. Name one quality and its true opposite, such as “Most appealing” and “Least appealing”. Don’t use “Best” and “Worst” when the quality is importance.
• Instructions: the line shown above each set.

Writing good items

The answers are only as good as the list. Each item should:

• Be the same kind of thing, at the same level of detail, as every other item. “Price” and “The checkout button is blue” can’t be meaningfully traded off against each other.
• Hold one idea. “Fast and reliable support” is two items.
• Avoid negation. Picking “least” for “Doesn’t require training” is a double negative.
• Be short, about 12 words or fewer. Respondents compare every item in a set at once.
• Be a real option. Leave out “None of these”, “Other” and “Don’t know”. They can’t be compared with real items.

What respondents see

Respondents see one set at a time, with “Set 3 of 9” above it. In each set they choose one item in the most column and a different item in the least column, then move to the next set. Their progress is saved after every set, so a respondent who leaves part-way keeps the sets they finished. If the question is required, the survey’s Next button waits until every set is answered.

Respondents see different sets. SurveyRock builds 100 balanced versions of the design when the survey is published, and each respondent is given one of them. Within a version, every item appears the same number of times, so across your sample each item is shown about equally often, next to every other item and in every position. A respondent who returns to an unfinished survey gets the same sets again.

After launch: the design is locked

Once a MaxDiff has responses, its items, Items per set and Times each item is shown can’t be changed. A new design would give later respondents sets the earlier ones never saw, and one estimate across both would compare unlike things. You can still fix the text of an item, the labels and the instructions. To change the design, use Duplicate survey in the settings panel: the copy starts with no responses.

Reading the results

On the Responses dashboard, a MaxDiff question’s card lists each item’s share of preference, highest first. The bar is the share and the whiskers are its 95% interval.

How to read a share. Shares add up to 100. If one item has a share of 24 and another a share of 12, the first was preferred about twice as often as the second. That is not the same as “twice as important”: MaxDiff measures relative preference, and whether the scores behave as a ratio scale is disputed in the research literature.

Overlapping intervals are not a test. Two items whose intervals overlap may or may not differ. SurveyRock doesn’t run significance tests between items or between groups.

The scores are for the group you’re viewing. A filter on the dashboard re-estimates the scores for the respondents it selects, so you can compare segments by switching filters.

Expand Data Table under the card for the numbers behind each bar:

Column	What it is
Share	Share of preference; all items add to 100
95% interval	The range the share is likely to fall in
Utility (SE)	The model’s score for the item, centred on zero, with its standard error. For researchers and exports
Best − worst	(times chosen most − times chosen least) ÷ times shown, from −1 to +1
Shown, Most, Least	Raw counts

Why the simple count and the model can differ slightly. The best − worst score counts choices directly; the share comes from a model that also accounts for which items each was shown against. In a balanced design they usually agree closely. A large disagreement points to a problem with the design or the data.

When there are too few responses

The card shows shares only when the data can support them. Otherwise it explains why in an amber note and shows the raw best − worst counts instead, labelled as counts:

• Fewer than 30 respondents have answered the MaxDiff. With weighting, this counts the effective number after weighting.
• An item was shown fewer than 30 times among the respondents you’re viewing, often because a filter is narrow.
• An item was chosen as most (or least) every time it was shown. Its score has no finite estimate. Collect more responses, or remove the item from a copy of the survey.
• Some items were never shown alongside the others in the responses you’re viewing, so their scores can’t be put on one scale.

Two warnings appear with the scores: when they rest on fewer than about 100 respondents (the intervals are wide), and when fewer than 70% of those who started the MaxDiff finished every set (people who stop early can differ from those who finish).

Exporting

From the card’s download menu:

• Download CSV: the results table, highest share first, with each item’s code, and the method used. It isn’t available while the card is showing counts instead of scores.
• Design and choices (CSV) or (Excel): one row per respondent and set, with the item codes shown in that set in display order, the codes picked as most and least, and each respondent’s weight when the survey has a weighting scheme. This is the long format that survival::clogit in R takes, and it can be reshaped for other choice-modelling tools. Use it to estimate individual-level scores, such as hierarchical Bayes, which SurveyRock doesn’t calculate. The Excel file adds an Items sheet listing each code’s text.

In the survey’s response export, a MaxDiff takes two columns per set: the item picked as most and the item picked as least. In SPSS these are the items’ codes with the item texts as value labels. An item’s code never changes, so a reordered or renamed item keeps its code from one export to the next.

What MaxDiff can’t tell you

• Whether any item matters in absolute terms. MaxDiff ranks items against each other. If every item on the list is unimportant to someone, the scores still rank them. That needs an anchored design, which SurveyRock doesn’t offer.
• Each respondent’s own scores. The scores describe a group. For individual scores, export the design and choices file.

How the scores are calculated

For researchers who want the detail:

• Model: multinomial logit, with each set contributing two choices. The “most” pick is a choice among the items shown; the “least” pick is a choice among the same items with the utilities reversed.
• Standard errors: cluster-robust (sandwich) by respondent, because one person’s sets aren’t independent of each other.
• Weighting: with weights, each respondent’s contribution is multiplied by their weight (weighted pseudo-likelihood), and the weights are treated as fixed.
• Shares: for each item, p = e^u / (e^u + k − 1), where u is its zero-centred utility and k the items per set: the probability it’s chosen as most from a set of items of average appeal. The p values are rescaled to add up to 100.
• Intervals: the delta method on the log of the share, so each interval is asymmetric and stays above zero.

Through the API

API v1’s createSurvey and addQuestions, and the MCP tools create_survey and add_questions, create a MaxDiff with question type maxdiff. Its choices are the items, and mostLabel, leastLabel, itemsPerSet and exposuresPerItem are optional. On a plan below Growth the request is refused with a feature_locked error. See Build a survey with questions through the API.