What Makes a Survey Question Bad? 6 Characteristics to Eliminate

Author
PulseAI Research Team
June 3, 2026

PulseAI ResearchMost Researchers Know What Bad Survey Questions Are. Fewer Know Why They're Bad.

Knowing that leading questions are bad is the start. Understanding why they're bad the psychological mechanism that produces distorted data is what makes you reliably avoid them, not just in the obvious cases but in the subtle ones too.

This guide is structured differently. Instead of listing question types, it identifies the six defining characteristics that make any survey question bad regardless of topic, format, or industry. Once you can name the characteristic, you can spot it in any question you write. For the structural decisions that shape question quality before individual questions are written, survey research design: the complete guide covers the complete design framework.

Characteristic 1: It Signals the Expected Answer

The psychological mechanism: When respondents sense what answer is expected of them, many will provide it not because they're being dishonest, but because social cooperation is a deeply embedded human tendency.

Where it appears: Leading questions ("don't you agree..."), questions with evaluative language ("our excellent service"), and framing that presents one option as the default.

The test: Read the question and ask is there a "right" answer implied by how this is phrased? If yes, rewrite.

Example:

"How satisfied are you with our award-winning support team?"

"How satisfied are you with the support you received?"

Characteristic 2: It Asks Two Questions at Once

The psychological mechanism: Respondents faced with two questions in one either average their views across both (producing a number that accurately describes neither) or anchor on whichever dimension was most salient in their recent experience.

Where it appears: Questions containing "and" between two different constructs: speed and quality, price and value, ease and satisfaction.

The test: Remove the "and" if you're left with two coherent questions, you had two questions. Ask them separately.

Example:

"Was the product easy to use and good value for money?"

Two questions: ease of use + value assessment

Characteristic 3: It Uses Undefined Terms

The psychological mechanism: Ambiguous words like "often," "sometimes," "usually," "good," and "a lot" are interpreted differently by different respondents. The same option means different things to different people making responses incomparable.

Where it appears: Frequency questions without time windows, quality assessments without defined benchmarks, agreement scales without specific propositions.

The test: Substitute yourself for the respondent. Could a reasonable person interpret the term to mean something significantly different from how you intended it? If yes, define it.

Example:

"Do you frequently recommend our brand to others?"

"In the past month, how many times did you recommend our brand to someone?"

For how undefined measurement terms connect to the broader problem of measurement validity in quantitative research, variables in research methodology: types, roles, and how to operationalise them covers the operationalisation framework.

Characteristic 4: It Assumes Something That May Not Be True

The psychological mechanism: Questions that embed a factual assumption about the respondent force them into a false frame. Respondents who don't fit the assumption either accept it (producing inaccurate data) or skip the question (producing missing data).

Where it appears: Questions that assume prior purchase, prior usage, prior awareness, or prior experience without first verifying that experience through a filter question.

The test: Could a respondent for whom the embedded assumption is false answer this question honestly? If not, add a filter.

Example:

"How has our loyalty programme benefited you?"

Filter first: "Are you a member of our loyalty programme?" → Show the benefits question only to members.

Characteristic 5: It Has No Honest Answer for Every Respondent

The psychological mechanism: When respondents can't find an accurate option, they either select the least inaccurate one (contaminating the data with false signals) or skip entirely (creating non-response that biases the remaining distribution).

Where it appears: Scales missing the neutral midpoint, "select all that apply" questions without "none," forced positive questions, binary yes/no for nuanced opinions.

The test: Take the perspective of a respondent whose honest answer isn't on the list. What would they do? If they'd have to lie or skip, the question needs a new option.

Example:

"How positive has your experience with us been?" (scale: Very positive to Slightly positive)

Include: Neither / Slightly negative / Very negative

Characteristic 6: It's Too Complex to Answer in Context

The psychological mechanism: Long, multi-clause questions, questions with extensive preamble, and questions requiring respondents to recall specific details they can't accurately remember all produce lower-quality data not because respondents are careless, but because the cognitive demand exceeds what's reasonable in a survey context.

Where it appears: Questions longer than two lines, questions requiring specific numerical recall ("how many times in the past 18 months"), and questions that include multiple conditional clauses.

The test: Read the question aloud. If it takes more than 10 seconds to read and requires you to hold multiple pieces of information in working memory simultaneously, simplify it.

Example:

"Thinking back over the past 12 months, considering all of the interactions you've had with our team across all channels including phone, email, and chat, how would you rate the overall consistency of the service quality you experienced?"

"Overall, how consistent has the quality of our customer service been over the past year?"

PulseAI Research

The Six Characteristics at a Glance

A bad survey question almost always has at least one of these:

It signals the expected answer — through leading language or evaluative framing

It asks two things at once — producing uninterpretable composite responses

It uses undefined terms — making responses incomparable across respondents

It assumes something untrue — forcing respondents into false frames

It has no honest option for some respondents — creating forced inaccurate responses

It's too complex for the survey context — degrading response quality through cognitive overload

For how these characteristics relate to the specific types of survey bias they produce, survey bias in questionnaires: types, examples, and how to fix each one covers the bias mechanisms in full.

Myth vs Fact: Common Misconceptions About Survey Question Quality

Myth: Longer questions are more precise. Fact: Longer questions increase cognitive load and produce lower-quality responses. Precision comes from specific language, not question length.

Myth: A high response rate means the questions were good. Fact: Response rate measures willingness to participate, not quality of responses. A survey with excellent completion rates can still produce biased, invalid data if the questions are poorly designed.

Myth: If respondents didn't complain, the questions were fine. Fact: Respondents rarely complain about confusing or biased questions they simply answer them inaccurately. The problems surface in the data, not in feedback.

Myth: Open-ended questions are more honest than closed-ended ones. Fact: Open-ended questions produce more varied responses but are not inherently more honest. Both formats are subject to social desirability bias and other response distortions.

For how these characteristics connect to broader sampling quality issues that affect survey reliability at the population level, sampling errors in surveys: types, examples, and how to avoid them covers the complementary framework.

FAQ

What are the characteristics of a bad survey question?

Six core characteristics: it signals the expected answer, asks two things at once, uses undefined terms, assumes something that may not be true about the respondent, provides no honest option for all respondents, or is too complex to answer accurately in a survey context.

What is the most common reason survey questions fail?

Ambiguous or undefined terms words like "often," "good," or "regularly" that different respondents interpret differently. This produces data that appears consistent but cannot be compared across respondents.

How do leading questions affect survey data?

They activate social compliance the tendency to provide the answer that feels expected or socially appropriate. This consistently inflates positive scores and produces data that reflects social pressure rather than genuine opinion.

What is a filter question and why is it important?

A filter question screens respondents for a relevant experience or behaviour before asking about it. It prevents respondents who don't qualify from answering a question that doesn't apply to them which would contaminate the data with guesses or forced inaccurate responses.

Can you fix a bad question after data collection?

No. Question-level flaws are built into the data. Statistical techniques can partially correct for some biases but cannot recover the information that a bad question failed to collect accurately.

Conclusion

The six characteristics covered here are not abstract theoretical concepts. They are patterns that appear in real surveys, commissioned by experienced teams, producing data that looks reasonable and isn't.

The practical takeaway is simple: apply the six tests to every question before fielding. Signal the expected answer? Rewrite. Two things at once? Split. Undefined terms? Define. Assumed behaviour? Filter first. No honest option? Add one. Too complex? Simplify.


Pulse AI Research builds six-point question quality checks into every research brief review before questionnaire design begins.

Read Similar Blogs

10 Market Research Techniques That Actually Deliver InsightsMarket Research Steps: A Practical Framework for Brand Teams Who Need...Primary Research: A Practical Guide for Brand TeamsConsumer Research Process: A Step-by-Step Workflow for Better InsightsFactors Influencing Consumer Behaviour and the One Your Research Is...How to Create a Survey Questionnaire That Delivers Reliable ResultsDifference Between Research Method and Research Methodology: Clearing Up...Where Market Research Is Headed: Trends Brands Can’t IgnoreHypothesis Testing in Research Methodology: A Practical GuideQualitative Research Questions: How to Ask Better Questions for Deeper...Quantitative Research Methodology: A Complete Guide for Brand Research...Consumer Research Methodology: A Step-by-Step GuideLikert Scale Survey Design: How to Use the Most Common Measurement Tool...Confusing Survey Questions: 25 Examples and How to Fix ThemBrand Tracking vs Brand Research: Ultimate Guide for Marketers and AnalystsWhy Customers Buy: Consumer Behaviour Insights for BrandsQualitative Research Techniques: How to Extract Better Consumer InsightsAdvanced AI Research Methods in Market Research MetaQuestionnaire Answer Choices (2026): How to Write Options That Improve...Quantitative vs Qualitative Consumer Research: Which One?