The questions were the variable.
We began with a practical question: What, if anything, is different about research moderated by a human and research moderated by conversational AI? Then the project gave us a more interesting question. What if the interviewer was not the biggest variable after all?
Leelan (Head of Research at Aylo) and Nicole (Head of Research at Dialogue AI) built this collaboration to compare human and AI moderation using the kind of subject that makes polite, generic answers especially easy to spot. Participants talked about adult content, desire, search, and the language they use when they are trying to find something specific.
The first study held the interview guide steady. Leelan moderated one set of conversations, while three AI moderators used the same questions with a much larger group. That let us compare coverage, depth, language, and conversation texture without pretending that one measure could settle whether an interview was “good.”
What we learned changed the next study. The AI could recover the thematic map, but its conversations were thinner. So instead of changing the moderator again, we changed the instrument. Study 2 rewrote the guide around remembered moments, concrete choices, and participant language, then sent the same AI moderator back into the field.
A skilled human moderator followed the same core guide used by the AI arms.
This was the moderator comparison: hold the guide steady and change who conducted the conversation.
This was the instrument test: keep the AI moderator and change how the questions invited people to answer.
When an interview changes, how much comes from the moderator, and how much comes from the questions we taught that moderator to ask?
Did changing the moderator change the research?
It changed the depth of the conversation more than the territory it covered. Every emergent theme appeared in both modes, while the human arm produced richer signal per interview.
The interviews did not have the same texture.
The human conversations gave participants more room to hold the floor. The AI returned at more regular intervals, producing a more evenly striped exchange. Select any ribbon to inspect it.
Human moderator
Conversational AI moderator
Long participant stretches dominate the exchange. The moderator appears mainly as brief checkpoints between extended answers.
Four examples from each arm, selected to make the contrast in participant floor time legible. Each ribbon runs from the first turn to the last; segment width reflects the words spoken in that turn.
What changed when the questions changed?
The vocabulary more than doubled. The same AI moved from broad judgments to concrete descriptions when the guide reconstructed moments instead of asking for opinions.
How quickly did each arm reach 34 terms?
The rewritten guide reached the human arm’s vocabulary threshold in 10 interviews. The human arm took 21; the same-guide AI took 57.
Ask for judgments
How do you decide which site to use?
How do you search? Tell me about a time you could not find something.
What should the industry improve?
Reconstruct the moment
Walk me through opening the site like I’m watching over your shoulder.
When you know what you’re in the mood for, how would you describe it?
What makes you click? What is an instant no? How fast do you leave?
Was AI about 80% of a human?
As a shorthand, yes. Across six reasonably comparable Study 1 measures, the median AI result was 79% of the human result. The spread matters more than the slogan.
Did feeling freer produce more explicit language?
Not in these interviews. Most people said the AI made candor easier, but the group reporting no difference used more explicit terms. That rate difference was not statistically significant.
said the AI made them feel freer to speak. Twenty-six explicitly reported no difference.
Directionally opposite to the easy story; not statistically significant.
Who produced the signal, and how usable was it?
The scale story only holds if the interviews are worth analysing. Study 2’s 100 transcripts averaged 3.85 of 5 on the Interview Quality Score; none was excluded, and every interview remains visible in the distribution.
Most interviews cleared the quality bar.
Mean IQS 3.85, median 3.92, range 2.79–4.81. Twenty-four transcripts carried a review flag; none was removed. Scores were computed from the transcript, not copied from the platform rating.
Gender identity
Age
Orientation
Visit frequency
Gender and orientation are participants’ own screener responses. Small categories are combined here for legibility; the technical appendix retains the exact study accounting.
Use the machine for reach. Invest in the instrument for change.
What we can say
AI reproduced the thematic map, supported far greater scale, and delivered roughly four-fifths of the human result across several comparable measures. The guide rewrite produced the largest observed language shift.
What remains bounded
One moderator cannot stand for every human. Study 2 had no human arm, so guide effects and between-study differences cannot be fully separated.
The open question is not whether AI makes people honest. It is whether the instrument unlocks a register, or extracts it.