Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Data

Researchers compared human survey responses from 420 Silicon Valley developers with synthetic data from five leading LLMs including ChatGPT, Claude, and Gemini. While AI models produced technically plausible results, they failed to capture counterintuitive insights and only replicated conventional wisdom rather than revealing novel findings.

• →Five leading LLMs including ChatGPT and Claude were tested against human survey data from 420 Silicon Valley developers.
• →AI models produced plausible results but failed to capture counterintuitive insights that made human surveys valuable.
• →All AI models showed similar deviations from human responses, suggesting they harmonize around conventional wisdom.
• →Synthetic survey responses cannot meaningfully model real human social beliefs in organizational contexts.
• →Researchers recommend using synthetic surveys as pre/post-fieldwork tools rather than replacements for human research.