Humans in Humans Out: GPT converges on common sense in successes and failures

Abstract: Increasing the computational scale and fine-tuning has led to a dramatic improvement in the quality of output from large language models (LLMs) such as GPT. Given that GPT-3 and GPT-4 have been trained on large amounts of human-generated text, we might wonder to what extent their outputs reflect human thought patterns, both for the correct and incorrect cases. . The Erotic Theory of Reason (ETR) provides a symbolic generative model of human success and failure in thought, through propositional, quantified, and probabilistic reasoning, and decision making. We presented GPT-3, GPT-3.5 and GPT-4 with 61 core inference and judgment problems from a recent book-length presentation of ETR, consisting of experimentally verified data points on human judgment and extrapolated data points predicted by ETR, with correct inference patterns as well as errors and framing effects (the ETR61 benchmark). ETR61 includes classics like Wason's Map Task, Illusory Inferences, Decoy Effect, and Opportunity Cost Neglect, among others. GPT-3 showed evidence of predicted ETR outputs for 59% of these samples, increasing to 77% in GPT-3.5 and 75% in GPT-4. Remarkably, the production of human-like misjudgments increased from 18% in GPT-3 to 33% in GPT-3.5 and 34% in GPT-4. This suggests that larger, more advanced LLMs may develop a tendency toward more human-like errors, as relevant thought patterns are inherent in human-produced training data. According to ETR, the same fundamental patterns are involved in both successful and unsuccessful ordinary reasoning, so the "bad" cases could paradoxically be learned from the "good" cases. We further present preliminary evidence that rapid ETR-inspired engineering could reduce instances of these errors.