Vincent Adeyemi 
Emory University Hospital Midtown, Georgia, USA 
Correspondence to: vincrayun@gmail.com
Additional information
- Ethical approval: N/a
- Consent: N/a
- Funding: No industry funding
- Conflicts of interest: N/a
- Author contribution: Vincent Adeyemi – Conceptualization, Writing – original draft, review and editing
- Guarantor: Vincent Adeyemi
- Provenance and peer-review:
Commissioned and externally peer-reviewed - Data availability statement: N/a
Keywords: Large language models, Hypothesis generation, Psychological research, Ethical considerations, Interdisciplinary collaboration.
Peer Review
Received: 16 November 2024
Revised: 4 December 2024
Accepted: 5 December 2024
Published: 23 December 2024
Abstract
Large language models (LLMs) are revolutionizing psychology research, particularly in the area of hypothesis generation. Psychologists have traditionally depended on established methods like comprehensive literature reviews, empirical research, and theoretical frameworks. Despite their effectiveness, these methods frequently fall short of the increasing expectations of the data-driven world of today. Since the introduction of LLMs, academics have had access to instruments that can analyze large volumes of text, spot complex patterns, and produce original ideas that may inspire new, creative research questions. This article examines how LLMs can improve hypothesis formation in the social, developmental, and therapeutic domains, among the other subfields of psychology.
It looks at their main benefits, namely their capacity to automatically scan vast amounts of psychological literature, creatively combine disparate materials, and identify patterns at scale. It also recognizes their drawbacks, including their dependence on perhaps skewed training data, their inability to comprehend complex situations, and their inability to match artificial intelligence (AI)-driven theories with accepted research practices. Their responsible implementation also emphasizes the importance of ethical factors, such as encouraging transparency, guaranteeing accountability, and safeguarding privacy. Future developments could include the development of specialized LLMs for psychology research, hybrid strategies that incorporate AI and human expertise, and interdisciplinary collaboration to optimize the use of these tools. By carefully incorporating LLMs into research procedures, hypothesis development could be transformed, opening the door to a more profound understanding and a more thorough comprehension of human behavior.
Introduction
The processes that have traditionally driven psychological research include expert knowledge, empirical observation, and iterative hypothesis generation—processes requiring creativity and the ability to synthesize large amounts of information. Technological advances have allowed social and behavioral scientists to collect diverse psychological data at scales previously unimaginable, driving major advances in psychological science and related domains.1 Until recently, the start of the digital revolution and, along with that, the availability of “big data” have resulted in enormous advances in the knowledge of psychological processes: for the first time, researchers were able to analyze patterns, behaviors, and interactions in a way and on a scale hardly imaginable.2
Recently, large language models (LLMs) have transformed into groundbreaking tools in artificial intelligence (AI). These are extensive neural networks, reaching billions to over a hundred trillion parameters, and are trained on vast text datasets. In this setting, these models demonstrate exceptional efficacy in language understanding, generation, and translation. The self-supervised or semi-supervised training approach that enables these models to find intricate patterns in unstructured textual data is the foundation for their comprehensive handling of language-related challenges. LLMs’ emergent tendencies, which are similar to cognitive processes, increase their usefulness in comprehending psychological ideas and human language.5,6
The convergence of LLMs and psychological research offers promising prospects for hypothesis formulation. Psychological research is often initiated on a ground of hypothesis, which stands as the foundation of scientific inquiry. These hypotheses guide investigations into attitudes, behaviors, and beliefs, addressing questions about the most basic ways humans think and behave.7 Traditional methods of hypothesis generation often take a lead from established theories, serendipitous observations, or interdisciplinary insights.8 However, LLMs now offer a new paradigm for hypothesis generation; their unparalleled ability to analyze big data, observe patterns, and infer relationships has the potential to enormously accelerate discovery in social, developmental, and clinical psychology. The potential to redefine the research process makes the LLMs among the most capable tools for starting the investigation of new frontiers.
Traditional Methods of Hypothesis Generation in Psychology
Traditional hypothesis generation methods have always served as the foundation of psychological research. These methodologies, although effective, frequently fail under the pressures of modern research. One of the traditional methods involves the application of theoretical approaches. This method typically formulates hypotheses based on foundational theories, such as Piaget’s stages of cognitive development or Freud’s psychoanalytical theory, which offer systematic frameworks for elucidating psychological phenomena. Attachment theory by Bowlby has also been important in the formulation of hypotheses on early childhood relationships and subsequent emotional development.9 Theoretical frameworks guide the formulation of hypotheses but may at the same time inadvertently reduce creativity by reinforcing biases already present in the original theories. Empirical observations, both from case studies and through naturalistic observation, also have an important role in the generation of hypotheses. For instance, hypotheses about social media influences on learning behaviors in classrooms can be generated for testing by researchers studying classroom group dynamics.10 These approaches have their value but tend to be very labor-intensive and pose scaling challenges when larger datasets are involved or when new phenomena are under exploration.11
The systematic review of the existing literature is an essential component in traditional hypothesis generation. By examining trends and recognizing deficiencies in previous research, scholars might formulate new study questions and hypotheses. Meta-analyses, for example, frequently uncover determinants affecting therapy outcomes and stimulate additional research.12 As the body of psychological literature grows, the manual synthesis and identification of significant patterns becomes progressively more difficult.13 While these traditional approaches have been essential to the growth of psychological research, their limitations underscore the need for complementary tools. Emerging technologies, such as LLMs, offer the potential to augment traditional methods by addressing the issues of scale, bias, and complexity, paving the way for more efficient and innovative hypothesis generation.
Capabilities of LLMs Relevant to Hypothesis Generation
The adoption of LLMs, such as GPT and BERT, signifies significant progress in psychology. Their capacity to analyze vast datasets—from surveys to interview transcripts and scholarly articles—allows academics to concentrate more on hypothesis formation rather than laborious manual analysis. This benefit is especially pertinent now since psychology increasingly depends on datasets that are too extensive for conventional methods to manage efficiently.16 A notable capability of LLMs is their adeptness at identifying patterns and trends in intricate data. This ability is essential in psychology, where comprehending the complex interactions between elements such as environmental conditions and mental health necessitates advanced analysis. By seeing these nuanced relationships, LLMs might formulate hypotheses that researchers may otherwise neglect owing to cognitive or resource constraints.17
Apart from pattern recognition, LLMs demonstrate the ability for creative hypothesis through the recombination of previously learned knowledge in new and innovative combinations. This novelty in the creation of research questions invites exploratory research and challenges the status quo. An LLM trained on psychotherapy data could, for instance, formulate hypotheses about emergent patterns in interactions between clients and therapists that have not yet been discussed as an area of formal academic study.19 Applications of such models can also master the automation of routine tasks; these include qualitative coding of data and summarizing findings—a task that saves researchers from focusing their efforts on the design of experiments and interpretation of results.20
Practical Applications in Psychology
LLMs can process and synthesize large volumes of psychological literature quickly, identify gaps in research, and propose new lines of investigation. For instance, when researching adolescent mental health, an LLM could identify recurring themes—such as screen time affecting emotional well-being—in a vast pool of texts and save researchers’ considerable time and effort by guaranteeing completeness of the coverage.22 Such a capability for streamlining literature synthesis contributes to overcoming one of the principal difficulties to hypothesis generation through increased speed and scope. Other than literature review, LLMs can be useful in clinical psychology where large volumes accrue, such as therapy transcripts or patient notes that seldom get utilized. Therefore, extracting meaningful patterns from these datasets using language models will help generate meaningful hypotheses. For instance, a recent study by Tai et al used LLMs to deductively code interview transcripts, comparing their outputs with those of human coders to demonstrate how LLMs can provide systematic and reliable code identification while minimizing analysis misalignment.39
LLMs also handle large-sized behavioral data analyses, including longitudinal analyses, that provide complex relationships. For instance, they will be able to evaluate variables such as work hours in relation to stress levels to hypothesize the predictors of burnout, thus enabling a more complex analysis of psychological phenomena.24 Recent applications of LLMs have surpassed traditional data sources for the analysis of behavioral trends on social media websites. These latter platforms generate reams of unstructured data in real time, which can be processed by LLMs to determine the patterns of societal behaviors and psychological trends. For example, during a worldwide crisis, an LLM could analyze tweet content to assess collective anxiety levels and resilience patterns, offering actionable insights for public mental health treatments.25
Ethical and Practical Considerations
The integration of LLMs into psychology to generate hypotheses is a creation of an array of opportunities but at the same time raises critical ethical and practical challenges. Among the main risks is excessive reliance on the generated hypotheses. While LLMs can efficiently process voluminous datasets to come up with novel ideas, there is a danger in accepting these outputs without thorough validation. While LLMs’ hypotheses may sound plausible, they are often scientifically weak and can mislead research efforts. Human judgment is thus held to be moderately indispensable to ensure that the hypotheses generated through AI techniques are relevant and valid.26,27 Another concern is bias in LLMs. With the large datasets that they are trained on, LLMs inevitably show these biases. This can be even more serious in psychological research, as such studies presuppose the sample’s representativeness for all the necessary demographics. If LLMs show partiality to some opinions only because of biased training data, then their results cannot be generalized and fair. Researchers must proactively work on identification and mitigation if they want to keep their contributions impeccable.21,28
Practical solutions for mitigating biases include using diverse and balanced training datasets to improve the representativeness of model outputs, employing bias detection and correction tools, and involving interdisciplinary reviews to adjust training processes. For example, Ruggeri et al suggested developing a methodology for the analysis of machine learning to detect and understand biased decisions.29 Another important aspect is transparency in hypothesis generation. It has been a cause for concern that LLM models “black-box” how individual outputs came to be, reducing the possibility for the reproduction and trust of research findings. Several explainable AI techniques can be used to enhance the transparency and accountability of the models through detailed records of how decisions are made via modeling.30,31
The application of LLMs in psychology also greatly involves adherence to standards on privacy. Since psychological data is sensitive in nature, participant confidentiality is a very important aspect to observe. Researchers should follow ethical instructions on the protection of privacy and the building of confidence for individuals, such as data anonymization and free and informed consent.31 More generally, the use of LLMs raises ethical issues amid a lack of accountability and possible misuses. To overcome the problems, an ethical framework is required. The varied viewpoints required to negotiate these problems can be obtained through cooperative efforts involving psychologists, ethicists, and AI developers.30
Limitations of Using LLMs in Hypothesis Generation
LLMs offer remarkable potential for producing psychological theories, but they also have significant drawbacks. Their insufficient comprehension of psychological theories is one of the primary obstacles. Large datasets are used to teach the LLMs statistical patterns. This leads to a deficiency in comprehension, which is essential for understanding more intricate psychological constructs. This may yield outputs that appear appropriate at face value yet are contextually inappropriate or superficially theoretical.33 Besides, LLMs tend to generate very general or unrelated hypotheses, especially in highly domain-specific research. The generation of novel or topic-specific hypotheses may be restricted because of overdependence on patterns already contained in data. This limitation underlines the main role for human oversight concerning the relevance and novelty of generated ideas.34
Resource availability is also a barrier. Advanced computational resources and technical expertise are required, both in implementing and fine-tuning LLMs, which may not be equally accessible to all researchers. This may further increase inequity between well-resourced and under-resourced institutions with regard to the adoption of AI tools.35 Another challenge involves the integration of LLMs with traditional research methods. Researchers need frameworks that can align the hypotheses generated through AI with the existing designs in research and validation processes. Using LLMs in psychology is further complicated by ethical issues related to data privacy, informed consent, and the problem of bias perpetuation. This implies that the responsible use of LLMs in psychological research requires interdisciplinary cooperation and ethical attention in order to overcome such significant constraints.
Future Directions
To maximize the potential of LLMs for hypothesis generation in psychology, specific developments and strategies are required. First is the development of LLMs that are designed for use in psychology. It would arguably make them more contextually fitting and theoretically anchored if they were to be used for hypothesis generation, given their training within domain-specific data and literature. Custom LLMs would raise the relevance of their AI outputs by further responding to the needs of research in psychology.37 An alternative method to developing hypothesis generation is through the hybrid approach: AI combined with the capabilities and judgments of humans. An LLM can promptly identify a pattern and offer an initial hypothesis that can be further developed and verified by researchers. This combined approach strikes a balance between scientific rigor and computing efficiency.38
Collaborations among data scientists, psychologists, and AI experts can resolve technical issues, improve model interpretability, and create ethical standards for using AI in research. Such collaborations could lead to innovative tools that support hypothesis generation while upholding rigorous ethical standards.39 Broadly speaking, the impact of LLMs on experimental psychology is huge since they speed up the generation of hypotheses and make the creative process in finding new research directions faster. Beyond this, it is fair to say that their consistent processes may be considered yet another factor that could make psychological studies more reproducible.40 In light of these considerations, pursuing these future directions will enable psychology to responsibly integrate LLMs into its research practices in such a way that it offers innovation but further extends knowledge on human behavior.
References
1. Lazer D, Pentland A, Adamic L, Aral S, Barabási AL, Brewer D, et al. Social science: Computational social science. Science (80-) [Internet]. 2009 Feb 6 [cited 2024 Nov 10];323(5915):721-3. Available from: https://www.science.org/doi/10.1126/science.1167742
2. Grossmann I, Feinberg M, Parker DC, Christakis NA, Tetlock PE, Cunningham WA. AI and the transformation of social science research. Science (80-) [Internet]. 2023 Jun 16 [cited 2024 Nov 10];380(6650):1108-9. Available from: https://www.science.org/doi/10.1126/science.adi1778
https://doi.org/10.1126/science.adi1778
1 Abdurahman S, Atari M, Karimi-Malekabadi F, Xue MJ, Trager J, Park PS, et al. Perils and opportunities in using large language models in psychological research. PNAS Nexus [Internet]. 2024 Jun 28 [cited 2024 Nov 10];3(7):245. Available from: https://dx.doi.org/10.1093/pnasnexus/pgae245
https://doi.org/10.1093/pnasnexus/pgae245
2 Ke L, Tong S, Cheng P, Peng K. Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Review. 2024 Jan 3 [cited 2024 Nov 10]; Available from: https://arxiv.org/abs/2401.01519v3
3 Wei J, Tay Y, Bommasani R, Raffel C, Zoph B, Borgeaud S, et al. Emergent Abilities of Large Language Models. 2022 [cited 2024 Nov 10]; Available from: https://arxiv.org/abs/2206.07682v2
4 Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN,
et al. Attention is all you need. Adv Neural Inf Process Syst [Internet]. 2017 Jun 12 [cited 2024 Nov 10];2017:5999-6009. Available from: https://arxiv.org/abs/1706.03762v7
5 Dougherty M, Thomas R, Lange N. Toward an integrative theory of hypothesis generation, probability judgment, and hypothesis testing. Psychol Learn Motiv Adv Res Theory. 2010;52(C):299-342.
https://doi.org/10.1016/S0079-7421(10)52008-5
6 Jaccard J, Jacoby J. Theory Construction and Model-Building Skills: A Practical Guide for Social Scientists. New York: Guilford Press [Internet]. 2010. p. 39-74. Available from: http://eprints.qums.ac.ir/788/1/Theory Construction and Model Building Skills.pdf
7 Johnson DW, Johnson RT. An educational psychology success story: Social interdependence theory and cooperative learning. Educ Res. 2009;38(5):365-79. Available from: https://doi.org/10.3102/0013189X09339057
https://doi.org/10.3102/0013189X09339057
8 Patton MQ. Qualitative Research & Evaluation Methods: Integrating Theory and Practice. SAGE. 2014. Available from: https://us.sagepub.com/en-us/nam/qualitative-research-evaluation-methods/book232962
9 Cuijpers P, van Straten A, Warmerdam L. Behavioral activation treatments of depression: A meta-analysis. Clin Psych Rev. 2007;27(3):318-26. Available from: https://doi.org/10.1016/j.cpr.2006.11.001
https://doi.org/10.1016/j.cpr.2006.11.001
10 Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485-514. Available from: https://doi.org/10.1111/1468-0009.12210
https://doi.org/10.1111/1468-0009.12210
11 Yarkoni T, Westfall J. Choosing prediction over explanation in psychology: Lessons from machine learning. Perspect Psychol Sci. 2017;12(6):1100-22. Available from: https://doi.org/10.1177/1745691617693393
https://doi.org/10.1177/1745691617693393
12 Anderson ML, Kedersha J. Big data and behavioral science: Opportunities and challenges. Ann Rev Psychol. 2021;72:21-47. Available from: https://doi.org/10.1146/annurev-psych-020821-102938
13 Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, et al. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020;33:1877-1901. Available from: https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
14 Tenney I, Das D, Pavlick E. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics 2019;(pp. 4593-4601). Association for Computational Linguistics. Available from: https://doi.org/10.18653/v1/P19-1452
https://doi.org/10.18653/v1/P19-1452
15 Bommasani R, Hudson DA, Adeli E, Altman R, Arora S, von Arx S, et al. On the Opportunities and Risks of Foundation Models. arXiv preprint, arXiv:2108.07258. 2021. Available from: https://arxiv.org/abs/2108.07258
16 Amodei D, Olah C, Steinhardt J, Christiano P, Schulman J, Mane D. Concrete Problems in AI Safety. arXiv preprint, arXiv:1606.06565. 2016. Available from: https://arxiv.org/abs/1606.06565
17 Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners. OpenAI Blog. 2019;1(8):1-24. Available from: https://openai.com/research/language-unsupervised
18 Bender EM, Gebru T, McMillan-Major A, Shmitchell S. On the dangers of stochastic parrots: can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency 2021;(pp. 610-623). ACM. Available from: https://doi.org/10.1145/3442188.3445922
https://doi.org/10.1145/3442188.3445922
19 Devlin J, Chang M-W, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT. 2019. Association for Computational Linguistics.
20 Reiter H, Stewart J. AI in behavioral research: Opportunities and challenges. J Behav Sci. 2022;45(3):241-58.
21 Samuel J, Kleinberg S. Harnessing AI for mental health insights. Comput Psychiatry Rev. 2021;12(2):89-104.
22 Jackson A, Roberts P. Social media as a data source for psychology. Trends Psychol Res. 2023;16(4):333-45.
23 Abdurahman S, Atari M, Karimi-Malekabadi F, Xue MJ, Trager J, Park PS, et al. Perils and opportunities in using large language models in psychological research. PNAS Nexus. 2023;3(7):245. Available from: https://academic.oup.com/pnasnexus/article/3/7/pgae245/7712371
https://doi.org/10.1093/pnasnexus/pgae245
24 Watkins R. Guidance for researchers and peer-reviewers on the ethical use of Large Language Models (LLMs) in scientific research workflows. AI Ethics. 2023;3:1-6. Available from: https://link.springer.com/article/10.1007/s43681-023-00294-5
25 Antoniak M, Naik A, Alvarado CS, Wang LL, Chen IY. NLP for Maternal Healthcare: Perspectives and Guiding Principles in the Age of LLMs. arXiv preprint. 2023. Available from: https://arxiv.org/abs/2312.11803
26 Ruggeri M. On Detecting Biased Predictions with Post-Hoc Explanation. 2023. Available from: https://mcanini.github.io/papers/bias.safe23.pdf
https://doi.org/10.1145/3630050.3630179
27 Flanagin A, Bibbins-Domingo K, Berkwits M, Christiansen MA. Guidance on reporting use of AI in research and scholarly publication. JAMA. 2023;330(6):563-4. Available from: https://jamanetwork.com/journals/jama/fullarticle/2807956
28 Demszky D, Yang D, Yeager DS, et al. Using large language models in psychology. Nat Rev Psychol. 2023;2:1-3. Available from: https://www.nature.com/articles/s44159-023-00241-5
29 Lin Z. Beyond principlism: Practical strategies for ethical AI use in research practices. AI Ethics. 2024;4:1-12. Available from: https://link.springer.com/article/10.1007/s43681-024-00585-5
https://doi.org/10.1007/s43681-024-00585-5
30 Taylor A, Kaplan K. The Limitations of Large Language Models for Scientific Research. Direct MIT. 2024. Available from: https://direct.mit.edu/opmi/article/doi/10.1162/opmi_a_00160/124234
31 Brown C, Smith J. The Potential and Limitations of Large Language Models in Identifying Research Gaps. Illinois Experts. 2024. Available from: https://experts.illinois.edu/en/publications/the-potential-and-limitations-of-large-language-models-in-identif
32 Jones M, Patel R. Navigating Ethical Challenges in Large Language Models for Psychology. Nature. 2024. Available from: https://www.nature.com/articles/s44159-023-00241-5.pdf
33 Chen X, Wu Y, Zhou Q. Exploring the Frontiers of Psychology-Specific Large Language Models. arXiv. 2024. Available from: https://arxiv.org/pdf/2409.02387
34 Green P, Anderson T. Hybrid Models for Integrating LLMs into Psychology Research. arXiv. 2024. Available from: https://arxiv.org/abs/2402.04470v4
35 Roberts L, Martinez F. Interdisciplinary Collaboration for Ethical AI in Psychology. Springer. 2024. Available from: https://link.springer.com/article/10.1007/s10648-024-09868-z
36 Smith R, Johnson K. Broader Impacts of Large Language Models in Experimental Psychology. Nature. 2024. Available from: https://www.nature.com/articles/s44184-024-00056-z.pdf
37 Tai RH, Bentley LR, Monteith BG. An Examination of the Use of Large Language Models to Aid Analysis of Textual Data. All Articles. 2024. https://doi.org/10.1177/16094069241231168
https://doi.org/10.1177/16094069241231168
38 Li B, Sun X, Tang F. Perils and opportunities in using large language models in psychological research. Proc Natl Acad Sci Nexus. 2024:3(7):245. Available from: https://academic.oup.com/pnasnexus/article/3/7/pgae245/7712371
https://doi.org/10.1093/pnasnexus/pgae245
