Cover not available
Part of
Ethical Issues in Applied Linguistics Scholarship
Edited by Peter I. De Costa, Amr Rabie-Ahmed and Carlo Cinaglia
[Research Methods in Applied Linguistics 7] 2024
► pp. 10–27

Chapter 1
A taxonomy of questionable research practices in quantitative humanities

| Northern Arizona University
| Northern Arizona University
| Indiana State University
| Uppsala University
| University of South Florida
| Northern Arizona University
A growing body of research has begun to address ethical issues in the context of Applied Linguistics (e.g., De Costa, 2016; Isbell et al., 2022). One of the messages inherent in this line of inquiry is that ethical concerns are embedded throughout the research cycle from study conceptualization to realization, dissemination, application, and beyond (see Bernstein et al., this volume). With this concern in mind, the present study sought to catalog and develop a taxonomy of what are often referred to as ‘questionable research practices’ (QRPs; Steneck, 2006) and related decisions that come into play in the conduct of quantitative Applied Linguistics research. These include practices such as selective reporting and obscuring of methodological details to limit criticism. Using existing taxonomies developed in neighboring disciplines as a starting point (e.g., Tauginienė et al., 2019), we employed the Delphi method to elicit responses on potential QRPs in an iterative fashion from an expert panel as well as from peer scholars. The analyses of these data resulted in a domain-specific taxonomy that laid the groundwork for a large-scale survey that assessed the prevalence and perceived severity of ethical issues and QRPs found specifically in quantitative Applied Linguistics research (Larsson et al., 2023). The results are also used to inform materials for methodological training in research ethics in Applied Linguistics and related disciplines (see De Costa et al., 2021; Wood et al., 2024, in press).
Article outline
  • Introduction
  • Literature review
  • Method
  • The expert panel
  • QRP item generation and revision
  • Results
  • Discussion
  • Conclusion and future directions
  • Notes
  • References
This content is being prepared for publication; it may be subject to changes.
References (41)
References
Al-Marzouki, S., Roberts, I., Marshall, T., & Evans, S. (2005). The effect of scientific misconduct on the results of clinical trials: A Delphi survey. Contemporary Clinical Trials, 26(3), 331–337. DOI logoGoogle Scholar
Andringa, S., & Godfroid, A. (2020). Sampling bias and the problem of generalizability in applied linguistics. Annual Review of Applied Linguistics, 40, 134–142. DOI logoGoogle Scholar
Crossley, S., Marsden, E., Ellis, N., Kormos, J., Morgan-Short, K., Thierry, G. (2020). Introduction of methods showcase articles in Language Learning. Language Learning, 70, 5–10. DOI logoGoogle Scholar
Crowther, D., Kim, S., Lee, J., Lim, J., & Loewen, S. (2021). Methodological synthesis of cluster analysis in second language research. Language Learning, 71, 99–130.Google Scholar
De Costa, P. I. (Ed.). (2016). Ethics in applied linguistics research: Language researcher narratives. Routledge.Google Scholar
De Costa, P. I., Sterling, S., Lee, J., Li, W., & Rawal, H. (2021). Research tasks on ethics in applied linguistics. Language Teaching, 54, 58–70. DOI logoGoogle Scholar
Farangi, M. R., & Nejadghanbar, H. (2024). Investigating questionable research practices among Iranian applied linguists: Prevalence, severity, and the role of artificial intelligence tools. System, 125, 103427.Google Scholar
Gass, S., Loewen, S., & Plonsky, L. (2021). Coming of age: The past, present, and future of quantitative SLA research. Language Teaching, 54, 245–258. DOI logoGoogle Scholar
Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS One, 4(5), e5738.Google Scholar
Gass, S., & Plonsky, L. (2020). Introducing the SSLA methods forum. Studies in Second Language Acquisition, 42, 667–669. DOI logoGoogle Scholar
Gass, S. M., Plonsky, L., & Huntley, E. (2022). Taking the Long view: A bibliometric analysis. In A. Benati & J. W. Schwieter (Eds.), Second language acquisition as shaped by the scholarly legacy of Professor Michael Long (pp. 9–27). John Benjamins. DOI logoGoogle Scholar
Gries, S. Th. (2021). (Generalized linear) mixed effects modeling: A learner corpus example. Language Learning, 71, 757–798. DOI logoGoogle Scholar
Hall, J., & Martin, B. R. (2019). Towards a taxonomy of research misconduct: The case of business school research. Research Policy, 48, 414–427. DOI logoGoogle Scholar
Hu, Y., & Plonsky, L. (2021). Statistical assumptions in L2 research: A systematic review. Second Language Research, 37, 171–184. DOI logoGoogle Scholar
Isbell, D., Brown, D., Chen, M., Derrick, D., Ghanem, R., Gutiérrez Arvizu, M. N., Schnur, E., Zhang, M., & Plonsky, L. (2022). Misconduct and questionable research practices: The ethics of quantitative data handling and reporting in applied linguistics. Modern Language Journal, 106, 172–195. DOI logoGoogle Scholar
Jordan, S. R. (2013). Conceptual clarification and the task of improving research on academic ethics. Journal of Academic Ethics, 11, 243–256. DOI logoGoogle Scholar
Kumar, M. N. (2008). A review of the types of scientific misconduct in biomedical research. Journal of Academic Ethics, 6, 211–228. DOI logoGoogle Scholar
Larson-Hall, J. (2017). Moving beyond the bar plot and the line graph to create informative and attractive graphics. Modern Language Journal, 101, 244–270. DOI logoGoogle Scholar
Larsson T., Plonsky, L., & Hancock, G. R. (2021). On the benefits of structural equation modeling for corpus linguists. Corpus Linguistics and Linguistic Theory, 17, 683–714.Google Scholar
Larsson, T., Plonsky, L., Sterling, S., Kytö, M., Yaw, K., & Wood, M. (2023). On the prevalence and perceived severity of questionable research practices. Research Methods in Applied Linguistics, 2, 100064. DOI logoGoogle Scholar
Loewen, S., Lavolette, E., Spino, S., Papi, M., Schmidtke, J., Sterling, S., & Wolff, D. (2014). Statistical literacy among applied linguists and second language acquisition researchers. TESOL Quarterly, 48, 360–388. DOI logoGoogle Scholar
Marsden, E., Morgan-Short, K., Thompson, S., & Abugaber, D. (2018). Replication in second language research: Narrative and systematic reviews and recommendations for the field. Language Learning, 68, 321–391. DOI logoGoogle Scholar
Marsden, E., & Plonsky, L. (2018). Data, open science, and methodological reform in second language acquisition research. In A. Gudmestad, & A. Edmonds (Eds.), Critical reflections on data in second language acquisition (pp. 219–228). John Benjamins. DOI logoGoogle Scholar
Norouzian, R., de Miranda, M. A., & Plonsky, L. (2018). The Bayesian revolution in second language research: An applied approach. Language Learning, 68, 1032–1075. DOI logoGoogle Scholar
Norris, J. M., Plonsky, L., Ross, S. J., & Schoonen, R. (2015). Guidelines for reporting quantitative methods and results in primary research. Language Learning, 65, 470–476. DOI logoGoogle Scholar
Plonsky, L. (2023). Sampling and generalizability in Lx research: A second order synthesis. Languages, 8, 75. DOI logoGoogle Scholar
(2024). Study quality as an intellectual and ethical imperative: A proposed framework. Annual Review of Applied Linguistics. DOI logoGoogle Scholar
Plonsky, L., Brown, D., Chen, M., Ghanem, R., Gutiérrez Arvizu, M. N., Isbell, D. R., & Zhang, M. (2024). “Significance sells”: Applied linguists’ view on questionable research practices. Research Methods in Applied Linguistics, 3, 100099. DOI logoGoogle Scholar
Porte, G., & McManus, K. (2019). Doing replication research in applied linguistics. Routledge.Google Scholar
Resnik, D. B., Neal, T., Raymond, A., & Kissling, G. E., (2015). Research misconduct definitions adopted by U.S. research institutions. Accountability in Research, 22, 14–21.Google Scholar
Rodríguez-Lifante, A., & Pereira, M. M. B. (2021). The Delphi method in applied linguistics in the light of a theoretical and critical analysis. Revista Brasileira de Linguística Aplicada, 21, 271–293. DOI logoGoogle Scholar
Steneck, N. H. (2006). Fostering integrity in research: Definitions, current knowledge, and future directions. Science and Engineering Ethics, 12(1), 53–74. DOI logoGoogle Scholar
Sterling, S., & Gass, S. (2017). Exploring the boundaries of research ethics: Perceptions of ethics and ethical behaviors in classroom-based applied linguistics research. System, 70, 50–62. DOI logoGoogle Scholar
Sterling, S., Plonsky, L., Larsson, T., Kytö, M., & Yaw, K. (2023). Introducing and illustrating the Delphi method for applied linguistics research. Research Methods in Applied Linguistics, 2, 100040. DOI logoGoogle Scholar
Sudina, E. (2023). Scale quality in second-language anxiety and WTC: A methodological synthesis. Studies in Second Language Acquisition, 47, 1427–1455. DOI logoGoogle Scholar
Tauginienė, L., Gaižauskaitė, I., Razi, S., Glendinning, I., Sivasubramaniam, S., Marino, F., Cosentino, M., Anohina-Naumeca, A., & Kravjar, J. (2019). Enhancing the taxonomies relating to academic integrity and misconduct. Journal of Academic Ethics, 17, 345–361. DOI logoGoogle Scholar
Turoff, M., & Linstone, H. A. (2002). The Delphi method: Techniques and applications. Addison-Wesley.Google Scholar
Wood, M., Larsson, T., Plonsky, L., Sterling, S., Kytö, M., & Yaw, K. (2024). Addressing questionable research practices in applied linguistics: A practical guide. Applied Linguistics Press.Google Scholar
Wood, M., Sterling, S., Larsson, T., Plonsky, L., Kytö, M., & Yaw, K. (in press). Researchers training researchers: Ethics training in Applied Linguistics. TESOL Quarterly.
Yaw, K., Plonsky, L., Larsson, T., Sterling, S., & Kytö, M. (2023). Timeline: Research ethics in applied linguistics. Language Teaching, 56, 478–494. DOI logoGoogle Scholar