Knowledge mapping of single-case design research: An analysis with the use of large language transformer model

Mikyung Shin, Ph.D., Department of Education, Center for Learning Disabilities, West Texas A&M University,


Introduction

  • A single-case design (SCD) research focuses on individual performance and measures the causal relationships between variables (Kazdin, 2019). By applying SCDs, researchers can assess an intervention’s effect and estimate variance in intervention effects in a population (Gabler et al., 2011) among students with disabilities, where each participant has a distinct degree of a present level of academic and functional performance. SCDs have been used in many areas, including the educational, behavioral, and biomedical fields.

  • A few research studies focus on how SCD knowledge is shared among scholars across different disciplines, particularly in relation to the use of technology. There is an ongoing need for unpacking knowledge and sharing research findings on single-case research in relation to the use of technology. Through the implementation of the bibliometric analysis using text mining, unstructured text data can be transformed into a meaningful structured format.

Purpose. By applying a bibliometric analysis of studies published between January 1970 and August 2023, the researcher aims to empirically map the evolution of knowledge on single-case research with the application of technology.

Method

Identification and Extraction of Search Terms.

  • To identify targeted studies published between January 1970 and December 2023, the searcher applied four search strategies, adapting the selection procedures of studies based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Page et al., 2021).

  • Although studies were originally published in different languages, if their titles, keywords, and abstracts were reported in English within the Web of Science (WoS) database.

  • single-case design related 18 keywords (e.g., alternat* treatment* design OR applied behavior* analys* OR chang* criteri* design)

  • technology-related 50 keywords (e.g., tablet* OR technolog* OR “tele* OR text* to* speech)

  • Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2018) is a model to generate embeddings and attention masks from text data such as abstract and keywords, subsequently employing a multilayer perceptron network (Goodfellow et al., 2016).

PubMLP: Automatic Publication Classifier Through a Large Language Model.

PubMLP: A Web-Based Publication Classifier Tool

Figure 1: PubMLP: A Web-Based Publication Classifier Tool

  • Multilayer perceptron is a type of artificial neural network used in artificial intelligence (AI) for image and text classification tasks (Maulana & Maharani, 2021).

  • The tabular data include text, categorical data, and numeric features. The utilized functions include the BERT (Bidirectional Encoder Representations from Transformers) tokenizer for text preprocessing and an MLP (multilayer perceptron) for classifying whether the publication meets the researcher-designated inclusion criteria.

Extraction of Data.

  • The researcher initially downloaded all 4,863 bibliographic data by exporting full records and cited references from WoS to the EndNote 21 Desktop. The maximum number of extracted records was 500, resulting in multiple files until the completion of data extraction.

  • The extracted data included metadata, including content (e.g., title, abstract), author (e.g., authors’ name, country), document (e.g., publication year, document title), and citation (e.g., cited references) information.

Structural Topic Modeling.

  • Based on the generative process for each document, a topic is predicted based on a clustered list of words where each word has a probability of belonging to a topic (β). Document (d) consists of a mixture of topic K where the sum of topic proportions per document (θ) is one (Roberts et al., 2019).

  • Considering the continuous change in topic prevalence over time, the proportions of topics over time were examined via the estimateEffect() function of the stm R package (Roberts et al., 2019).

TextAnalysisR: A Web-Based Text Analysis Tool

TextAnalysisR: A Web-Based Text Analysis Tool

Figure 2: TextAnalysisR: A Web-Based Text Analysis Tool

Results

Diagnostics Values by Number of Topics

  • Based on the goodness-of-fit results of models with topic numbers ranged from 2 to 15, topic number 9 showed relatively low residuals, high semantic coherence, a maximized lower bound, and a high held-out likelihood.

Highest (Top 5) Word Probabilities for Each Topic

  • The topic model results of 9 different topics across 1970 and 2023 show different word probabilities for each topic. The highest probability represents the related unlabeled topic, respectively.


Semantic Mapping of Associated Words

  • The figure demonstrates the semantic mapping of associated words with 9 minimum co-occurence numbers per study. Minimum of 0.2 correlation betweewn pairwise words indicated several clusters related to interventions and data analysis methods using technology in SCDs.

Conclusion

  • Understanding the trends and associated keywords can help practitioners and researchers review the patterns of SCD studies across fields. Reflecting the frequently used words/topics and associated word patterns, we can inspect the knowledge clusters and patterns over last decades.

  • Future research can further focus on how the topic proportions differ by other document-level variables, inducing a time variable as before and after the rapid development of emerging technology such as AI, and disseminate the recent changes to the public.

References

Contact. Mikyung Shin, West Texas A&M University,

https://github.com/mshin77/PCRC-2024-open