Lemmatize Tokens with Batch Processing

Converts tokens to their lemmatized forms using spaCy, with batch processing to handle large document collections without timeout issues.

Usage

lemmatize_tokens(
  tokens,
  batch_size = 50,
  model = "en_core_web_sm",
  verbose = TRUE
)

Arguments

tokens

A quanteda tokens object to lemmatize.

batch_size

Integer; number of documents to process per batch (default: 50).

model

Character; spaCy model to use (default: "en_core_web_sm").

verbose

Logical; print progress messages (default: TRUE).

Value

A quanteda tokens object containing lemmatized tokens.

Details

Uses spaCy for linguistic lemmatization producing proper dictionary forms (e.g., "studies" -> "study", "better" -> "good"). Batch processing prevents timeout errors with large document collections.

See also

Other preprocessing: get_available_dfm(), get_available_tokens(), import_files(), prep_texts(), process_pdf_unified(), unite_cols()

Examples

if (FALSE) { # \dontrun{
tokens <- quanteda::tokens(c("The studies showed better results"))
lemmatized <- lemmatize_tokens(tokens, batch_size = 50)
} # }