Skip to contents

Multimodal Analysis

Source: vignettes/multimodal_analysis.Rmd
multimodal_analysis.Rmd
library(TextAnalysisR)

sample_text <- c(
  "Figure 1 shows the distribution of student outcomes.",
  "Table 2 reports the effect sizes for each intervention."
)
toks <- prep_texts(
  data.frame(united_texts = sample_text),
  text_field = "united_texts"
)
quanteda::ntoken(toks)
## text1 text2 
##     7     8

Extract text from PDFs with charts, diagrams, and images using vision AI. R-native pipeline – no Python required.

How It Works

  1. Extracts text from each page using pdftools::pdf_text() (R-native)
  2. Renders each page as a PNG image via pdftools::pdf_render_page()
  3. Identifies sparse-text pages (< 500 characters) that likely contain figures
  4. Sends only those pages to a vision LLM for description
  5. Merges extracted text + image descriptions into a single text corpus

Functions

process_pdf_unified() runs the full pipeline with automatic fallback:

  1. Multimodal (pdftools + vision LLM) – extracts text and describes visual content
  2. Text-only (pdftools) – fallback when no vision provider is set

describe_image() describes a single base64-encoded PNG. Both require a vision-provider API key (OpenAI/Gemini) and network access; see their reference pages for usage.

Provider Comparison

Provider Cost Privacy Accuracy Setup
OpenAI Per use Cloud Best API key
Gemini Free on hosted app (Google Cloud Research); otherwise per use Cloud Best API key