Large Language Models and Large Multimodal Models in Medical Imaging: A Primer for Physicians

Tokenization breaks text into more fundamental units called tokens. Each token is then represented by embedding vector. Embeddings for tokens with similar meanings tend to group together in vector space (illustrated here in 3-dimensional space for simplicity).

Attention module is core building block of transformer networks. In self-attention, embeddings for single token of input sequence (e.g., “eyes”) are updated by first computing attention weights using query-key comparisons and then using attention weights to do weighted sum of value vectors. Resulting update vector is then added to original embeddings.

Transformers can be organized as encoder-only, decoder-only, and encoder–decoder networks, depending on prediction task. MLP = multilayer perceptron.

Pipeline for creating chatbot LLM.

There are 3 primary techniques for integrating language and images in multimodal models: contrastive learning, late-fusion (i.e., cross-attention), and early fusion.