Understanding The Basics of LLMs

Introduction

I am currently reading AI Engineering and it dives into how Large Language Models (LLMs) have revolutionized how we interact with artificial intelligence, powering everything from chatbots to content generation tools. But how do these sophisticated systems actually work? At their core, LLMs rely on three fundamental concepts that work together seamlessly: tokens, vectors, and embeddings.

Why This Matters: Understanding how these concepts work together in the grand scheme is crucial for grasping modern AI. Each component builds on the others to create the sophisticated language understanding we see in today's AI systems.

What Are Tokens?

Think of tokens as the basic vocabulary units that language models use to understand text. When you type a sentence, the model doesn't process it as a continuous stream of characters. Instead, it breaks your text into discrete pieces called tokens.

Input Text: "Hello, world!"
     ↓
Tokenization: ["Hello", ",", "world", "!"]

Different models use different tokenization strategies. Some focus on whole words, others break words into smaller sub-word units. This process standardizes how the model receives input, converting every piece of text into a sequence of tokens that can be systematically processed.

The Role of Vectors in Language Processing

Once text becomes tokens, these tokens need to be converted into a format that mathematical models can work with. This is where vectors come in, essentially numerical fingerprints for each token.

Token: "cat"
  ↓
Vector: [0.2, -0.7, 0.1, 0.9, ...]

Each token gets mapped to a unique vector with potentially hundreds or thousands of dimensions. These vectors allow the model to perform mathematical operations on language, treating words and phrases as points in a high-dimensional space.

Embeddings

Embeddings are special vectors that capture semantic relationships and contextual meanings. Through training on vast amounts of text, models learn to position similar concepts close together in embedding space.

Semantic Space Visualization:
    
    🐕 dog ←→ puppy 🐶
         ↑
    similar meaning
         ↓
    ✈️ airplane (far away)

Words like "dog" and "puppy" end up with similar embeddings, while "dog" and "airplane" are positioned far apart. Embeddings also capture context. For example, "bank" has different representations in "river bank" versus "savings bank."

The Complete Pipeline

Here's how these components work together in practice:

Text Input: "The cat sat on the mat"
     ↓
Tokenization: ["The", "cat", "sat", "on", "the", "mat"]
     ↓
Vectorization: [[0.1, 0.5, ...], [0.2, -0.7, ...], ...]
     ↓
Embeddings: Context-aware representations
     ↓
Model Processing: Understanding + Response Generation
     ↓
Output: Coherent text response

The model follows this pipeline to transform raw text into meaningful understanding, enabling complex language tasks like translation, summarization, or question answering.

Why Understanding These Fundamentals Matters

Grasping how tokens, vectors, and embeddings work together helps explain both the capabilities and limitations of current LLMs. The quality of tokenization affects multilingual performance, embedding sophistication determines semantic understanding, and vector operations enable rapid text generation.

Conclusion

As LLMs evolve, improvements in each area contribute to better AI performance. Understanding these building blocks provides the foundation for grasping how machines process human language and opens the door to deeper understanding of the technology.