A tokenizer splits text into tokens and maps them to integer IDs that LLMs can process. Learn how subword tokenization works, what BPE, WordPiece, and SentencePiece do differently, and why tokenization affects API cost, context limits, and multilingual performance.

Embeddings convert text, images, and other data into fixed-length numeric vectors that preserve semantic meaning. This guide covers pronunciation, the math behind vector spaces, model comparison (OpenAI, Cohere, Voyage), Python examples, and production usage in semantic search and RAG.