CPU, GPU, and Training Large Language Models

ByF published on 2025-08-20 included in category AI Technology and series AI LLM Systematic Tutorials

AI Tutorial: CPU/GPU and Large Model Training

This is a highly condensed reference: clearly structured, right to the point — covering CPU/GPU fundamentals, tensors and numerical precision, CUDA and PyTorch in practice, hardware selection, common interview questions, and a debugging checklist.

0. Quick Overview (30 Seconds)

CPU vs GPU: CPUs excel at general-purpose/sequential work; GPUs excel at massive parallelism (matrices/vectors).
Large models need GPUs: training/inference is fundamentally matrix multiplication and parallelization — exactly what a GPU’s high concurrency + high-bandwidth memory deliver.
Tensors and precision: all data becomes tensors; precision (FP16/FP8) and quantization (INT8/INT4) trade speed/VRAM against quality.
The PyTorch GPU mantra: device = "cuda" if ...; model.to(device); data.to(device)
Pick a GPU by VRAM first: VRAM first, then bandwidth/compute; for production, prefer full-strength high-quality models or cloud-hosted APIs.

1. CPU vs GPU: Differences, Workloads, and Analogies

1.1 The One-Line Comparison

Dimension	CPU	GPU
Architecture	Few cores, complex control flow	Massive small cores, SIMT parallelism
Excels at	Branching/system tasks/small-scale compute	Matrix multiplication, convolution, attention, graphics rendering
Task model	Time-sliced, low-latency switching	Batch- and throughput-oriented
Typical use	Business logic, scheduling, I/O	Main training/inference operators (GEMM, Conv, etc.)

1.2 An Intuitive Analogy

CPU = a veteran expert: meticulous thinking, does one thing at a time with fast switching.
GPU = a massive army: hordes of soldiers working simultaneously — built for parallel homogeneous small tasks.

1.3 Optional Mermaid Diagram (CPU Execution vs GPU Parallelism)

flowchart LR
    subgraph CPU["CPU (sequential/few cores)"]
      A1[Task1-SliceA] --> A2[Task2-SliceB] --> A3[Task3-SliceC]
    end
    subgraph GPU["GPU (parallel/many cores)"]
      B1[Element1 compute]:::p
      B2[Element2 compute]:::p
      B3[Element3 compute]:::p
      B4[Element4 compute]:::p
    end
    classDef p fill:#e9f5ff,stroke:#3b82f6,stroke-width:1px;

2. Tensors, Precision, and Quantization (with Examples)

2.1 Tensor Hierarchy

0D: scalar 3.14
1D: vector [1,2,3]
2D: matrix (e.g. a 3×3 table)
3D+: still called a tensor (e.g. batch×channel×height×width)

Image example: a batch of 32 224×224 RGB images → 32×3×224×224 (or N×H×W×C, depending on the framework).

Prompt Engineering: From Prompts to Context Engineering

ByF published on 2025-08-13 included in category AI Technology and series AI LLM Systematic Tutorials

AI Tutorial: Prompt Engineering

Prompt engineering focuses on designing, optimizing, and strategizing prompts — helping users mobilize the capabilities of large language models more effectively, and pushing their adoption across real-world scenarios and research domains.

1. Basic Concepts

1.1 What Is Prompt Engineering

A prompt is simply this: you use natural language to tell the model what to do, how to do it, what it may do, and what it must not do. That is all there is to it.

Transformer Architecture Deep Dive: The Attention Mechanism

ByF published on 2025-08-05 included in category AI Technology and series AI LLM Systematic Tutorials

AI Tutorial — Transformer

🧩 1. What Is the Transformer?

The Transformer is a deep learning architecture for processing sequential information — text, speech, code, and so on.

It was first proposed by Google in the 2017 paper Attention Is All You Need.

That paper laid the foundation for nearly every large language model today. GPT, BERT, Claude, Gemini, Qwen, ERNIE Bot — all of them are built on the Transformer.

AI Technical Glossary: A Complete Guide to 270+ Terms

ByF published on 2025-07-30 included in category AI Technology and series AI LLM Systematic Tutorials

AI Technical Glossary

This reference gathers the core terminology of the LLM field, from basic concepts to advanced technical architecture, to help you build a systematic understanding of the AI technology landscape.

📚 Fundamentals

Term	Technical definition	Plain-language explanation	Example
AGI (Artificial General Intelligence)	An AI system with human-level intelligence	An all-capable AI that can think, learn, and create like a person	A robot that can write poetry, code, cook, and chat at the same time
LLM (Large Language Model)	A large neural network model trained on massive data	A “super brain” that understands and generates human language	GPT-4, Claude, ERNIE Bot, and the like are all LLMs
Training	The process of fitting neural network parameters on large data	The AI’s “study phase” — like a person absorbing knowledge from books	Training a model on all text on the internet until it learns language
Inference	A trained model generating output from input	The AI’s “application phase” — like a person answering questions from what they learned	The model generating an answer after you ask it a question
Token	The smallest unit of text a model processes, a fragment split by a tokenization algorithm	The particles of AI language, processed one at a time	`"我喜欢苹果"` → `["我", "喜欢", "苹果"]`

🏗️ Architecture

Term	Technical definition	Plain-language explanation	Example
Transformer	A deep learning architecture based on self-attention, proposed by Google in 2017	The “neural skeleton” of modern AI that lets models understand language efficiently	GPT, BERT, and every other large model is built on Transformer
Encoder	A neural network component that encodes an input sequence into semantic representations	The AI’s “understanding unit” — turns text into vectors machines understand	BERT uses an encoder for text understanding tasks
Decoder	A neural network component that generates output token by token based on context	The AI’s “writing unit” — generates answers from what it understood	The GPT series are all decoder-only models
Self-Attention	A mechanism that computes how much each element in a sequence relates to the others	The AI automatically “focuses on what matters”, like a person picking out key points while reading	In “deposit money at the bank”, “bank” attends to “money”; in “fish by the river bank”, it attends to “river”
Multi-Head Attention	Several self-attention mechanisms run in parallel to capture different types of dependencies	The AI understands text from multiple angles at once	One head tracks syntax while another tracks semantics
Positional Encoding	Vector representations that add position information to each token	Lets the model know “who comes first, who comes later”	“The dog bit the man” and “the man bit the dog” mean different things
Query	The vector that actively asks for related information — what the current word needs	The numeric expression of “what am I looking for”	“Apple” queries attributes like taste and color
Key	The identifier vector for information being queried — what each word can offer	The label of “what I can provide”	“Sweet” serves as the Key for a taste feature, waiting to be queried
Value	The representation vector holding the actual content and true semantic information	“My actual content”, in numbers	The actual semantic representation of “sweet”: `[0.8, 0.2, -0.1]`
Attention Weight	Importance scores expressing how much to attend, usually normalized via softmax	“How much to pay attention”, quantified	0.8 means strong attention, 0.1 weak; all weights sum to 1
Cross-Attention	Attention across two sequences — Query comes from one, Key/Value from another	Cross-modal information exchange	In image-text matching, text Queries attend to image Keys/Values
Causal Attention	Attention restricted to the current position and earlier, preventing future information leaks	Attention that can “only look backward”	When GPT generates the 5th word it can only see the previous 4
Softmax Function	An activation function that turns any real-valued vector into a probability distribution	Converts scores into “importance percentages”	`[2,1,0] → [0.67,0.24,0.09]`, preserving relative magnitudes

🔢 Mathematical Representation

Term	Technical definition	Plain-language explanation	Example
Vector	A mathematical object with magnitude and direction; an ordered list of numbers	A “numeric ID card” that describes a thing with numbers	`[25, 180, 70]` can represent a person’s age, height, and weight
Embedding	The technique of mapping discrete symbols into a continuous vector space	Turns words into “numeric coordinates”	`"king"→[0.25, -0.12, 0.78, ...]`
Query / Key / Value	The three core vector matrices in self-attention: what is asked, what is labeled, what is delivered	Query = what I want, Key = what I can offer, Value = my actual content	`Query=[0.1,0.2]` asks about taste, `Key=[0.8,0.1]` labels sweetness, `Value=[0.9,0.05]` is the actual representation of sweetness
Feed-Forward Network	Applies an independent nonlinear transform at each position	Deepens the model’s understanding of each word	From “spring” the model further associates “warmth, growth”
Layer Normalization	Standardizes a layer’s inputs	A “stabilizer” for training	Prevents gradient explosion or divergence
Residual Connection	A cross-layer connection that preserves the original information	An “express lane” for information, preventing loss	Like a shortcut path that keeps deep networks from degrading

🔄 Processing Pipeline

Term	Technical definition	Plain-language explanation	Example
Tokenizer	Converts text into a sequence of tokens	A “knife for chopping text”	`"Hello world" → ["Hello", " world"]`
Context Window	The maximum number of tokens a model can process	The AI’s “memory limit”	GPT-4 has a 128K context
Decoding	Generates text token by token from a probability distribution	The AI’s “writing process”	Starts generating from the most probable word
Temperature	A parameter controlling generation randomness	A “creativity dial”	High temperature is more creative, low more stable
Top-p Sampling	A sampling strategy based on cumulative probability	An “essence filter”	Only considers candidates whose cumulative probability reaches 90%
Max Tokens	Caps the length of generated output	A “word-count limiter”	Keeps the AI from answering too long

🛠️ Engineering Practice

Term	Technical definition	Plain-language explanation	Example
RAG (Retrieval-Augmented Generation)	An AI approach combining retrieval and generation	An “open-book exam” AI	Look up references first, then answer the question
Prompt Engineering	The craft of designing and optimizing prompts	“The art of asking”	Helping the AI understand your needs better
Fine-tuning	Training a pretrained model on a specific task	“Targeted job training”	Turning a general model into a medical assistant
BPE (Byte Pair Encoding)	A common tokenization algorithm	A “text compression technique”	`"unhappiness" → ["un","happi","ness"]`
Detokenization	Turns a token sequence back into readable text	“Reassembling the pieces”	`["我","喜欢","苹果"]→"我喜欢苹果"`
Streaming	Generates output token by token in real time	The “typewriter effect”	A chatbot thinking while it types

🧠 Classic Models Compared

Term	Technical definition	Plain-language explanation	Example
RNN (Recurrent Neural Network)	A neural network that processes sequences step by step	A “read-one-word-at-a-time AI”	Translating `"我爱你"` word by word
LSTM (Long Short-Term Memory)	An improved RNN that handles long-range dependencies	“A better memory”	Can remember content from the beginning
CNN (Convolutional Neural Network)	A neural network that excels at image patterns	An “image specialist”	Recognizing cats, dogs, and faces
Encoder-Decoder Architecture	A model containing both understanding and generation modules	An “all-round AI”	Machine translation models

📊 Application Scenarios

Term	Technical definition	Plain-language explanation	Example
Chat product	A user-facing AI application interface	An “AI chat shell”	ChatGPT, Claude
API call	An interface for program-to-program communication	The “AI phone line”	An application calling the OpenAI API
Context management	The technique of maintaining conversation history	The “AI’s memory”	A chatbot remembers what you said
Multi-turn dialogue	A continuous human-machine interaction mode	“Ongoing conversation”	Ask about the weather, then what to wear
Function Calling	The model invoking external APIs to perform tasks	The “AI’s ability to act”	The AI checks the weather or searches automatically

🧩 Model Optimization and Training Techniques

Term	Technical definition	Plain-language explanation	Example
LoRA (Low-Rank Adaptation)	Fine-tunes model parameters via low-rank matrices	“Lightweight fine-tuning”	Lets an LLM quickly adapt to a new domain
Quantization	Represents model parameters at lower precision	“Slimming the model down”	FP32→INT8 speeds up inference
Pruning	Removes redundant neurons or connections	“Trimming the branches”	Cutting useless parameters
Distillation (Knowledge Distillation)	A large model teaches a small one	“Teacher trains the student”	GPT-4 teaching a small model
Checkpoint	A saved intermediate state during model training	A “training save point”	Prevents losing progress on a power cut

🔍 Vector Retrieval and Knowledge Integration

Term	Technical definition	Plain-language explanation	Example
Embedding Model	A model that converts text into semantic vectors	A “semantic coordinate machine”	text-embedding-3-large
Vector Database	A database supporting vector retrieval	A “semantic warehouse”	Milvus, Pinecone, FAISS
Cosine Similarity	Measures how similar two vectors’ directions are	A “semantic similarity meter”	`A cat is sleeping ≈ The cat is resting`
Knowledge Graph	Stores knowledge as nodes and relationships	A “knowledge map”	`apple → is a → fruit`
Hybrid Search	Combines semantic retrieval with keyword matching	“Belt-and-suspenders search”	Searching both `cat` and `pet animal` at once

🧩 Multimodal and Agents

Term	Technical definition	Plain-language explanation	Example
Multimodal Model	Handles text, images, audio, and other modalities at once	A “full-senses AI”	GPT-4V, Gemini
VLM (Vision-Language Model)	Vision-Language Model	An AI that can “see pictures”	A visual question-answering AI
Speech Recognition	Converts speech to text	A “dictation AI”	Voice input methods
TTS (Text-to-Speech)	Converts text to speech	An “AI announcer”	The AI reads its answer aloud
AI Agent	An AI capable of autonomous action and decision-making	An AI assistant “that can act”	Devin, AutoGPT

⚙️ Model Evaluation and Safety

Term	Technical definition	Plain-language explanation	Example
Hallucination	The model generating false information	“Confident nonsense”	Inventing papers or facts
Alignment	Bringing the model in line with human values	“Values training”	Tuning a model with RLHF
RLHF (Reinforcement Learning from Human Feedback)	Optimizes a model with human preferences	“Humans teaching AI to speak”	How ChatGPT was trained
Red Teaming	Adversarial testing of model safety	A “security penetration test”	Testing whether the model leaks secrets
Bias	Systematic prejudice in model outputs	“The AI plays favorites”	Preference for a gender or language

🧰 Emerging Trends and Future Directions

Term	Technical definition	Plain-language explanation	Example
Mixture of Experts	A structure with multiple sub-models activated dynamically	A “panel-of-experts AI”	The `Gemini 1.5 Pro` architecture
Context Compression	Compresses conversation history to save tokens	“Memory compression”	Summarizing long conversations
Memory-Augmented Model	An AI combined with long-term memory mechanisms	An AI “with a memory”	`ChatGPT` long-term memory
Autonomous Agent	An AI that can plan and execute tasks on its own	A “self-managing AI”	`AutoGPT`, `Devin`
Synthetic Data	Virtual training data generated by AI	“AI-made textbooks”	Expanding a training set with AI

💡 Study Advice

🎯 Priority for Mastering Core Concepts

Beginner (must know): Token, Embedding, Transformer, LLM
Intermediate (important): Self-Attention, RAG, Context Window
Advanced (optional): LoRA, Mixture of Experts, Red Teaming

📖 Suggested Learning Path

Understand the basics: what a token is, and why vector representation is needed
Master the core architecture: the Transformer encoder-decoder structure
Practice application techniques: combining prompt engineering with RAG
Go deeper into technical details: attention mechanisms and alignment training

🔗 Concept Map

A Complete Guide to LLMs: Tokens and Vectors in Depth

ByF published on 2025-07-22 included in category AI Technology and series AI LLM Systematic Tutorials

AI Tutorial: A Guide to AI LLMs, from Basics to Depth

This article takes you deep into the core concepts of large language models — from basic principles to vector representation — building a complete knowledge system step by step.

1. Foundations of AI Application Development

1.1 Basic Principles and Concepts

In Plain Words

Core mechanism: predict the next word from the previous one — like a word-chain game
How it works: output is generated token by token

A Deeper Look

A large AI model involves two key stages:

Timeline Recorder: Technical Architecture from MVP to Scale

ByF published on 2025-07-14 included in category Engineering Practice and series Project Design Documents

Timeline Recorder: Requirements & Technical Architecture (MVP → Scalable)

Target: personal/small-team local or centralized deployment; web + CLI + TUI; text/images/linked audio and video; draft/publish; users may edit only their own content; everyone’s content viewable, with mute support; no built-in transcription; decentralization as an optional later capability.

0. Background and Goals

Goal: build a “local-first, centrally deployable” timeline recorder that guarantees data sovereignty and simple deployment; an extremely fast capture experience without sacrificing a clean timeline.

Daily Deep Think

The views expressed here are the author's own and do not represent any organization, company or institution.