by: AI

This glossary covers the foundational concepts, file structures, and technical operations needed to work with Large Language Models (LLMs).

Core Concepts & Definitions

  • LLM (Large Language Model)
    • A neural network trained on large text datasets to predict the next token in a sequence.
  • Transformer
    • The neural network architecture powering modern LLMs (used for Chat, Code, Speech, and more).
  • Attention
    • A mechanism that allows the model to weigh the importance of different tokens in a sequence.
  • Multi-Head Attention
    • Multiple attention layers running in parallel, each capturing different aspects of the input.
  • Inference
    • The process of running a trained model to generate output, as opposed to training or fine-tuning it.
  • Parameters (e.g., 7B, 13B, 70B)
    • The “size” of a model, measured in billions of parameters. More parameters generally mean stronger reasoning but require more hardware resources.
  • Embedding
    • A numerical vector representation of text that allows a model to process semantic meaning.

Tokenization & Context

  • Tokens
    • The smallest unit of data a model processes. One token is roughly 3–4 characters or 0.75 words.
  • Context Window
    • The maximum number of tokens a model can consider in a single prompt (e.g., 64k or 128k tokens).
  • Temperature
    • A setting that controls output randomness. Higher values produce more varied responses; lower values make the model more predictable.
  • System Prompt
    • A hidden set of instructions that defines the assistant’s persona, constraints, and behavior.

Hardware & Performance

  • VRAM (Video RAM)
    • Dedicated memory on a GPU, required to load and run model weights.
  • CUDA
    • NVIDIA’s parallel computing platform and API for GPU acceleration.
  • ROCm
    • AMD’s open-source software stack for GPU-accelerated computing.
  • CPU Offloading
    • A technique that moves parts of the model to system RAM and the CPU when VRAM is insufficient.
  • Accelerate
    • A library that handles device mapping, weight streaming, and CPU offloading.

Model Training & Adaptation

  • Pretrained Model
    • A base model trained on a general dataset before any task-specific tuning.
  • Fine-tuning
    • Further training a pretrained model on a smaller, task-specific dataset to specialize its behavior.
  • LoRA (Low-Rank Adaptation)
    • A lightweight fine-tuning method that adds small, trainable adapter layers to the model instead of updating all parameters.
  • QLoRA
    • A memory-efficient version of LoRA that uses 4-bit quantization, enabling fine-tuning on consumer hardware.
  • Quantization
    • The process of compressing model weights to reduce memory usage and increase speed, usually with a minor quality trade-off. Models are commonly reduced from 16-bit to 4-bit or 8-bit precision.
    • Common formats: Q4_K_M, Q8_0, 4-bit GPTQ, AWQ.

File Formats

  • GGUF
    • The standard format for llama.cpp, optimized for CPU and GPU inference.
  • Safetensors
    • A secure, fast tensor format that prevents arbitrary code execution, replacing the older .bin format.
  • GPTQ / AWQ
    • Quantized formats optimized for fast, GPU-based inference.
  • Checkpoint
    • A file containing a model’s trained weights (e.g., .safetensors, .gguf).

The Anatomy of a Model Directory

When downloading a model from Hugging Face, you will typically find these files:

  • config.json
    • Defines the model architecture: layers, attention heads, vocabulary size, and model type.
  • .safetensors (preferred) or .bin
    • The trained weights, the model’s “knowledge.”
  • tokenizer.json
    • Defines the vocabulary and rules for converting text into tokens.
  • tokenizer_config.json
    • Tokenizer settings such as padding and maximum length.
  • special_tokens_map.json
    • Maps functional tokens like <bos> (start), <eos> (end), and <pad>.
  • generation_config.json
    • Optional default inference settings (temperature, top_p, penalty).
  • preprocessor_config.json
    • Required for multimodal models (vision/speech) to resize or normalize inputs.
  • model_index.json
    • Metadata used by the Hugging Face Hub and its pipelines.

Diffusers (Multimodal Pipelines)

For image, video, or audio generation, models use a modular directory structure called a Pipeline:

  • model_index.json
    • The blueprint for the entire pipeline.
  • vae/
    • The Variational Autoencoder, which encodes images into a latent representation and decodes them back to pixels.
  • unet/ / text_encoder/
    • Components that guide the generation process based on text prompts.
  • scheduler/
    • Defines the noise schedule and the number of steps taken during inference.
  • safety_checker/ / feature_extractor/
    • Optional components for detecting and filtering NSFW content.