{
  "id": "qlora",
  "title": "QLoRA",
  "lang": "en",
  "source_lang": "en",
  "summary": "QLoRA is a memory-efficient fine-tuning method that trains low-rank adapters while keeping a pretrained model’s base weights frozen in a quantized representation.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Quantized low-rank adaptation",
    "Quantized LoRA"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/qlora/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/qlora/",
      "json": "https://aiwiki.page/zh/qlora.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "QLoRA is a method for fine-tuning pretrained neural networks that combines a frozen, low-bit base model with trainable low-rank adaptation (LoRA) modules. Introduced in 2023, it substantially reduces the memory needed to adapt large language models by storing most model weights in a four-bit representation while training only small, higher-precision adapter matrices. The original method combines a four-bit numerical format called NormalFloat, double quantization of scaling constants, and paged optimizers. Its central distinction is that low-precision weight storage does not require all training computations to use the same low precision."
    },
    {
      "heading": "Background and purpose",
      "text": "Conventional full fine-tuning updates all or most parameters of a pretrained neural network. Besides the weights themselves, training requires memory for gradients, optimizer states, intermediate activations, and temporary computation buffers. Optimizers such as Adam maintain additional state for each trainable parameter, making total training memory substantially larger than the memory needed simply to load a model for inference.\n\nLoRA reduces this burden by freezing the original weights and learning a low-rank modification to selected weight matrices. It greatly reduces the number of trainable parameters, but ordinary LoRA still requires the frozen base model to fit in memory, often in a 16-bit floating-point representation.\n\nQLoRA addresses this remaining cost through neural-network quantization. Most base-model weights are stored using four bits per parameter, together with scaling information and other metadata. The adapter parameters and relevant training computations retain higher precision. This combination makes adaptation possible on hardware that could not accommodate conventional full fine-tuning of the same model.\n\nQLoRA is therefore a parameter-efficient training method, not merely a technique for compressing a model after training."
    },
    {
      "heading": "Mathematical formulation",
      "text": "Consider a pretrained linear layer with weight matrix \\(W_0 \\in \\mathbb{R}^{d_{\\mathrm{out}}\\times d_{\\mathrm{in}}}\\). LoRA represents a trainable update as\n\n\\[\\Delta W = sBA,\\]\n\nwhere\n\n\\[A \\in \\mathbb{R}^{r\\times d_{\\mathrm{in}}}, \\qquad B \\in \\mathbb{R}^{d_{\\mathrm{out}}\\times r},\\]\n\nand \\(r\\) is much smaller than the layer dimensions. The scaling factor \\(s\\) is commonly \\(\\alpha/r\\), although other scaling conventions exist.\n\nIn QLoRA, the pretrained matrix is first quantized:\n\n\\[W_q = Q(W_0).\\]\n\nA simplified forward computation is\n\n\\[y = \\operatorname{dequant}(W_q)x + sBAx.\\]\n\nHere, \\(W_q\\) remains frozen, while \\(A\\) and \\(B\\) are optimized against the training loss function. The reconstructed base weights approximate the original weights:\n\n\\[\\operatorname{dequant}(W_q) \\approx W_0.\\]\n\nOnly \\(r(d_{\\mathrm{in}}+d_{\\mathrm{out}})\\) adapter parameters are needed instead of \\(d_{\\mathrm{in}}d_{\\mathrm{out}}\\) independently trainable parameters for that layer.\n\nBackpropagation still passes through computations involving the frozen base model so that gradients can reach the adapters and earlier trainable components. Freezing the base weights eliminates their parameter gradients and optimizer states; it does not eliminate all backward computation through the network."
    },
    {
      "heading": "Core techniques",
      "text": "Four-bit NormalFloat\n\nThe original QLoRA method introduced NormalFloat 4, usually abbreviated NF4, as a four-bit storage format suited to approximately normally distributed weights.\n\nFour bits encode 16 possible values. Unlike a uniformly spaced integer quantizer, NF4 assigns these values nonuniformly, with more resolution near the center of the distribution. Its construction uses quantiles of a normal distribution and includes an exact representation of zero.\n\nWeights are quantized in blocks, each with a scale that maps its values into the range represented by the codebook. The stored representation consists of four-bit codes plus the associated quantization metadata.\n\nThe original paper describes NF4 as information-theoretically optimal for normally distributed weights. This characterization depends on the distributional assumptions and quantization construction; it does not establish that NF4 minimizes every error measure for every model. Actual weight distributions, block sizes, and unusually large values can affect its behavior.\n\nNF4 is also not interchangeable with every four-bit floating-point format. Different four-bit formats use different representable values and may yield different results.\n\nDouble quantization\n\nBlockwise quantization requires scaling constants, which consume memory in addition to the four-bit weight codes. Double quantization reduces this overhead by quantizing the scaling constants themselves.\n\nThe second stage introduces its own scales and metadata, so the final representation is not exactly four bits per parameter. Nevertheless, amortizing the metadata across blocks can substantially reduce its cost. The original QLoRA configuration reported an average saving of approximately 0.37 bits per parameter from double quantization.\n\nThe technique is particularly consequential for models containing billions of parameters, where even a fraction of a bit per weight represents a substantial memory saving.\n\nPaged optimizers\n\nThe original implementation also used paged optimizers to manage temporary memory pressure. These rely on unified-memory mechanisms that can move optimizer-state memory between host memory and GPU memory when necessary.\n\nPaging is distinct from quantization: it changes where optimizer state resides, not its numerical representation. Its purpose is to help absorb memory spikes rather than require all optimizer state to remain permanently resident on the accelerator.\n\nPaging does not eliminate activation memory or guarantee that any model will fit. When transfers between host and device become frequent, they can also reduce training speed."
    },
    {
      "heading": "Storage precision and computation precision",
      "text": "“Four-bit fine-tuning” can misleadingly suggest that every operation uses four-bit arithmetic. In QLoRA, four-bit precision primarily describes the storage of frozen base-model weights.\n\nDuring computation, quantized weights are dequantized into a suitable computation format, commonly a 16-bit format such as bfloat16. Matrix multiplication and gradient calculations then use precisions supported by the implementation and hardware. Adapter parameters, gradients, and optimizer states may use different precisions from one another.\n\nAn efficient implementation need not materialize a full higher-precision copy of the entire model at once. Specialized kernels can reconstruct weights in blocks as they are needed.\n\nConsequently, memory use and speed depend on more than the nominal weight bit width. Kernel support, temporary buffers, adapter placement, and hardware characteristics all matter."
    },
    {
      "heading": "Memory requirements and training behavior",
      "text": "A useful decomposition of training memory is\n\n\\[M_{\\mathrm{total}} = M_{\\mathrm{base}} + M_{\\mathrm{adapters}} + M_{\\mathrm{gradients}} + M_{\\mathrm{optimizer}} + M_{\\mathrm{activations}} + M_{\\mathrm{temporary}}.\\]\n\nQLoRA primarily reduces base-weight storage and the gradient and optimizer-state costs associated with updating the base weights. It does not remove the other terms.\n\nFor \\(N\\) quantized parameters, four-bit codes alone require approximately\n\n\\[M_{\\mathrm{codes}} = \\frac{N}{2}\\ \\text{bytes}.\\]\n\nActual base-model memory is higher because of scales, metadata, and components that remain at higher precision.\n\nActivation memory can become a dominant cost with larger batches or longer context windows. Gradient checkpointing, which recomputes some intermediate results during the backward pass, is often combined with QLoRA to reduce this cost. Memory-efficient attention implementations, including FlashAttention, address another part of the memory budget.\n\nThe original QLoRA paper reported fine-tuning a 65-billion-parameter model on a single GPU with 48 GB of memory. This was a result under a particular configuration, not a universal hardware requirement. Sequence length, batch size, adapter configuration, software implementation, and checkpointing strategy all affect whether a run fits.\n\nReduced memory consumption also does not necessarily imply faster training. Dequantization and memory paging add overhead, and performance depends on the available kernels and hardware."
    },
    {
      "heading": "Development and experimental results",
      "text": "QLoRA was introduced in the 2023 paper QLoRA: Efficient Finetuning of Quantized LLMs, presented at NeurIPS. Its experiments investigated whether low-bit storage could preserve the effectiveness of adapter-based fine-tuning while sharply reducing memory requirements.\n\nThe paper also introduced the Guanaco family of conversational models, produced through instruction fine-tuning with QLoRA. These models illustrated that a quantized, frozen backbone could support effective task adaptation.\n\nThe reported comparisons showed that QLoRA could match higher-precision fine-tuning baselines in the evaluated settings. Such results are empirical findings rather than a general equivalence theorem. Their interpretation depends on the model, training data, training configuration, and evaluation procedure.\n\nConversational evaluations introduce additional uncertainty because outcomes can depend on prompt selection, reference answers, and the preferences of human or automated judges. A favorable benchmark score does not establish equivalent performance on all downstream tasks."
    },
    {
      "heading": "Applications and deployment",
      "text": "QLoRA is used for instruction tuning, domain adaptation, and task-specific adaptation of pretrained language models. It is particularly useful when a base model is available but accelerator memory is insufficient for conventional fine-tuning.\n\nThe resulting adapter checkpoint can be much smaller than the base model. Multiple adapters can therefore be stored for different tasks while sharing a common frozen backbone. Each adapter still depends on the correct base model and a compatible configuration.\n\nAt deployment, a system can retain the quantized base and apply the adapter contribution separately. Alternatively, the adapter update can be merged into dequantized base weights, with optional requantization afterward. Merging or requantization can change numerical behavior and may require substantial temporary memory.\n\nQLoRA is distinct from inference quantization. An inference-quantized checkpoint is intended for efficient execution, whereas QLoRA specifies a way to train adapters around a quantized backbone. Compatibility depends on whether the software supports the relevant quantized operations during training."
    },
    {
      "heading": "Limitations and distinctions",
      "text": "Quantization error and restricted updates\n\nThe frozen base model contains quantization error, while the trainable update is restricted by adapter rank and placement. Adapters may compensate for some effects of quantization, but there is no guarantee that they can recover every behavior of the original higher-precision model.\n\nQLoRA therefore cannot be assumed to reproduce full fine-tuning for every model or task. Results depend on adapter rank, selected layers, learning rate, data, and other hyperparameters.\n\nMemory and hardware constraints\n\nQLoRA reduces memory requirements rather than removing them. Long sequences, large batches, or extensive adapter coverage can still exhaust device memory. Low-bit storage is most effective when supported by suitable kernels; a nominally smaller representation can be slow or inconvenient on unsupported hardware.\n\nRelationship to other adaptation methods\n\nMethod | Base weights during adaptation | Main trainable components\nFull fine-tuning | Updated | All or most model parameters\nStandard LoRA | Frozen, typically at higher precision | Low-rank adapters\nQLoRA | Frozen and quantized | Low-rank adapters\nQuantization-aware training | Quantization effects represented during training | Often the model weights themselves\nPost-training quantization | Quantized after training | Usually no task-adaptation parameters\n\nQLoRA is also distinct from knowledge distillation, which trains a student model using information from a teacher, and from retrieval-augmented generation, which supplies retrieved information during model use. These techniques address different constraints and can be combined.\n\nData quality and generalization\n\nMemory efficiency does not resolve problems of overfitting, data leakage, or distribution shift. A small adapter can still learn undesirable correlations or perform poorly outside its training distribution. Evaluation on appropriately separated test data remains necessary to establish the effects of adaptation."
    }
  ],
  "links": [
    "fine-tuning",
    "low-rank-adaptation",
    "large-language-model",
    "artificial-neural-network",
    "adam-optimizer",
    "neural-network-quantization",
    "matrix",
    "loss-function",
    "backpropagation",
    "graphics-processing-unit",
    "context-window",
    "flashattention",
    "training-data",
    "neural-network-inference",
    "learning-rate",
    "hyperparameter",
    "knowledge-distillation",
    "retrieval-augmented-generation",
    "overfitting",
    "data-leakage",
    "distribution-shift",
    "test-set"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}