{
  "id": "low-rank-adaptation",
  "title": "Low-rank adaptation",
  "lang": "en",
  "source_lang": "en",
  "summary": "Low-rank adaptation efficiently fine-tunes pretrained neural networks by learning small, factorized weight updates while keeping their original parameters frozen.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "LoRA",
    "Low-Rank Adaptation of Large Language Models"
  ],
  "generated_at": "2026-11-02",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/low-rank-adaptation/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/low-rank-adaptation/",
      "json": "https://aiwiki.page/zh/low-rank-adaptation.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Low-rank adaptation, commonly abbreviated LoRA, is a method for adapting a pretrained neural network to a new task by training a small set of additional parameters rather than updating all its original weights. It represents changes to selected weight matrices as products of two smaller matrices. The pretrained weights remain frozen, while the added matrices are optimized using the task’s training objective. LoRA is a prominent form of parameter-efficient fine-tuning, particularly for large language models and image-generation systems.\n\nThe method was introduced by Edward J. Hu and colleagues in a paper released in 2021 and presented at the International Conference on Learning Representations in 2022. Its importance lies in separating the cost of storing and optimizing task-specific changes from the much larger size of the pretrained model. Multiple adaptations can share a single base model, and an adaptation can often be merged into the original weights for inference without adding new network layers.\n\nLoRA does not make the underlying model smaller or eliminate the computations needed to run it. Instead, it restricts how its weights can change during fine-tuning. Its effectiveness depends on the selected layers, the rank of the update, optimization settings, and the relationship between the pretrained model and the target task. Subsequent developments have extended the approach to quantized models, adaptive rank allocation, alternative initialization schemes, and multi-adapter deployment."
    },
    {
      "heading": "Mathematical formulation and low-rank structure",
      "text": "A dense neural-network layer typically applies a linear transformation represented by a weight matrix. Let the pretrained weight be \\(W_0\\in\\mathbb{R}^{d_{\\mathrm{out}}\\times d_{\\mathrm{in}}}\\), and let \\(x\\in\\mathbb{R}^{d_{\\mathrm{in}}}\\) be its input. Ordinary fine-tuning directly updates the entries of \\(W_0\\). LoRA instead uses\n\n\\[h=W_0x+sBAx,\\]\n\nwhere \\(A\\in\\mathbb{R}^{r\\times d_{\\mathrm{in}}}\\), \\(B\\in\\mathbb{R}^{d_{\\mathrm{out}}\\times r}\\), and \\(s\\) is a scaling factor. Only \\(A\\) and \\(B\\) are trained. The effective weight update is therefore\n\n\\[\\Delta W=sBA,\\qquad \\operatorname{rank}(\\Delta W)\\leq r.\\]\n\nThe integer \\(r\\), called the rank, is usually much smaller than either dimension of the original matrix. The rank bound is the defining restriction of the method: the update acts through an intermediate space of dimension \\(r\\), rather than assigning an independently trainable value to every entry of the full weight matrix.\n\nThe number of trainable parameters changes from \\(d_{\\mathrm{out}}d_{\\mathrm{in}}\\) to \\(r(d_{\\mathrm{out}}+d_{\\mathrm{in}})\\). For a square \\(4096\\times4096\\) matrix, full fine-tuning involves 16,777,216 weights. A rank-8 LoRA update uses 65,536 trainable parameters, approximately 0.39 percent of that number. This comparison concerns one adapted matrix; the trainable fraction of an entire network also depends on how many matrices receive adapters and whether biases, embeddings, or output layers are trained separately.\n\nThe factorization is related to concepts in linear algebra, but LoRA is not ordinarily obtained by computing a low-rank approximation to a previously learned full update. Its two factors are trained directly. Although the update is low-rank, the effective matrix \\(W_0+\\Delta W\\) can remain full-rank. LoRA therefore constrains the change to a layer rather than compressing the pretrained layer itself.\n\nThe parameters are also non-unique. For any invertible \\(r\\times r\\) matrix \\(R\\), replacing \\(B\\) with \\(BR\\) and \\(A\\) with \\(R^{-1}A\\) leaves \\(BA\\) unchanged. Moreover, the set of matrices of rank at most \\(r\\) is not a single fixed linear subspace: both factors can move during training. This makes LoRA more flexible than learning coefficients in one permanently fixed collection of update directions, while still imposing a substantial structural constraint."
    },
    {
      "heading": "Origins and relationship to other adaptation methods",
      "text": "LoRA emerged from the broader problem of adapting increasingly large pretrained models without maintaining a separate fully trained copy for every task. In conventional transfer learning, a model first learns general representations from a large corpus and then undergoes additional optimization on task-specific training data. As pretrained networks grew from millions to billions of parameters, the storage of model copies and optimizer states became a practical obstacle, even when a downstream dataset was relatively small.\n\nEarlier parameter-efficient methods approached this problem in several ways. Adapter modules inserted small trainable networks between existing layers. Prompt tuning learned continuous input representations while freezing the model, and prefix tuning supplied trainable representations to attention computations. Other approaches trained only selected existing parameters, such as biases. These methods differ not only in parameter count but also in where they intervene in the computation and whether they add operations or sequence positions during inference.\n\nThe LoRA paper, LoRA: Low-Rank Adaptation of Large Language Models, proposed learning low-rank weight changes while leaving the pretrained model untouched. Its authors included Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. The study evaluated the method on models including RoBERTa, DeBERTa, GPT-2, and GPT-3, and investigated both downstream task performance and properties of the learned update matrices. RoBERTa is a BERT-derived model rather than BERT itself.\n\nFor the 175-billion-parameter GPT-3 setting examined in the original work, the authors reported approximately 10,000 times fewer trainable parameters and a threefold reduction in GPU memory requirements relative to fine-tuning with the Adam optimizer. These figures described particular experimental configurations, not universal performance guarantees. The magnitude of savings depends on precision, optimizer implementation, sequence length, selected target matrices, and the treatment of activations.\n\nLoRA’s distinguishing deployment property is that its update can be added directly to an existing weight matrix. Unlike a nonlinear adapter inserted between layers, the basic LoRA branch does not require a permanent architectural extension after merging. Unlike prompt-based adaptation, it does not inherently consume positions in the model’s context window. These differences helped make it attractive for adapting foundation models across many tasks. Nevertheless, LoRA belongs to a family of alternatives rather than replacing them categorically: prompt methods, inserted adapters, selective parameter updates, and full fine-tuning retain different advantages under different operational constraints."
    },
    {
      "heading": "Initialization, scaling, and optimization",
      "text": "Standard LoRA initialization is designed to preserve the pretrained network’s output at the beginning of adaptation. A common convention initializes \\(A\\) with small random values and \\(B\\) to zero, giving \\(BA=0\\). The network therefore starts with exactly the original linear transformation, apart from any other modifications made by the training procedure. If both matrices were initialized to zero, ordinary gradient-based training would initially have no nonzero gradient for either factor, preventing learning from starting.\n\nThe asymmetry can be seen directly from backpropagation. If \\(G\\) denotes the gradient of the loss with respect to the effective weight matrix, then, with the base weight frozen,\n\n\\[\\nabla_B\\mathcal{L}=sGA^{\\mathsf T}, \\qquad \\nabla_A\\mathcal{L}=sB^{\\mathsf T}G.\\]\n\nAt initialization, \\(B=0\\) makes the gradient for \\(A\\) zero, but the randomly initialized \\(A\\) generally permits a nonzero gradient for \\(B\\). Once \\(B\\) changes, both factors can learn. These equations also show why optimization in the two factors is not equivalent to applying an ordinary gradient step directly to a dense update matrix.\n\nThe original formulation commonly uses \\(s=\\alpha/r\\), where \\(\\alpha\\) is a scale hyperparameter. The scaling separates the magnitude of the adapter contribution from some effects of changing the rank. However, keeping \\(\\alpha\\) fixed while increasing \\(r\\) is not equivalent to keeping \\(\\alpha/r\\) fixed. Rank-stabilized LoRA, usually called rsLoRA, uses a scaling proportional to \\(\\alpha/\\sqrt{r}\\), motivated by the behavior of activations and gradients as rank changes. Rank comparisons therefore depend on the scaling convention as well as on the number of trainable parameters.\n\nAdapters are often optimized with Adam or AdamW. Their learning rate need not match the value used for full fine-tuning, because the parameterization and the number of trainable variables differ. Some methods also assign different learning rates to the two factors. Dropout may be applied to the input of the adapter branch, providing a form of regularization without changing the frozen base branch during training.\n\nLoRA itself does not specify a loss function. It can be used with next-token prediction, classification, sequence-to-sequence objectives, or other differentiable tasks. The training objective determines the desired behavior; the low-rank parameterization determines which weight changes are available to achieve it. Token masking, label construction, batch composition, and numerical precision consequently remain important parts of the adaptation procedure rather than being resolved by the use of LoRA."
    },
    {
      "heading": "Placement in transformers and rank allocation",
      "text": "In a transformer, LoRA can be applied to several types of linear transformation. These include the query, key, value, and output projections in self-attention, as well as the up, down, and gating projections in feed-forward blocks. Embedding and output matrices can also be adapted, although they require appropriate implementations and may interact with weight tying. The target choice determines both the parameter budget and the kinds of internal computations that can change.\n\nThe original LoRA experiments emphasized attention projections, particularly query and value matrices in several configurations. Adapting a query projection changes how a token selects information from other positions; adapting a value projection changes the information transmitted once attention weights have been computed. Changes to key and output projections affect other parts of the same process. Because multi-head attention distributes computation across heads and layers, a low-rank change to a shared projection can influence many token relationships rather than functioning as a simple task-specific output correction.\n\nLater practice has often extended adaptation to most or all large linear layers. This gives the optimizer more locations at which to change representations, including the feed-forward blocks that contain a substantial share of many transformer models’ parameters. More extensive coverage can improve adaptation, but it also increases adapter size, optimizer memory, and computation. A low rank applied broadly and a higher rank applied narrowly are different model classes even when their total trainable parameter counts are similar.\n\nFor a collection of adapted matrices indexed by \\(\\ell\\), the parameter budget is\n\n\\[P_{\\mathrm{LoRA}} = \\sum_{\\ell} r_\\ell \\left( d_{\\mathrm{in},\\ell} + d_{\\mathrm{out},\\ell} \\right).\\]\n\nThis expression makes it possible to allocate different ranks to different layers. Uniform rank is simple, but it assumes equal importance despite differences in layer width, depth, and function. Adaptive approaches attempt to direct capacity toward updates judged more useful during training. AdaLoRA, introduced in 2023, uses an importance-based allocation procedure with a singular-value-style parameterization and pruning under a specified budget.\n\nThe relevant rank is local to each adapted matrix. A network with rank-8 adapters in many layers is not restricted to an eight-dimensional change in its overall input–output behavior. Nonlinear activation functions, attention operations, and repeated composition make the global effect substantially richer. Conversely, increasing local rank does not guarantee a measurable quality improvement: the task may already be well represented at lower rank, or optimization and data limitations may dominate the result."
    },
    {
      "heading": "Memory, computation, and quantized adaptation",
      "text": "LoRA’s largest training-memory benefit usually comes from reducing trainable parameter state. Full fine-tuning may require model weights, gradients, and optimizer accumulators for every parameter, sometimes with additional higher-precision copies. With LoRA, these gradient and optimizer allocations are needed only for the adapters and any other deliberately unfrozen parameters. The base weights still occupy memory, and the network still has to perform its forward computation.\n\nFreezing parameters does not eliminate all backward computation through the pretrained network. Gradients must propagate through intermediate operations to reach adapters in earlier layers. Stored activations can therefore remain a major memory cost, especially with long sequences and large batches. Activation checkpointing reduces this storage by recomputing selected forward operations during the backward pass. Memory-efficient attention implementations such as FlashAttention address another part of the memory problem and can be combined with LoRA.\n\nFor a single adapted matrix, the additional branch costs approximately \\(O(r(d_{\\mathrm{in}}+d_{\\mathrm{out}}))\\) operations per input vector, alongside the base multiplication’s \\(O(d_{\\mathrm{in}}d_{\\mathrm{out}})\\) cost. This is small when \\(r\\) is small, but execution overhead and hardware utilization also matter. Many small matrix operations may use a GPU less efficiently than their arithmetic counts suggest. Consequently, reductions in trainable parameter count should not be interpreted as equal reductions in training time.\n\nQLoRA, introduced by Tim Dettmers and colleagues in 2023, combines low-rank adaptation with a quantized frozen base model. Its principal components include 4-bit NormalFloat quantization, double quantization of quantization constants, and paged optimizers. Low-bit storage reduces the memory occupied by base weights, while computations use dequantized representations in a suitable arithmetic format. Gradients pass through the frozen model’s computations to update the adapters; the low-bit base parameters themselves are not ordinarily optimized.\n\nThe QLoRA study reported fine-tuning a 65-billion-parameter model on a single 48 GB GPU under its experimental conditions. This result illustrated the combined value of base-weight compression and parameter-efficient optimization, rather than a capability of unquantized LoRA alone. Actual memory use remains sensitive to sequence length, batch size, optimizer behavior, and implementation.\n\nQuantization introduces approximation error, so an adapter trained against a quantized base is not necessarily interchangeable with one trained against a higher-precision base. Merging an update and then requantizing can produce a different effective model from keeping a quantized base and a separate higher-precision adapter. These numerical distinctions matter when evaluating whether a deployed artifact reproduces the behavior observed during training."
    },
    {
      "heading": "Merging, storage, and multi-adapter serving",
      "text": "For a standard LoRA layer, inference can use the merged matrix\n\n\\[W_{\\mathrm{merged}}=W_0+sBA.\\]\n\nThe resulting operation has the same matrix dimensions as the original layer. Once the update has been merged, the separate low-rank branch is unnecessary, so basic LoRA need not add inference operations or new sequence positions. This statement assumes that merging is supported by the deployment format and that the adapter has the usual linear form; quantization and other transformations can complicate the numerical equivalence.\n\nKeeping the adapter separate offers a different benefit. A serving system can hold one base model and load different small adapters for different tasks, domains, or users. The stored artifact typically contains the learned factors, scaling settings, target-module specification, and other required configuration. Any separately trained biases, embeddings, or output heads must also be preserved. The adapter is therefore much smaller than a full model in common settings, but it is not a standalone substitute for the base model.\n\nCompatibility requires more than matching the advertised model size. The base architecture, exact weight version, target tensor shapes, tokenizer, and relevant preprocessing must agree. An adapter trained on one checkpoint may load successfully onto another checkpoint with identical shapes yet produce different behavior. Reproducible deployment also depends on prompt formatting and generation settings. Model version identifiers are consequently important parts of adapter metadata.\n\nMultiple updates can be combined algebraically. If several adapters share the same base, a weighted combination has the form\n\n\\[\\Delta W=\\sum_{i=1}^{k}\\lambda_i B_iA_i.\\]\n\nThe combined rank is at most the sum of the component ranks, although overlapping update directions can make it smaller. Concatenating appropriately scaled factors can represent the sum exactly before any further compression. A singular value decomposition can subsequently produce a lower-rank approximation. However, exact addition of weight updates does not imply that the resulting model combines task abilities without interference, because the network’s overall behavior is nonlinear.\n\nMulti-adapter serving introduces scheduling problems absent from single-model inference. Requests using different adapters can share base-weight computation infrastructure, but their low-rank operations and memory accesses differ. Specialized batching and GPU kernels can reduce the overhead of switching among adapters. Systems may cache popular adapters and load others on demand. This makes adapter selection, memory residency, and request scheduling part of deployment engineering, even though each individual adapter is relatively small."
    },
    {
      "heading": "Applications across language and generative models",
      "text": "In natural language processing, LoRA is used for tasks such as classification, translation, summarization, information extraction, and instruction following. The method changes how an existing model responds to task examples without repeating pretraining. An adaptation can teach an output format, emphasize domain terminology, or alter response style. Its effectiveness depends on whether the base model already has representations and capabilities that can be redirected by the available update.\n\nInstruction tuning commonly trains on sequences containing a prompt and a desired response, with loss applied to selected response tokens. LoRA can supply the trainable parameters for this procedure without determining the instruction format itself. It can also be used in preference-based optimization and reinforcement learning from human feedback, where the objective differs from ordinary supervised prediction. In each case, the distinction between learning algorithm and weight parameterization remains important.\n\nLoRA is also widely used with diffusion models for image generation. Such models contain attention projections and other transformations that can receive low-rank updates. Adaptation may associate a subject with a textual identifier, alter a visual style, or specialize the model for a category of imagery. The stored adapter can then be distributed separately from the much larger base model. Depending on the architecture and training procedure, adaptation may affect the denoising network, a text encoder, or both.\n\nImage personalization illustrates a particular tension between specialization and generalization. A small collection of subject images can support an adapter that reproduces recognizable characteristics, but the model may also memorize backgrounds, poses, or visual details that recur in the examples. Captions, image diversity, training duration, and the chosen adapter capacity influence this behavior. A compact checkpoint does not by itself establish that the learned representation is abstract rather than memorized.\n\nLoRA has also been applied to computer vision, speech recognition, and multimodal learning. It is especially convenient when architectures contain large dense projections, but convolutional adaptations require choices about how to factor convolutional kernels or represent the extra branch. These implementations share the low-rank adaptation principle without necessarily using identical tensor layouts.\n\nAcross these applications, LoRA is distinct from retrieval-augmented generation. An adapter changes model parameters, whereas retrieval supplies external information during inference. Neither approach subsumes the other. A model can use LoRA to learn domain-specific response patterns while retrieving changing factual material from a separate knowledge source."
    },
    {
      "heading": "Extensions and alternative parameterizations",
      "text": "Subsequent methods have modified different parts of the LoRA design: initialization, rank allocation, factor optimization, scaling, and the representation of the update itself. These are separate design choices. A method can preserve the basic low-rank matrix product while changing its training dynamics, or it can introduce a more substantial departure from the original parameterization.\n\nRank-stabilized LoRA addresses the scale of activations and gradients at larger ranks. LoRA+, introduced in 2024, instead studies unequal learning rates for the two factors. Although \\(A\\) and \\(B\\) jointly define the same update matrix, they play different roles in the forward and backward computations, so treating their optimization identically is not always optimal. Neither modification requires the same assumptions as adaptive rank allocation: one changes scaling, another changes optimization, and adaptive rank methods change the distribution of trainable capacity.\n\nWeight-decomposed low-rank adaptation, or DoRA, introduced in 2024, separates aspects of weight magnitude and direction. It applies a low-rank update to the directional component while learning additional magnitude parameters. The aim is to provide behavior closer to full fine-tuning without making the entire dense weight matrix independently trainable. This introduces a normalization-related computation during training, although the resulting effective weights can be materialized for inference.\n\nInitialization methods such as PiSSA use principal singular components of pretrained weights to construct the trainable factors. This differs from the conventional arrangement in which one factor is random and the other is zero. In such decompositions, the frozen residual and trainable component can be arranged so that their sum initially reproduces the original weights. The choice of initial directions can affect convergence even when the eventual adapter parameter count is unchanged.\n\nQuantization-aware initialization is another direction. LoftQ, for example, jointly considers a quantized approximation of the base weight and a low-rank residual intended to reduce the resulting discrepancy. This targets an error introduced by low-bit representation rather than relying on downstream optimization alone to compensate for it. Its objective is therefore different from merely selecting a larger adapter rank.\n\nOther techniques periodically merge a trained low-rank update into the base and initialize a new adapter. Across successive stages, the accumulated effective update can exceed the rank of any one stage, while the number of actively optimized adapter parameters remains limited. This changes the training constraint: the model is no longer restricted to one fixed-rank displacement from its original checkpoint. Such procedures also weaken the simple separation between an unchanged shared base and a small final task-specific adapter."
    },
    {
      "heading": "Evaluation, limitations, and unresolved questions",
      "text": "A meaningful evaluation of LoRA separates task quality, resource use, and deployment cost. Quality measures may include accuracy, exact match, task-specific scores, or perplexity, depending on the application. Resource measurements include peak accelerator memory, training time, trainable parameter count, adapter storage, and inference throughput. A configuration can be superior on one measure and inferior on another; a small adapter does not automatically imply faster optimization or equal performance.\n\nComparisons with full fine-tuning require comparable tuning effort. LoRA and full fine-tuning can prefer different learning rates, regularization strengths, and training durations. Equalizing parameter count alone does not equalize optimization difficulty, while equalizing training steps does not necessarily equalize compute. Placement and rank must also be stated: “LoRA” may describe a few attention projections or nearly every linear transformation in the network.\n\nThe appropriate rank remains task- and model-dependent. Too little capacity can cause underfitting, particularly when the target behavior requires substantial changes across many representations. More rank can improve performance, but gains may saturate or be offset by overfitting on limited data. The relationship between low-rank updates and generalization is therefore empirical rather than a universal guarantee. A low-rank constraint can act as an inductive bias without establishing that every useful adaptation is intrinsically low-rank.\n\nEvaluation must also distinguish genuine task learning from memorization and benchmark contamination. Performance on a held-out test set is informative only when examples and close variants have not leaked into adaptation data or model-selection procedures. Distribution shift can expose failures that are absent from an in-domain evaluation. For generative tasks, human or model-based judgments introduce additional variation, making standardized prompts and clearly defined evaluation procedures important.\n\nFreezing the base model does not freeze its behavior. An adapter can substantially alter outputs and can degrade capabilities outside the target task. It does not inherently preserve safety behavior, guarantee factual accuracy, or prevent disclosure of memorized information. Small adapters may encode sensitive associations from their training data, so parameter efficiency is not itself a privacy mechanism. Their compact size also does not establish that their provenance or training process is trustworthy.\n\nThe theoretical question is why relatively low-rank updates often suffice. Possible explanations involve redundancy in pretrained representations, the limited number of directions needed for a particular task, and the ability of many small local changes to interact across network depth. The matrix rank of an update, the dimensionality of an effective optimization trajectory, and the complexity of the learned behavior are distinct quantities. Establishing reliable relationships among them remains an active problem in understanding parameter-efficient adaptation."
    }
  ],
  "links": [
    "artificial-neural-network",
    "large-language-model",
    "fine-tuning",
    "linear-map",
    "matrix",
    "linear-algebra",
    "transfer-learning",
    "training-data",
    "bert",
    "context-window",
    "foundation-model",
    "backpropagation",
    "hyperparameter",
    "adam-optimizer",
    "learning-rate",
    "dropout",
    "regularization",
    "loss-function",
    "transformer-architecture",
    "self-attention",
    "multi-head-attention",
    "activation-function",
    "flashattention",
    "graphics-processing-unit",
    "qlora",
    "tokenization",
    "singular-value-decomposition",
    "natural-language-processing",
    "machine-translation",
    "reinforcement-learning-from-human-feedback",
    "diffusion-model",
    "computer-vision",
    "speech-recognition",
    "multimodal-learning",
    "retrieval-augmented-generation",
    "perplexity",
    "underfitting",
    "overfitting",
    "generalization",
    "test-set",
    "distribution-shift",
    "data-privacy"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}