{
  "id": "prompt-tuning",
  "title": "Prompt Tuning",
  "lang": "en",
  "source_lang": "en",
  "summary": "Prompt tuning adapts a pretrained model to a task by training continuous prompt vectors while keeping the model’s original parameters fixed.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Soft prompt tuning",
    "Continuous prompt tuning"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/prompt-tuning/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/prompt-tuning/",
      "json": "https://aiwiki.page/zh/prompt-tuning.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Prompt tuning is a method for adapting a pretrained machine learning model by learning a small set of continuous input vectors, called soft prompts, while leaving the model’s original parameters unchanged. It is most closely associated with large language models, where learned vectors are typically prepended to the representations of an input sequence. Unlike ordinary written instructions, these vectors are optimized from training examples and need not correspond to readable words. Prompt tuning belongs to parameter-efficient fine-tuning, a family of methods that adapt models by training relatively few parameters."
    },
    {
      "heading": "Terminology and scope",
      "text": "In its narrow, widely used technical sense, prompt tuning means learning prompt vectors at a model’s input layer. The term is sometimes used more broadly for methods that learn continuous prompts at multiple locations within a model. The location of the learned parameters should therefore be specified when comparing implementations.\n\nPrompt tuning differs from several related practices:\n\n- Prompt engineering involves constructing or revising textual instructions, examples, or formatting. It does not necessarily involve numerical optimization or training.\n- Discrete prompt optimization searches for prompts made of actual vocabulary tokens. Soft prompts instead occupy a continuous vector space.\n- In-context learning conditions a model on examples included in its context, usually without updating parameters. Prompt tuning updates the soft prompt parameters during a separate training stage.\n- Full fine-tuning updates all or most of the pretrained model’s parameters. Prompt tuning freezes those parameters and learns a comparatively small task-specific addition.\n\nA trained soft prompt is therefore a parameter artifact used alongside a model, not simply a reusable paragraph of instructions."
    },
    {
      "heading": "Mechanism",
      "text": "Continuous prompt representations\n\nA language model normally converts text into tokens through tokenization and maps those tokens to vectors through an embedding layer. These representations are related to word embeddings, although modern models commonly use subword or byte-based tokens rather than whole words.\n\nLet an input sequence \\(x\\) have \\(n\\) tokens, with embedding dimension \\(d\\). Its embedded representation is\n\n\\[E(x)\\in\\mathbb{R}^{n\\times d}.\\]\n\nA soft prompt with \\(m\\) positions is a trainable matrix\n\n\\[P\\in\\mathbb{R}^{m\\times d}.\\]\n\nIn the basic input-level formulation, the model receives the concatenation\n\n\\[[P;E(x)].\\]\n\nThe prompt vectors participate in the model’s computations in much the same way as other input representations. In a transformer, they can influence later representations through attention mechanisms. The vectors do not have to equal any entries in the model’s vocabulary embedding table, and their nearest vocabulary tokens do not necessarily provide a meaningful interpretation.\n\nImplementations must also handle attention masks and positional information appropriately. In an encoder–decoder model, the soft prompt can be attached to the encoder input; in a decoder-only model, it commonly precedes the text to be processed or completed.\n\nTraining objective\n\nLet \\(\\theta\\) denote the frozen pretrained parameters and \\(P\\) the trainable prompt. For a dataset \\(D\\) of input–target pairs, a common objective is\n\n\\[P^{*} = \\underset{P}{\\operatorname{arg\\,min}} \\left[ -\\sum_{(x,y)\\in D}\\log p_{\\theta}(y\\mid P,x) \\right].\\]\n\nFor a target sequence \\(y=(y_1,\\ldots,y_T)\\), the conditional likelihood is usually factorized autoregressively:\n\n\\[\\log p_{\\theta}(y\\mid P,x) = \\sum_{t=1}^{T} \\log p_{\\theta}(y_t\\mid P,x,y_{<t}).\\]\n\nThis is a standard supervised learning formulation. Other tasks can use different objectives.\n\nTraining propagates gradients through the frozen model to the prompt vectors using backpropagation. “Frozen” means that the pretrained parameters are not updated; it does not mean that the model can be excluded from gradient computation. The numerical optimizer changes \\(P\\), while \\(\\theta\\) remains fixed.\n\nInitialization and prompt length\n\nSoft prompts may be initialized randomly or from embeddings of vocabulary tokens, including tokens related to task labels or a written instruction. Initialization can affect convergence and final performance, particularly when training data are limited.\n\nPrompt length is a hyperparameter. A longer prompt adds trainable capacity, but also consumes sequence positions and increases computation. Its benefit depends on the model, task, dataset, and optimization setup; increasing its length does not guarantee better results."
    },
    {
      "heading": "Historical development",
      "text": "Prompt tuning developed from research on adapting pretrained language models through prompts rather than task-specific architectural changes. Earlier work demonstrated that downstream tasks could be expressed as text completion or generation, but performance often depended strongly on the wording of the prompt.\n\nSeveral influential continuous-prompt approaches appeared in 2021. Xiang Lisa Li and Percy Liang introduced prefix-tuning, which learns continuous prefixes affecting transformer computation at multiple layers. Xiao Liu and colleagues introduced P-Tuning, which used trainable continuous prompts and a prompt encoder to reduce reliance on manually designed templates.\n\nBrian Lester, Rami Al-Rfou, and Noah Constant’s 2021 paper, The Power of Scale for Parameter-Efficient Prompt Tuning, established the simple input-level formulation widely associated with the term. Their experiments with T5 models found that the performance gap between prompt tuning and full model tuning narrowed as model size increased. This was an empirical finding for the evaluated settings, not a guarantee that prompt tuning will match full fine-tuning for every model and task.\n\nLater work explored prompts at multiple layers, prompt transfer between tasks, and applications beyond text."
    },
    {
      "heading": "Relationship to other adaptation methods",
      "text": "Prefix-tuning and deep prompts\n\nPrefix-tuning typically learns representations that are injected into attention computations across multiple transformer layers, often as additional key and value states. Basic prompt tuning instead learns embeddings at the input layer.\n\nDeep prompt methods introduce learned prompts at several layers. They provide more direct control over intermediate representations but differ from input-level prompt tuning in their parameterization and computational effects. Terminology varies across publications and software libraries, so descriptions such as “soft prompting” do not by themselves identify the exact method.\n\nAdapters and low-rank adaptation\n\nAdapters add small trainable modules within the model. Low-rank adaptation learns low-rank updates to selected weight matrices.\n\nThese methods share prompt tuning’s objective of reducing task-specific training and storage requirements, but intervene in different places:\n\nMethod | Typical trainable component | Original model weights\nInput-level prompt tuning | Input prompt vectors | Unchanged\nPrefix-tuning | Layerwise prefix representations | Unchanged\nAdapters | Added internal modules | Usually unchanged\nLow-rank adaptation | Low-rank weight updates | Frozen during adaptation\nFull fine-tuning | Most or all model parameters | Updated\n\nSmall trainable parameter counts do not imply identical accuracy, memory use, or inference cost across these approaches."
    },
    {
      "heading": "Applications",
      "text": "Prompt tuning is used to adapt shared pretrained models to tasks such as text classification, machine translation, summarization, and question answering. Classification can be expressed as generation of label words or other designated outputs, allowing a language model’s existing output mechanism to be reused.\n\nA central application is multitask deployment. A service can retain one large frozen model and load a different compact prompt for each task. This reduces the need to store a complete model copy for every adaptation. It does not eliminate the memory required to run the shared model.\n\nRelated techniques have also been applied in computer vision and multimodal learning. Learned prompts may condition text encoders, image encoders, or both. Visual prompt methods can introduce trainable tokens into vision transformers or use other trainable input modifications. These methods share the principle of adapting a frozen model through a small learned conditioning component, but their implementation differs from text-only prompt tuning."
    },
    {
      "heading": "Efficiency and deployment",
      "text": "For a basic prompt with length \\(m\\) and embedding dimension \\(d\\), the number of trainable parameters is\n\n\\[N_{\\text{prompt}}=md.\\]\n\nFor example, 100 prompt positions with dimension 4,096 contain 409,600 parameters. That is small relative to a model containing billions of parameters, although storage also depends on numerical precision and associated metadata.\n\nThe principal efficiency benefits are reduced task-specific checkpoint storage, fewer parameter gradients and optimizer states, and the ability to share a frozen backbone across tasks. Training still requires forward and backward computation through much of the model, so parameter savings should not be equated directly with proportional reductions in training time or total memory.\n\nAt inference time, soft prompts generally add sequence positions and associated computation. They may also occupy part of the available context window. Exact costs depend on the architecture and implementation.\n\nDeployment requires access to a model interface that can accept learned embeddings or an equivalent supported representation. An ordinary text-only API cannot generally reproduce an arbitrary soft prompt. Prompts are also usually tied to a particular model, embedding space, and insertion convention; changing the backbone can make them ineffective."
    },
    {
      "heading": "Limitations and evaluation",
      "text": "Performance and optimization\n\nPrompt tuning can be competitive with more extensive adaptation, but results depend on model scale, pretraining, architecture, task, prompt length, and training data. Input-level prompts can offer less expressive control than methods that modify internal layers or weights.\n\nDespite the small number of learned parameters, optimization can be sensitive to initialization, learning rate, and training duration. Limited data can still lead to overfitting. A frozen backbone does not make the learned prompt immune to memorizing task-specific patterns or examples.\n\nInterpretability and portability\n\nSoft prompts are not naturally readable instructions. Their effects are distributed through continuous representations and subsequent model computation. Mapping each vector to its nearest vocabulary token generally does not yield an equivalent textual prompt.\n\nPortability is also limited: a prompt trained for one model is not inherently reusable with another, even when the models perform similar tasks.\n\nEvaluation requirements\n\nComparisons should distinguish trainable parameter count from total training memory, computation, inference latency, and final performance. Results also depend on whether competing methods receive comparable training data and hyperparameter-search budgets.\n\nReliable evaluation uses a separate validation set for model selection and an untouched test set for final measurement. Performance on new domains should be assessed explicitly because parameter-efficient adaptation does not guarantee robustness under distribution shift."
    }
  ],
  "links": [
    "machine-learning",
    "large-language-model",
    "prompt-engineering",
    "in-context-learning",
    "fine-tuning",
    "tokenization",
    "word-embedding",
    "matrix",
    "transformer-architecture",
    "attention-mechanism",
    "supervised-learning",
    "backpropagation",
    "hyperparameter",
    "low-rank-adaptation",
    "machine-translation",
    "automatic-text-summarization",
    "computer-vision",
    "multimodal-learning",
    "neural-network-inference",
    "context-window",
    "learning-rate",
    "overfitting",
    "validation-set",
    "test-set",
    "distribution-shift"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}