Prompt tuning is a method for adapting a pretrained machine learning model by learning a small set of continuous input vectors, called soft prompts, while leaving the model’s original parameters unchanged. It is most closely associated with large language models, where learned vectors are typically prepended to the representations of an input sequence. Unlike ordinary written instructions, these vectors are optimized from training examples and need not correspond to readable words. Prompt tuning belongs to parameter-efficient fine-tuning, a family of methods that adapt models by training relatively few parameters.
Terminology and scope
In its narrow, widely used technical sense, prompt tuning means learning prompt vectors at a model’s input layer. The term is sometimes used more broadly for methods that learn continuous prompts at multiple locations within a model. The location of the learned parameters should therefore be specified when comparing implementations.
Prompt tuning differs from several related practices:
- Prompt engineering involves constructing or revising textual instructions, examples, or formatting. It does not necessarily involve numerical optimization or training.
- Discrete prompt optimization searches for prompts made of actual vocabulary tokens. Soft prompts instead occupy a continuous vector space.
- In-context learning conditions a model on examples included in its context, usually without updating parameters. Prompt tuning updates the soft prompt parameters during a separate training stage.
- Full fine-tuning updates all or most of the pretrained model’s parameters. Prompt tuning freezes those parameters and learns a comparatively small task-specific addition.
A trained soft prompt is therefore a parameter artifact used alongside a model, not simply a reusable paragraph of instructions.
Mechanism
Continuous prompt representations
A language model normally converts text into tokens through tokenization and maps those tokens to vectors through an embedding layer. These representations are related to word embeddings, although modern models commonly use subword or byte-based tokens rather than whole words.
Let an input sequence have tokens, with embedding dimension . Its embedded representation is
A soft prompt with positions is a trainable matrix
In the basic input-level formulation, the model receives the concatenation
The prompt vectors participate in the model’s computations in much the same way as other input representations. In a transformer, they can influence later representations through attention mechanisms. The vectors do not have to equal any entries in the model’s vocabulary embedding table, and their nearest vocabulary tokens do not necessarily provide a meaningful interpretation.
Implementations must also handle attention masks and positional information appropriately. In an encoder–decoder model, the soft prompt can be attached to the encoder input; in a decoder-only model, it commonly precedes the text to be processed or completed.
Training objective
Let denote the frozen pretrained parameters and the trainable prompt. For a dataset of input–target pairs, a common objective is
For a target sequence , the conditional likelihood is usually factorized autoregressively:
This is a standard supervised learning formulation. Other tasks can use different objectives.
Training propagates gradients through the frozen model to the prompt vectors using backpropagation. “Frozen” means that the pretrained parameters are not updated; it does not mean that the model can be excluded from gradient computation. The numerical optimizer changes , while remains fixed.
Initialization and prompt length
Soft prompts may be initialized randomly or from embeddings of vocabulary tokens, including tokens related to task labels or a written instruction. Initialization can affect convergence and final performance, particularly when training data are limited.
Prompt length is a hyperparameter. A longer prompt adds trainable capacity, but also consumes sequence positions and increases computation. Its benefit depends on the model, task, dataset, and optimization setup; increasing its length does not guarantee better results.
Historical development
Prompt tuning developed from research on adapting pretrained language models through prompts rather than task-specific architectural changes. Earlier work demonstrated that downstream tasks could be expressed as text completion or generation, but performance often depended strongly on the wording of the prompt.
Several influential continuous-prompt approaches appeared in 2021. Xiang Lisa Li and Percy Liang introduced prefix-tuning, which learns continuous prefixes affecting transformer computation at multiple layers. Xiao Liu and colleagues introduced P-Tuning, which used trainable continuous prompts and a prompt encoder to reduce reliance on manually designed templates.
Brian Lester, Rami Al-Rfou, and Noah Constant’s 2021 paper, The Power of Scale for Parameter-Efficient Prompt Tuning, established the simple input-level formulation widely associated with the term. Their experiments with T5 models found that the performance gap between prompt tuning and full model tuning narrowed as model size increased. This was an empirical finding for the evaluated settings, not a guarantee that prompt tuning will match full fine-tuning for every model and task.
Later work explored prompts at multiple layers, prompt transfer between tasks, and applications beyond text.
Relationship to other adaptation methods
Prefix-tuning and deep prompts
Prefix-tuning typically learns representations that are injected into attention computations across multiple transformer layers, often as additional key and value states. Basic prompt tuning instead learns embeddings at the input layer.
Deep prompt methods introduce learned prompts at several layers. They provide more direct control over intermediate representations but differ from input-level prompt tuning in their parameterization and computational effects. Terminology varies across publications and software libraries, so descriptions such as “soft prompting” do not by themselves identify the exact method.
Adapters and low-rank adaptation
Adapters add small trainable modules within the model. Low-rank adaptation learns low-rank updates to selected weight matrices.
These methods share prompt tuning’s objective of reducing task-specific training and storage requirements, but intervene in different places:
| Method | Typical trainable component | Original model weights |
|---|---|---|
| Input-level prompt tuning | Input prompt vectors | Unchanged |
| Prefix-tuning | Layerwise prefix representations | Unchanged |
| Adapters | Added internal modules | Usually unchanged |
| Low-rank adaptation | Low-rank weight updates | Frozen during adaptation |
| Full fine-tuning | Most or all model parameters | Updated |
Small trainable parameter counts do not imply identical accuracy, memory use, or inference cost across these approaches.
Applications
Prompt tuning is used to adapt shared pretrained models to tasks such as text classification, machine translation, summarization, and question answering. Classification can be expressed as generation of label words or other designated outputs, allowing a language model’s existing output mechanism to be reused.
A central application is multitask deployment. A service can retain one large frozen model and load a different compact prompt for each task. This reduces the need to store a complete model copy for every adaptation. It does not eliminate the memory required to run the shared model.
Related techniques have also been applied in computer vision and multimodal learning. Learned prompts may condition text encoders, image encoders, or both. Visual prompt methods can introduce trainable tokens into vision transformers or use other trainable input modifications. These methods share the principle of adapting a frozen model through a small learned conditioning component, but their implementation differs from text-only prompt tuning.
Efficiency and deployment
For a basic prompt with length and embedding dimension , the number of trainable parameters is
For example, 100 prompt positions with dimension 4,096 contain 409,600 parameters. That is small relative to a model containing billions of parameters, although storage also depends on numerical precision and associated metadata.
The principal efficiency benefits are reduced task-specific checkpoint storage, fewer parameter gradients and optimizer states, and the ability to share a frozen backbone across tasks. Training still requires forward and backward computation through much of the model, so parameter savings should not be equated directly with proportional reductions in training time or total memory.
At inference time, soft prompts generally add sequence positions and associated computation. They may also occupy part of the available context window. Exact costs depend on the architecture and implementation.
Deployment requires access to a model interface that can accept learned embeddings or an equivalent supported representation. An ordinary text-only API cannot generally reproduce an arbitrary soft prompt. Prompts are also usually tied to a particular model, embedding space, and insertion convention; changing the backbone can make them ineffective.
Limitations and evaluation
Performance and optimization
Prompt tuning can be competitive with more extensive adaptation, but results depend on model scale, pretraining, architecture, task, prompt length, and training data. Input-level prompts can offer less expressive control than methods that modify internal layers or weights.
Despite the small number of learned parameters, optimization can be sensitive to initialization, learning rate, and training duration. Limited data can still lead to overfitting. A frozen backbone does not make the learned prompt immune to memorizing task-specific patterns or examples.
Interpretability and portability
Soft prompts are not naturally readable instructions. Their effects are distributed through continuous representations and subsequent model computation. Mapping each vector to its nearest vocabulary token generally does not yield an equivalent textual prompt.
Portability is also limited: a prompt trained for one model is not inherently reusable with another, even when the models perform similar tasks.
Evaluation requirements
Comparisons should distinguish trainable parameter count from total training memory, computation, inference latency, and final performance. Results also depend on whether competing methods receive comparable training data and hyperparameter-search budgets.
Reliable evaluation uses a separate validation set for model selection and an untouched test set for final measurement. Performance on new domains should be assessed explicitly because parameter-efficient adaptation does not guarantee robustness under distribution shift.