{
  "id": "clip-model",
  "title": "CLIP",
  "lang": "en",
  "source_lang": "en",
  "summary": "CLIP is a vision–language model that learns shared image and text representations through contrastive training, enabling recognition and retrieval from natural-language descriptions.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "product",
  "aliases": [
    "Contrastive Language–Image Pre-training",
    "CLIP model"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/clip-model/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/clip-model/",
      "json": "https://aiwiki.page/zh/clip-model.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "CLIP, short for Contrastive Language–Image Pre-training, is a family of multimodal machine-learning models developed by OpenAI to associate images with natural-language descriptions. Introduced in January 2021, it learns an image encoder and a text encoder jointly from large collections of image–text pairs. Their outputs occupy a shared representational space in which a matching image and description are trained to have higher similarity than mismatched pairs. CLIP therefore produces representations and similarity scores rather than generating captions or answering questions directly.\n\nIts principal contribution was to make natural language a practical interface for transferring visual knowledge across tasks. Instead of training a separate classifier with a fixed set of labels, CLIP can compare an image with descriptions supplied at inference time, such as “a photograph of a bicycle” or “a photograph of a dog.” This supports zero-shot learning, cross-modal search, and reusable visual representation learning. The original work also showed that training with broad web-derived supervision could improve performance on some distribution shifts, although it did not eliminate bias, dataset dependence, or failures of visual reasoning.\n\nThe name refers both to the original training method and to models trained using it. Later implementations often retain CLIP’s dual-encoder structure while changing the data, architecture, training objective, or scale. These variants share an approach, but their capabilities and limitations cannot be assumed to be identical."
    },
    {
      "heading": "Origins and conceptual contribution",
      "text": "Before CLIP, much of modern computer vision centered on supervised learning with manually assigned category labels. A model trained on ImageNet, for example, learned to distinguish a specified vocabulary of objects. Its intermediate representations could transfer to other tasks, but doing so commonly required a new classification head, labeled examples, and additional optimization. The label system was part of the training problem: categories absent from it were not directly available as outputs, even when the visual representation contained relevant information.\n\nLearning from paired images and language was already an established research direction. Earlier systems had investigated image captioning, semantic image embeddings, and matching pictures with descriptions. Methods such as DeViSE connected visual features with linguistic representations, while subsequent work explored joint image–sentence embeddings and web-supervised visual learning. CLIP did not originate the idea of associating visual and linguistic information. Its distinctive contribution was a simple contrastive formulation combined with sufficiently large and varied training data, substantial computation, and systematic evaluation of transfer without task-specific training.\n\nOpenAI introduced CLIP alongside DALL·E on 5 January 2021. The technical paper, Learning Transferable Visual Models From Natural Language Supervision, was authored by Alec Radford and collaborators and appeared at the 2021 International Conference on Machine Learning. The original study trained models on 400 million image–text pairs and examined transfer across more than 30 computer-vision datasets. It compared different visual architectures and scales, rather than presenting only a single fixed network. The release of inference code and pretrained weights also made the approach accessible for experimentation.\n\nThe central conceptual change was to treat descriptions as supervision that could define tasks dynamically. A category such as “Siberian husky” need not correspond to a permanently installed output neuron. Instead, its textual representation can be computed when needed and compared with an image representation. This separates much of the expensive representation-learning process from the specification of downstream categories.\n\nCLIP consequently became an influential example of a foundation model for vision and language: one pretrained system could support multiple downstream uses. This designation does not mean that it possesses general visual understanding. Its broad applicability follows from transferable representations and a flexible text interface, while its training signal remains statistical agreement between images and their accompanying language."
    },
    {
      "heading": "Dual-encoder architecture and representation space",
      "text": "CLIP uses two separately operating neural networks. The image encoder maps an image \\(x\\) to a feature vector \\(f_\\theta(x)\\), and the text encoder maps a text sequence \\(t\\) to a vector \\(g_\\phi(t)\\). Learned projections place the outputs in a common dimensionality. The vectors are then normalized, usually to unit Euclidean length:\n\n\\[v=\\frac{f_\\theta(x)}{\\|f_\\theta(x)\\|_2}, \\qquad u=\\frac{g_\\phi(t)}{\\|g_\\phi(t)\\|_2}.\\]\n\nTheir dot product is then their cosine similarity. Matching is determined by the angle between representations rather than their unnormalized magnitudes.\n\nThe original paper investigated both modified residual convolutional networks and Vision Transformers as image encoders. Its residual networks used changes including anti-aliased downsampling and attention-based pooling. In the transformer versions, an image is divided into patches, the patches are embedded as a sequence, and attention mixes information across that sequence. Model identifiers such as ViT-B/32 and ViT-L/14 indicate an architecture scale and a patch size: the latter uses patches 14 pixels wide and high. Patch size, input resolution, network width, and depth all affect the cost and granularity of visual processing.\n\nThe original text encoder was a Transformer using masked self-attention, with a maximum input sequence length of 77 tokens. Its tokenization used byte-level byte-pair encoding, and a representation at the end-of-text position was projected into the shared space. The causal mask should not be confused with an autoregressive training objective: the text tower was optimized to match images, not to predict each next word. Likewise, the 77-token limit is a property of the original implementation, not a necessary feature of every CLIP-style system.\n\nThe shared embedding space is useful without requiring individual coordinates to have human-readable meanings. Nearby points can reflect object identity, style, scene type, or other associations learned from the data. There is no guarantee that a coordinate represents a discrete visual attribute, or that every linguistic distinction will have an equally clear visual counterpart. The geometry is determined by the matching objective and the distribution of training examples.\n\nUnlike systems that jointly process every image–text pair through cross-attention, CLIP encodes each modality independently. This makes representations reusable: millions of image vectors can be computed before any search query arrives. The trade-off is that the encoders cannot perform detailed, pair-specific interaction during scoring. A single similarity calculation must summarize compatibility, which helps efficiency but can obscure relationships between individual words and particular image regions."
    },
    {
      "heading": "Contrastive objective and optimization",
      "text": "CLIP learns through contrastive learning. A training batch contains \\(N\\) observed image–text pairs, conventionally written \\((x_i,t_i)\\). Each image is compared with every text in the batch, producing an \\(N\\times N\\) similarity matrix. The diagonal entries correspond to the observed pairs, while off-diagonal entries are treated as alternatives that should generally receive lower scores. The training problem is therefore not to reconstruct an image or generate its caption, but to identify the correct pairing among many candidates.\n\nFor normalized image vectors \\(v_i\\) and text vectors \\(u_j\\), a common expression for the logits is\n\n\\[s_{ij}=\\frac{v_i^\\top u_j}{\\tau},\\]\n\nwhere \\(\\tau>0\\) is a temperature controlling the sharpness of the comparison. The image-to-text loss is\n\n\\[\\mathcal{L}_{I\\rightarrow T} = -\\frac{1}{N} \\sum_{i=1}^{N} \\log \\frac{\\exp(s_{ii})} {\\sum_{j=1}^{N}\\exp(s_{ij})}.\\]\n\nThe text-to-image loss reverses the direction of classification:\n\n\\[\\mathcal{L}_{T\\rightarrow I} = -\\frac{1}{N} \\sum_{i=1}^{N} \\log \\frac{\\exp(s_{ii})} {\\sum_{j=1}^{N}\\exp(s_{ji})}.\\]\n\nThe final objective averages the two directions. The original implementation learned a logit-scale parameter corresponding to the inverse temperature, rather than relying only on a fixed temperature.\n\nThis symmetric objective encourages each image to identify its associated text and each text to identify its associated image. Both encoders receive updates through backpropagation. Original CLIP training used the Adam optimizer with decoupled weight decay and a scheduled learning rate. The objective is often described as self-supervised because it does not require a manually designed category label for every example. More precisely, it uses naturally occurring image–text associations as supervision; those associations still contain human language and human choices about what to describe.\n\nLarge batches are important because they provide many competing examples. The original models used a batch size of 32,768 pairs. Such a batch offers tens of thousands of candidate matches for each image, making it harder to succeed through weak, generic signals alone. It also imposes substantial memory and communication demands. Distributed implementations gather representations across accelerators so that examples processed on different devices can participate in the contrastive comparison.\n\nThe assumption that every off-diagonal pair is negative is imperfect. Two different photographs may both depict a golden retriever, and either caption might plausibly describe both images. These false negatives show that the loss is an operational learning rule, not a complete model of semantic truth. Conversely, a supplied positive pair may be loosely related or misleading. Later contrastive methods have explored multiple positives, alternative negative handling, and different losses, but such changes define variants rather than requirements of the original CLIP formulation."
    },
    {
      "heading": "Training corpus, preprocessing, and scale",
      "text": "The original CLIP models were trained on a web-derived dataset called WebImageText, containing 400 million image–text pairs. The collection process used a large set of text queries to obtain a broad range of visual concepts, with limits intended to prevent individual queries from dominating the dataset. This was a major departure from training solely on a comparatively narrow set of manually curated object classes. The relevant supervision could include names, descriptions, stylistic terms, and contextual information that a conventional category label would omit.\n\nWeb text is nevertheless not equivalent to a carefully verified image caption. It may describe a surrounding article, convey a joke, mention an entity not visible in the picture, or contain promotional language. The model therefore learns from a mixture of visually grounded information and contextual associations. Large-scale aggregation can make useful patterns emerge despite noisy individual examples, but it does not guarantee that an association is visually justified. A representation may encode correlations between presentation style and subject matter alongside object appearance.\n\nThe original WebImageText corpus was not released as a downloadable training dataset. The publication of code and model weights consequently enabled inference and downstream experiments without enabling exact reproduction of the full data pipeline. Later projects, including OpenCLIP, supplied open implementations and trained related models on other datasets, notably LAION collections. These efforts expanded reproducibility and model availability, but a model trained on a different corpus is not an exact replication of the original system, even when its architecture has the same name.\n\nPreprocessing is also part of the model’s effective specification. Original CLIP training used image resizing and random cropping, while evaluation generally used a deterministic resize and center crop. Pixel values were normalized using specified channel statistics. Text was tokenized and constrained to the encoder’s sequence length. Changing the crop can remove an object of interest; changing resolution can alter the visibility of small details. Using a model with incompatible preprocessing can therefore reduce performance independently of its learned weights.\n\nThe original experiments trained for 32 epochs and compared multiple computational scales. Their findings connected greater training compute with improved transfer, while also showing differences between residual and transformer-based visual encoders. Scaling is not just a matter of parameter count: image resolution, dataset composition, number of processed pairs, batch size, and optimization all contribute. The associated demands on graphics processing units, storage throughput, and inter-device communication make large CLIP training a distributed-computing problem as well as a statistical one."
    },
    {
      "heading": "Zero-shot classification and adaptation",
      "text": "To use CLIP for image classification, a task designer supplies candidate class descriptions. If class \\(k\\) is represented by text \\(t_k\\), its normalized embedding \\(u_k\\) acts as a classifier weight. For an image embedding \\(v\\), the prediction is\n\n\\[\\hat{k}=\\operatorname*{arg\\,max}_{k} v^\\top u_k.\\]\n\nNo gradient updates are required to introduce this category vocabulary. A classifier can thus be assembled from descriptions of animals, actions, scenes, or other concepts supported by the learned representation. In this setting, “zero-shot” means that labeled training examples from the target classification task are not used to fit the classifier; it does not establish that the model has never encountered the concept during pretraining.\n\nWording has a substantial effect. A bare category name may be ambiguous or unlike the text associated with images during training. The original work found that templates such as “a photo of a {label}” improved performance relative to using labels alone. Dataset-specific wording can also help distinguish, for example, a picture of an object from a satellite image or a texture sample. This is an early prominent example of prompt engineering for a non-generative model: the prompt determines a representation rather than eliciting a written response.\n\nPrompt ensembling reduces dependence on one formulation. Several descriptions of a class are encoded, their representations are combined, and the resulting class vector is used for comparison. Templates can vary viewpoint, photographic framing, or other descriptions of the same class. This can improve robustness, but prompt selection is itself an evaluation choice. Extensive tuning against a target test set compromises claims that the reported performance measures an untouched transfer setting.\n\nSimilarity logits can be passed through a softmax to produce scores across candidate categories:\n\n\\[p(k\\mid x,\\mathcal{C}) = \\frac{\\exp(v^\\top u_k/\\tau)} {\\sum_{j\\in\\mathcal{C}}\\exp(v^\\top u_j/\\tau)}.\\]\n\nThese scores are conditional on the candidate set \\(\\mathcal{C}\\). Adding or removing labels changes the denominator and can change apparent confidence. The outputs should therefore not be interpreted automatically as calibrated probabilities that a statement is true, nor as evidence that the image belongs to one of the supplied classes.\n\nCLIP also supports supervised adaptation. A linear probe trains a classifier on frozen image features, testing how accessible task information is without modifying the encoder. Full fine-tuning updates some or all pretrained parameters. Other methods learn prompt vectors or small adaptation modules while retaining most weights. These approaches can improve target-task accuracy, but they differ from zero-shot evaluation and may alter robustness or transfer behavior. Their results depend on the amount and distribution of downstream data, not only on the quality of the pretrained model."
    },
    {
      "heading": "Evaluation, transfer, and robustness",
      "text": "The original CLIP study evaluated a wide range of tasks rather than treating ImageNet accuracy as a sufficient measure of visual competence. Its strongest reported zero-shot ImageNet model achieved approximately 76.2% top-1 accuracy, comparable to the widely used supervised ResNet-50 reference result. This comparison was significant because the CLIP classifier was constructed from textual category descriptions instead of being trained on ImageNet’s labeled training split. It did not imply that CLIP used less total data or computation: its pretraining corpus was much larger.\n\nEvaluation distinguished zero-shot transfer from linear-probe performance. Zero-shot results jointly measure the visual representation, text representation, and compatibility of the chosen descriptions with the task. Linear probing more directly assesses whether target labels can be predicted from frozen image features using labeled examples. A model can therefore perform well under one protocol and less well under the other. Comparisons that mix these settings conflate different amounts of task-specific supervision.\n\nA major finding concerned distribution shift. Standard image classifiers can lose accuracy when the same categories appear as sketches, artistic renditions, unusual viewpoints, or naturally difficult photographs. CLIP showed improved effective robustness on several such benchmarks: relative to its standard ImageNet accuracy, its shifted-distribution performance often declined less than that of conventional supervised models. Broad training data and natural-language supervision were associated with this behavior, but the result was not a universal guarantee across every kind of shift.\n\nPerformance remained uneven across task types. The original analysis identified difficulties with some fine-grained recognition tasks, counting, abstract classification, and specialized imagery. Distinguishing two visually similar vehicle models may require details that are weakly represented in web captions. Counting requires sensitivity to cardinality rather than merely recognizing the presence of a category. Medical or remote-sensing images can depart substantially from the photographic and linguistic conventions represented in general web data.\n\nLarge web datasets also complicate claims of strict separation between pretraining and evaluation. The original study investigated overlaps between its training data and evaluation datasets, but detecting duplicate images does not resolve every form of data leakage. Near-duplicates, related image collections, and exposure to benchmark descriptions can create subtler dependencies. Accordingly, a zero-shot protocol describes how the downstream classifier is constructed; it is not by itself proof that the pretraining corpus contains no benchmark-related information. Strong evaluation distinguishes exact overlap, broader distributional familiarity, and genuine transfer to new conditions."
    },
    {
      "heading": "Retrieval, generative systems, and multimodal applications",
      "text": "CLIP’s dual-encoder design is particularly well suited to information retrieval. In text-to-image search, a collection of images is encoded in advance, and a text query is encoded when a search is performed. Images are ranked by similarity to the query vector. The same mechanism supports image-to-image search and, when text records are indexed, image-to-text retrieval. Queries can describe concepts rather than exact filenames or manually assigned tags, allowing searches such as “a red boat near a snowy mountain” without requiring that phrase to exist in image metadata.\n\nAt large scale, retrieval systems typically combine CLIP embeddings with approximate nearest-neighbor indexing. The expensive visual encoding step can be separated from online query handling, while vector compression and indexing reduce storage and search costs. A production system may add filters, metadata constraints, or a more computationally expensive reranker. These components matter because a high CLIP score can reflect broad semantic resemblance without satisfying every query condition, such as a precise count, a specific object relationship, or a required location.\n\nCLIP has also been used with generative image systems, although it is not itself an image generator. In CLIP-guided generation, similarity between a generated image and a target description supplies an optimization signal. Gradients can adjust image pixels, a generator’s latent variables, or a sampling process so that the output better matches the description according to CLIP. Early CLIP-guided image-synthesis experiments demonstrated that a recognition model could provide useful language-directed guidance without learning to render images itself.\n\nLater architectures used CLIP representations in more structured ways. DALL·E 2 learned a prior that mapped text information to an image-embedding representation, followed by a decoder that generated images conditioned on that representation. Some diffusion models, including early Stable Diffusion releases, used a pretrained CLIP text encoder to condition image generation. These are distinct roles: using CLIP as a similarity-based guidance model, predicting its image embeddings, and reusing its text encoder are not interchangeable descriptions of the same mechanism.\n\nCLIP-derived visual encoders also became components of systems combining images with large language models. A projection or connector translates visual features into representations usable by the language model, often followed by additional multimodal training. Such systems can generate responses because they include a generative language component, not because the original CLIP encoder generates language. Related applications extend CLIP features to object detection and semantic segmentation, but those tasks require additional machinery to localize regions or assign pixel-level labels beyond the original global image–text similarity score."
    },
    {
      "heading": "Limitations, bias, and subsequent development",
      "text": "A central limitation is compositional understanding. CLIP may recognize objects mentioned in a sentence while inadequately representing how they relate to one another. Descriptions such as “a dog chasing a person” and “a person chasing a dog” contain similar words but describe different events. Attributes can also be assigned to the wrong objects, and spatial terms may be weakly distinguished. These failures are consistent with a training objective that rewards broad image–description agreement without explicitly requiring detailed grounding of each phrase in a particular visual region.\n\nThe distinction between semantic association and visual evidence also appears in typographic attacks. Demonstrations showed that written words placed in an image could strongly influence CLIP’s classification, sometimes overriding the appearance of the depicted object. Recognizing visible text is not inherently an error: signs, packaging, and labels are often informative. The vulnerability arises when a model relies on that text despite a task requiring identification of the underlying object. This illustrates how a useful learned capability can become a shortcut under a different evaluation condition.\n\nBias is another consequence of learning from large web corpora. The original research included tests showing that classification outcomes for images of people could change materially with the candidate labels supplied. Associations involving demographic attributes, occupations, and evaluative categories can reflect both the training distribution and the wording of the task. English-language coverage was also much stronger than coverage of many other languages in the original model. A shared embedding space does not remove cultural or linguistic imbalance; it can reproduce those imbalances in retrieval rankings and classification scores.\n\nCLIP-based evaluation introduces a separate methodological problem. A CLIP similarity score can be useful for measuring image–text compatibility, but it is not an exhaustive measure of image quality or factual correctness. A generated picture may receive a high score while containing incorrect counts, malformed details, or mistaken relationships. Optimizing a generator directly against the same evaluator can further exploit that evaluator’s weaknesses. This is especially important when comparing systems whose training already uses CLIP-derived features or rewards.\n\nSubsequent research has pursued larger and more diverse datasets, multilingual text encoders, region-level supervision, caption augmentation, and objectives that improve attribute and relation sensitivity. OpenCLIP expanded the ecosystem of reproducible implementations and pretrained weights. Related approaches, such as SigLIP, replaced the original batchwise softmax formulation with a pairwise sigmoid-based loss, changing how positive and negative image–text pairs contribute to training. These developments preserve the broader idea of transferable vision–language representations while varying the mechanisms used to obtain them. Differences in data provenance, preprocessing, architecture, and training objective remain essential to interpreting any particular model’s measured behavior."
    }
  ],
  "links": [
    "multimodal-learning",
    "zero-shot-learning",
    "representation-learning",
    "computer-vision",
    "supervised-learning",
    "imagenet",
    "foundation-model",
    "artificial-neural-network",
    "cosine-similarity",
    "convolutional-neural-network",
    "vision-transformer",
    "transformer-architecture",
    "self-attention",
    "tokenization",
    "latent-space",
    "contrastive-learning",
    "loss-function",
    "backpropagation",
    "adam-optimizer",
    "graphics-processing-unit",
    "distributed-computing",
    "image-classification",
    "prompt-engineering",
    "test-set",
    "fine-tuning",
    "distribution-shift",
    "data-leakage",
    "information-retrieval",
    "generative-artificial-intelligence",
    "diffusion-model",
    "large-language-model",
    "object-detection",
    "semantic-segmentation"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}