{
  "id": "roberta",
  "title": "RoBERTa",
  "lang": "en",
  "source_lang": "en",
  "summary": "RoBERTa is a Transformer-based language model that improves BERT through changes to pretraining data, objectives, and optimization rather than a fundamentally new architecture.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "product",
  "aliases": [
    "Robustly Optimized BERT Pretraining Approach"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/roberta/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/roberta/",
      "json": "https://aiwiki.page/zh/roberta.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "RoBERTa is a family of pretrained language models introduced in 2019 in the paper RoBERTa: A Robustly Optimized BERT Pretraining Approach by researchers at Facebook AI and collaborators. Built on BERT, it uses a bidirectional Transformer encoder to learn contextual representations of text. Its central contribution was to show that substantially improving BERT’s training procedure—using more data, larger batches, dynamic masking, and revised input construction—could produce strong gains without fundamentally changing the model architecture. RoBERTa became a widely used base model for natural language processing, particularly text classification, question answering, and other language-understanding tasks."
    },
    {
      "heading": "Background and development",
      "text": "BERT established a successful approach to language-model pretraining: first learn general-purpose representations from unlabeled text, then adapt the model to particular tasks through fine-tuning. Its results encouraged the development of new architectures and pretraining objectives.\n\nThe RoBERTa study examined whether some apparent improvements over BERT instead reflected differences in training resources and implementation choices. It investigated masking, the next-sentence prediction objective, batch size, training duration, and the amount and composition of training data.\n\nThe study concluded that the original BERT models were substantially undertrained relative to what their architecture could achieve. With a revised training recipe, RoBERTa matched or exceeded contemporary systems on prominent benchmarks, including GLUE, SQuAD, and RACE. These results supported a methodological point: comparisons between pretrained models must account for data, computation, and optimization, not merely architectural differences."
    },
    {
      "heading": "Architecture and model sizes",
      "text": "RoBERTa retains BERT’s encoder-only structure. Each layer combines multi-head self-attention with a position-wise feed-forward network, residual connections, and layer normalization. Bidirectional attention allows a token’s representation to depend on both preceding and following tokens within the input.\n\nThe original widely distributed configurations are:\n\nConfiguration | Encoder layers | Hidden dimension | Attention heads | Approximate parameters\nRoBERTa-base | 12 | 768 | 12 | 125 million\nRoBERTa-large | 24 | 1,024 | 16 | 355 million\n\nThese layer configurations correspond to BERT-base and BERT-large, although RoBERTa has a larger vocabulary and therefore a larger embedding table. The standard models support inputs of up to 512 tokens, including special tokens.\n\nRoBERTa uses learned positional embeddings to represent token order. Unlike BERT’s original two-segment setup, it does not use distinct learned segment embeddings to distinguish the two parts of a sentence pair. Special separator tokens still mark boundaries between input segments.\n\nTokenization\n\nRoBERTa uses byte-level byte-pair encoding rather than BERT’s WordPiece tokenizer. Its vocabulary contains approximately 50,000 entries and represents text through a combination of frequent multi-byte sequences and more elementary byte-level units.\n\nThis tokenization scheme can represent arbitrary input without requiring a conventional unknown-word token for unfamiliar words. It does not, however, guarantee efficient segmentation or strong understanding of every language or writing system. Rare words, specialized terminology, and languages poorly represented in training may require many tokens.\n\nBecause whitespace affects byte-level tokenization, the same visible word can have different token representations depending on whether it occurs at the beginning of a string or after a space. RoBERTa token identifiers and special-token conventions are not interchangeable with BERT’s."
    },
    {
      "heading": "Pretraining objective",
      "text": "RoBERTa is trained primarily through masked language modeling, a form of self-supervised learning. Selected token positions are corrupted, and the model learns to predict the original tokens from the surrounding text.\n\nFor a token sequence \\(x\\), selected positions \\(M\\), and corrupted sequence \\(\\tilde{x}\\), the objective can be written as\n\n\\[\\mathcal{L}_{\\mathrm{MLM}} = -\\mathbb{E}_{x,M} \\left[ \\sum_{i \\in M} \\log p_{\\theta}(x_i \\mid \\tilde{x}) \\right].\\]\n\nHere, \\(\\theta\\) denotes the model parameters. The loss is calculated at the selected prediction positions, rather than at every input position.\n\nFollowing BERT’s basic corruption recipe, approximately 15% of tokens are selected. Of those selected tokens, 80% are replaced with a mask token, 10% with a randomly chosen token, and 10% remain unchanged. Consequently, not every prediction target is visibly masked.\n\nDynamic masking\n\nRoBERTa uses dynamic masking: masking patterns can change when a training sequence is encountered again. This exposes the model to different prediction problems derived from the same text.\n\nThe distinction from BERT is not that BERT used only one mask pattern for each passage. BERT’s original preprocessing could generate multiple masked copies, but their masks were fixed in advance. RoBERTa instead generates masks dynamically during training.\n\nRemoval of next-sentence prediction\n\nOriginal BERT training included a binary next-sentence prediction objective, which asked whether one text segment followed another in the source material. RoBERTa removed this objective and trained using masked-token prediction alone.\n\nIts experiments found that removing next-sentence prediction, together with suitable construction of longer text sequences, could match or improve downstream performance. This result concerned the tested training setups; it did not establish that every objective involving sentence relationships is ineffective."
    },
    {
      "heading": "Data and optimization",
      "text": "The full RoBERTa training mixture contained roughly 160 GB of uncompressed English text, substantially more than the approximately 16 GB used for the original BERT experiments. Its sources included:\n\n- BookCorpus and English Wikipedia;\n- CC-News, a collection of English-language news articles;\n- OpenWebText, a web-text collection;\n- Stories, a corpus drawn from Common Crawl data.\n\nRoBERTa also used larger training batches and longer training than the original BERT setup. The strongest reported configuration used batches containing approximately 8,000 sequences and trained for up to 500,000 updates.\n\nThese changes affect different aspects of learning. More varied text broadens the evidence available to the model; additional updates provide more opportunities to learn from it; and larger batches alter the behavior and efficiency of optimization. Appropriate learning-rate schedules and other hyperparameters remain important: increasing batch size alone does not guarantee an improvement.\n\nThe study examined sequence construction as well, including packing complete sentences and using document boundaries. RoBERTa’s gains therefore reflect a combined training recipe, not a single architectural innovation."
    },
    {
      "heading": "Adaptation and applications",
      "text": "RoBERTa produces contextual token representations: a word’s output vector changes with its surrounding text. This makes it useful for transfer learning across several task types.\n\nSequence classification. A classification head can be attached to the representation of a special sequence-level token. Applications include sentiment analysis, topic classification, and natural-language inference.\n\nToken classification. Predictions can be attached to individual token representations for tasks such as named-entity recognition. When a word is split into subword tokens, labels and predictions must be aligned with that segmentation.\n\nExtractive question answering. A question and passage are encoded together, and output layers predict the start and end positions of an answer span in the passage. This differs from generating an answer freely.\n\nSentence-pair scoring. Jointly encoding two texts allows RoBERTa to assess relationships such as entailment or relevance. This supports cross-encoder reranking in information retrieval, although jointly processing every candidate pair can be computationally expensive.\n\nText representations. RoBERTa can supply features for downstream models. However, simply averaging its unmodified token vectors does not necessarily produce high-quality sentence-similarity embeddings; additional task-specific or contrastive training is often used.\n\nThe original implementation was released through fairseq, a PyTorch-based modeling toolkit. RoBERTa subsequently became available through other widely used model libraries."
    },
    {
      "heading": "Relationship to other language models",
      "text": "RoBERTa is an encoder model, not an autoregressive text generator. Its pretraining task predicts selected tokens using context from both directions. By contrast, models in the GPT family generally predict the next token using preceding context.\n\nThis difference shapes their standard uses. RoBERTa is well suited to producing representations, classifying inputs, and identifying spans. Its pretrained objective does not directly provide the left-to-right generation procedure normally used by conversational models. This does not mean encoder representations cannot contribute to a generative system, but additional architecture or training is required.\n\nRoBERTa also differs from encoder–decoder models, which combine an input encoder with a decoder trained to generate output sequences.\n\nXLM-R, also called XLM-RoBERTa, is a related multilingual model trained on multilingual web text. It is not simply the original English RoBERTa model applied to additional languages; it has its own training corpus, vocabulary, and model configurations."
    },
    {
      "heading": "Limitations and evaluation",
      "text": "RoBERTa inherits limitations associated with large pretrained text encoders:\n\n- Finite input length. Standard checkpoints cannot directly encode arbitrarily long documents. Splitting a document into chunks may lose relationships between distant passages. Dense self-attention also has quadratic attention cost in sequence length.\n- Dependence on training distribution. Performance can deteriorate under distribution shift, including changes in domain, genre, terminology, or language.\n- Bias in learned representations. Patterns in web, news, and book data can be reproduced in model outputs. Strong aggregate benchmark scores do not establish equal performance across demographic groups or linguistic varieties.\n- Lack of factual guarantees. Predicting plausible masked tokens is not the same as verifying statements against evidence. RoBERTa is not inherently a current or authoritative knowledge source.\n- Evaluation sensitivity. Fine-tuning results depend on random seeds, preprocessing, training schedules, and model selection. Data leakage and overlap between training material and evaluation examples can also complicate interpretation.\n\nRoBERTa’s historical importance lies partly in how it changed the interpretation of progress in language modeling: stronger benchmark results could arise from better-controlled training and greater resources even when the underlying encoder architecture remained largely unchanged."
    }
  ],
  "links": [
    "language-model",
    "bert",
    "transformer-architecture",
    "natural-language-processing",
    "fine-tuning",
    "training-data",
    "multi-head-attention",
    "residual-connection",
    "layer-normalization",
    "positional-encoding",
    "byte-pair-encoding",
    "tokenization",
    "self-supervised-learning",
    "loss-function",
    "learning-rate",
    "hyperparameter",
    "transfer-learning",
    "information-retrieval",
    "contrastive-learning",
    "pytorch",
    "generative-pre-trained-transformer",
    "encoder-decoder-architecture",
    "distribution-shift",
    "data-leakage"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}