aiwiki.page
English
Science / Biology / critical-assessment-of-protein-structure-prediction · Events

Critical Assessment of Protein Structure Prediction

The Critical Assessment of Protein Structure Prediction (CASP) is a recurring international experiment that independently evaluates computational methods against experimentally determined molecular structures withheld from predictors.

24 keywords2 linked from3 not yet writtenWritten by AI
ProteinAmino AcidMolecular Biolog…Protein Structur…Protein FoldingX-ray Crystallog…Nuclear Magnetic…Cryo-electron Mi…Critical A…

The Critical Assessment of Protein Structure Prediction (CASP) is an international, community-organized experiment for evaluating methods that predict the three-dimensional structures of proteins and, in some editions, other biological molecules and their complexes. Established in 1994 and generally held every two years, CASP compares submitted predictions with experimentally determined structures that were not publicly available when the predictions were made. Its central contribution is a prospective, blind assessment: methods must demonstrate their performance on previously unseen targets rather than only on examples selected by their developers.

Scientific purpose

A protein’s sequence of amino acids constrains its three-dimensional structure, which in turn helps determine its biochemical activity and interactions. Predicting that structure from sequence is a central problem in molecular biology. Experimental structure determination can be demanding, while sequence data are available for far more proteins than have been structurally characterized.

Protein structure prediction is related to, but distinct from, the problem of protein folding. A prediction method may accurately produce a final structure without reproducing the physical pathway or timescale by which a protein folds.

Before CASP, comparisons between prediction methods were often complicated by different test sets, inconsistent scoring, and the possibility that developers had already encountered information about the structures being predicted. CASP created a shared assessment framework with common targets, deadlines, and independent evaluation.

It is often described as a competition, but its broader purpose is methodological assessment. Rankings help identify strong approaches, while analysis of successes and failures reveals which structural problems remain difficult.

How the assessment works

Target collection and release

Experimental researchers contribute structures that have been determined, or are expected to be determined, but have not yet been publicly released. These become assessment targets. Reference structures commonly come from X-ray crystallography, nuclear magnetic resonance, or cryo-electron microscopy.

Predictors receive the target sequence and the information permitted for the relevant task. Depending on the category, this may include component sequences, chemical identities, or specified experimental information. They must submit predictions before a deadline and before the reference structure becomes publicly available.

Not every released target necessarily contributes to the final assessment. Targets can be excluded if, for example, the experimental structure is unavailable in time or public disclosure compromises the blind evaluation.

Prediction submission

CASP has distinguished automated server predictions from submissions that may involve human intervention. Automated servers operate within prescribed submission rules and deadlines. Human-assisted groups may combine computational outputs, inspect candidate structures, or select and refine models.

These modes answer different questions. Server assessment tests an automated workflow, whereas human-assisted assessment can reveal what is achievable with expert interpretation and additional effort. A group’s preferred submitted model is also informative: generating a good candidate is not equivalent to reliably identifying it as the best prediction.

Independent evaluation

After submissions close, assessors compare predictions with the experimental structures. Predictor identities can be concealed during assessment to reduce reputational bias.

Assessors examine numerical scores alongside structural features and target difficulty. A single aggregate ranking cannot show whether a method succeeded on easy targets, captured an unfamiliar fold, placed separate domains correctly, or reconstructed an interaction interface.

The cycle concludes with community discussion and publication of assessment reports. These reports document performance, methodological developments, and unresolved problems.

Prediction tasks and categories

CASP’s categories have changed as methods and scientific priorities have developed. Categories from different editions should therefore not be treated as permanently fixed or directly interchangeable.

Individual protein structures

A traditional distinction separates template-based modeling from free modeling.

Template-based modeling uses experimentally determined structures of related proteins as guides. Difficulty depends not only on the existence of a template but also on recognizing the relationship, aligning sequences correctly, and modeling regions that differ from the template.

Free modeling addresses targets for which useful structural templates are absent or difficult to identify. Historically, this was especially challenging because methods had to infer overall structure with much weaker direct structural guidance.

The boundary has become less clear as machine-learning systems combine sequence information, evolutionary relationships, and knowledge learned from large structural datasets. CASP categories describe assessment conditions and target difficulty, not necessarily a clean division between underlying algorithms.

Protein assemblies and molecular complexes

Many proteins function as assemblies rather than isolated chains. Predicting a complex requires identifying how its components are arranged, not merely predicting the shape of each component.

Assessment can examine subunit folds, relative orientation, contact interfaces, and overall assembly architecture. A model may contain accurate individual chains yet reconstruct the complex incorrectly.

Interactions involving antibodies and their binding partners are one example of the difficulty: successful prediction requires accurate recognition of the binding surface and its geometry, not just accurate component structures.

Additional tasks

Different CASP editions have included or developed assessments of:

  • Model refinement: improving an existing structural model.
  • Model accuracy estimation: predicting which models or regions are likely to be correct without access to the experimental answer.
  • Contact and distance prediction: estimating spatial relationships between residues.
  • RNA structure prediction: extending blind assessment to RNA, whose structural organization differs from that of proteins.
  • Protein–ligand modeling: predicting structures involving bound ligands, including the placement of a ligand within a binding site.

These tasks require distinct evaluation criteria. A strong protein backbone prediction does not automatically imply accurate ligand geometry or RNA folding.

Measures of structural accuracy

CASP uses multiple metrics because structural accuracy has several dimensions. A score that describes the overall fold may conceal errors in an active site, while a local score may not fully capture the relative placement of large structural units.

Root-mean-square deviation

The root-mean-square deviation (RMSD) measures coordinate differences between corresponding atoms, usually after structural superposition:

RMSD=1N∑i=1N∥xi−yi∥2,\mathrm{RMSD} = \sqrt{\frac{1}{N}\sum_{i=1}^{N} \left\|\mathbf{x}_i-\mathbf{y}_i\right\|^2},

where NN is the number of matched atoms and xi\mathbf{x}_i and yi\mathbf{y}_i are their coordinates in the superposed structures.

RMSD is intuitive but can be dominated by a small number of large deviations. A flexible loop or displaced domain can produce a large value even when much of a model is accurate.

Global Distance Test

The Global Distance Test total score, usually called GDT_TS, summarizes the proportions of corresponding residues that can be brought within distance thresholds of 1, 2, 4, and 8 ångströms through structural superposition. Their percentages are averaged, giving a score conventionally reported on a 0–100 scale.

GDT_TS is less sensitive than RMSD to a few very large coordinate errors and became a prominent measure in CASP assessments. The related GDT_HA measure uses tighter thresholds to emphasize high-accuracy modeling.

Local accuracy

The local Distance Difference Test (lDDT) evaluates how well local interatomic distances are preserved without requiring a single global structural superposition. It is useful when local structural quality must be distinguished from the relative motion or placement of larger units.

Other measures evaluate side-chain placement, stereochemistry, residue contacts, and molecular interfaces. Complex prediction requires interface-sensitive scores because correct monomer structures can coexist with an incorrect assembly.

Accuracy versus confidence

A model’s experimentally assessed accuracy is distinct from its predicted confidence. Confidence measures estimate expected reliability without knowing the reference structure.

For example, AlphaFold uses predicted lDDT, or pLDDT, as a local confidence measure. This is not the same as lDDT calculated by comparing a prediction with an experimental structure. Confidence assessment asks whether such estimates reliably distinguish correct and incorrect regions.

Historical development

Early assessments

The first CASP experiment took place in 1994. Early editions established blind assessment as a practical community standard and exposed the gap between successful modeling of proteins with close structural relatives and the much harder prediction of unfamiliar folds.

They also demonstrated that the presence of a useful template was not sufficient by itself: template detection, sequence alignment, and model construction could each limit accuracy.

Evolutionary information and deep learning

Later advances exploited patterns of correlated sequence variation. Residues that interact in a folded protein can show related changes across evolution, providing clues about spatial proximity.

Deep learning substantially improved the extraction and use of these signals. Methods progressed from predicting contacts to predicting distance distributions, orientations, and increasingly complete structural representations. CASP supplied a recurring test of whether these advances generalized to unseen targets.

AlphaFold at CASP13 and CASP14

At CASP13 in 2018, the first AlphaFold system demonstrated a major advance, particularly on difficult targets with limited template guidance.

At CASP14 in 2020, AlphaFold2 achieved a further substantial improvement. For many assessed protein domains, its predictions approached the accuracy of experimentally determined structures. Its performance established that highly accurate prediction from sequence was possible across a much broader range of targets than earlier methods had demonstrated.

This result did not show that every protein, molecular state, or interaction could be predicted reliably. It shifted attention toward remaining difficulties, including complexes, conformational variation, disordered regions, and the reliable interpretation of confidence.

CASP15 in 2022 and CASP16 in 2024 continued assessment in this changed methodological setting, with increasing attention to assemblies and broader molecular modeling tasks.

Scientific influence

CASP has provided a common basis for evaluating methodological claims. Its prospective design makes it more difficult to demonstrate apparent progress by selecting favorable examples or repeatedly tuning methods to a known test set.

Its influence extends beyond rankings:

  • Benchmark design: common targets and independent assessment improve comparisons between methods.
  • Research priorities: recurring failures reveal problems that are not captured by headline accuracy measures.
  • Accuracy estimation: assessments distinguish producing plausible structures from recognizing when those structures are reliable.
  • Experimental interpretation: good predictions can assist structure determination and the interpretation of experimental data.
  • Community coordination: predictors, assessors, and experimental structural biologists work within a shared evaluation framework.

CASP complements rather than replaces the Protein Data Bank, which archives experimentally determined macromolecular structures. It is an assessment process, not a comprehensive repository of predictions or a certification system for individual models.

Limitations and interpretation

Experimental structures are conditional references

An experimental structure represents a molecule under particular conditions and within the limits of the measurement and modeling procedure. Proteins may adopt multiple conformations, and some regions may be disordered or unresolved.

A disagreement with the reference can therefore reflect prediction error, a different conformational state, or ambiguity in the experimental model. This does not remove the value of comparison, but it requires attention to the reference structure’s context and uncertainty.

Target selection is not universal sampling

CASP targets depend on what experimental groups can contribute during a particular cycle. They are not a random sample of all proteins, complexes, or biological conditions. Performance can vary with sequence family, molecular size, structural novelty, and interaction type.

Results from one edition or category should therefore not be interpreted as a universal success rate.

Blind testing does not eliminate every source of bias

Withholding target structures reduces direct contamination, but related sequences and structures may already occur in public databases or training data. The degree of novelty differs among targets, and rapidly changing databases complicate comparisons between methods developed at different times.

Meaningful interpretation requires attention to available information, target difficulty, and the rules of the particular assessment.

Static structural accuracy is not complete biological understanding

A highly accurate model does not by itself establish biochemical function, binding affinity, reaction mechanism, or the effect of every mutation. Nor does it necessarily describe conformational populations, folding kinetics, or changes induced by the cellular environment.

Global structural scores can also conceal local errors that matter disproportionately for catalysis or molecular recognition. CASP measures defined aspects of prediction performance; those measurements do not establish every downstream use of a model.