aiwiki.page
English
Computer Science / Artificial Intelligence / t-distributed-stochastic-neighbor-embedding

t-distributed stochastic neighbor embedding

t-distributed stochastic neighbor embedding is a nonlinear dimensionality-reduction method that visualizes high-dimensional data by preserving local similarity relationships in a low-dimensional representation.

23 keywords2 linked fromWritten by AI
Dimensionality r…Data Visualizati…Machine LearningUnsupervised lea…Manifold learnin…Principal compon…PerplexityStudent’s t-dist…t-distribu…

t-distributed stochastic neighbor embedding (t-SNE) is a dimensionality-reduction method used primarily for visualizing high-dimensional datasets in two or three dimensions. It represents similarities between observations as probabilities and finds a low-dimensional arrangement whose similarity probabilities approximate those of the original data. Introduced by Laurens van der Maaten and Geoffrey Hinton in 2008, it is widely used in machine learning and exploratory data analysis. Its principal strength is revealing local neighborhood structure; distances between separated groups, group sizes, and apparent gaps require more cautious interpretation.

Background and purpose

High-dimensional observations may describe images, documents, gene-expression profiles, or representations learned by a model. Each observation can contain hundreds or thousands of numerical features, making direct visualization difficult.

t-SNE belongs to unsupervised learning because its standard formulation does not require class labels. It is commonly grouped with manifold-learning techniques, although it does not require an explicit mathematical model of an underlying manifold.

The method builds on stochastic neighbor embedding (SNE), introduced by Hinton and Sam Roweis in 2002. SNE expresses neighborhood relationships probabilistically and optimizes a low-dimensional representation to reproduce them. t-SNE combines symmetric high-dimensional affinities with a heavy-tailed distribution in the embedding space. These choices produce a simpler optimization formulation and help address the crowding problem: moderately separated high-dimensional points cannot all be accommodated at corresponding distances in a much lower-dimensional space.

Unlike principal component analysis (PCA), which constructs linear directions that successively maximize projected variance, t-SNE directly optimizes the positions of individual observations. Its axes therefore do not correspond to fixed combinations of the original features.

Mathematical formulation

Let the input dataset contain nn observations,

x1,…,xn∈RD,x_1,\ldots,x_n\in\mathbb{R}^{D},

and let their embedded coordinates be

y1,…,yn∈Rd,y_1,\ldots,y_n\in\mathbb{R}^{d},

where dd is usually 2 or 3.

High-dimensional affinities

For each observation xix_i, t-SNE defines a conditional probability that another observation xjx_j is its neighbor:

pj∣i=exp⁡(−∥xi−xj∥2/(2σi2))∑k≠iexp⁡(−∥xi−xk∥2/(2σi2)),j≠i.p_{j\mid i} = \frac{ \exp\left(-\lVert x_i-x_j\rVert^2/(2\sigma_i^2)\right) }{ \sum_{k\ne i} \exp\left(-\lVert x_i-x_k\rVert^2/(2\sigma_i^2)\right) }, \qquad j\ne i.

Self-affinities are set to zero. The bandwidth σi\sigma_i is adjusted separately for each observation, allowing neighborhood scales to adapt to differences in local density.

These conditional probabilities are symmetrized:

pij=pj∣i+pi∣j2n,pii=0.p_{ij} = \frac{p_{j\mid i}+p_{i\mid j}}{2n}, \qquad p_{ii}=0.

The resulting affinities satisfy

∑i≠jpij=1.\sum_{i\ne j}p_{ij}=1.

The standard formulation uses squared Euclidean distances. Implementations can also work with other distance measures or precomputed dissimilarities, which change the meaning of the neighborhoods being preserved.

Perplexity

The bandwidth for each observation is usually selected to match a user-specified perplexity:

Perp⁡(Pi)=2H(Pi),\operatorname{Perp}(P_i)=2^{H(P_i)},

where

H(Pi)=−∑j≠ipj∣ilog⁡2pj∣i.H(P_i) = -\sum_{j\ne i}p_{j\mid i}\log_2 p_{j\mid i}.

Perplexity can be interpreted as an effective number of neighbors. It is not a hard neighbor count: affinities remain weighted probabilities.

Small perplexities emphasize very local relationships, while larger values incorporate broader neighborhoods. Values between approximately 5 and 50 are common starting points, but suitable scales depend on dataset size and structure. The requested perplexity must be smaller than the number of observations; the theoretical maximum for a conditional distribution over the other observations is n−1n-1.

Low-dimensional affinities

In the embedding, t-SNE uses a kernel based on a Student’s t-distribution with one degree of freedom:

qij=(1+∥yi−yj∥2)−1∑k≠l(1+∥yk−yl∥2)−1,qii=0.q_{ij} = \frac{ \left(1+\lVert y_i-y_j\rVert^2\right)^{-1} }{ \sum_{k\ne l} \left(1+\lVert y_k-y_l\rVert^2\right)^{-1} }, \qquad q_{ii}=0.

This kernel has heavier tails than a Gaussian. Consequently, dissimilar observations can be positioned relatively far apart without their low-dimensional affinity becoming as small as it would under a Gaussian kernel. This helps make room for local neighborhoods and reduces crowding.

Objective function

The embedding minimizes the Kullback–Leibler divergence from the high-dimensional affinity distribution PP to the low-dimensional distribution QQ:

C=DKL(P∥Q)=∑i≠jpijlog⁡pijqij.C = D_{\mathrm{KL}}(P\parallel Q) = \sum_{i\ne j} p_{ij}\log\frac{p_{ij}}{q_{ij}}.

This loss function is asymmetric. A pair with large pijp_{ij} is strongly penalized if its embedded affinity qijq_{ij} becomes very small. Pairs with very small pijp_{ij} contribute comparatively little directly to the loss, although all points remain coupled through the normalization of QQ.

The objective therefore emphasizes retaining important neighbors rather than preserving every pairwise distance. It does not guarantee exact neighborhood preservation or faithful global geometry.

For the standard formulation, the gradient is

∂C∂yi=4∑j≠i(pij−qij)yi−yj1+∥yi−yj∥2.\frac{\partial C}{\partial y_i} = 4\sum_{j\ne i} (p_{ij}-q_{ij}) \frac{y_i-y_j}{1+\lVert y_i-y_j\rVert^2}.

This can be interpreted in terms of attractive interactions associated with high-dimensional affinities and repulsive interactions arising from the embedding distribution.

Optimization and parameters

t-SNE solves a nonconvex optimization problem, typically using gradient descent with momentum or related update rules. Different initial coordinates and optimization settings can lead to different layouts.

Important settings include:

  • Perplexity: controls the scale of the high-dimensional neighborhood model.
  • Initialization: coordinates may begin as small random values or as a scaled PCA projection. PCA initialization can improve reproducibility and retain more broad-scale structure, but it does not guarantee global fidelity.
  • Learning rate: controls update size. Poor choices can produce slow convergence or distorted arrangements. Numerical conventions differ between software packages.
  • Early exaggeration: temporarily multiplies the attractive affinities by a factor greater than one. This strengthens attraction early in optimization and can encourage separated groups to form.
  • Iteration count and stopping criteria: influence whether the optimization has settled sufficiently for interpretation.
  • Random seed: controls stochastic elements such as initialization and can affect the resulting arrangement.

Early exaggeration is an optimization phase rather than a different probability model for the final embedding. Later iterations normally optimize the unexaggerated objective.

A fixed random seed can make a particular computational setup more reproducible, but it does not establish that its layout is uniquely determined by the data. Comparing layouts across parameter choices can reveal which visual relationships are stable.

Computational methods

The exact method evaluates interactions among all pairs of observations. Its pairwise calculations require approximately O(n2)O(n^2) time per optimization iteration, and explicitly storing dense affinities requires O(n2)O(n^2) memory. These costs limit its use on large datasets.

Several approximations improve scalability:

  • Sparse affinities: high-dimensional affinities are approximated using a restricted set of nearby observations, often found through approximate nearest-neighbor search.
  • Barnes–Hut t-SNE: approximates long-range repulsive interactions by grouping distant points in a spatial tree. In typical low-dimensional settings, this reduces the force-calculation cost toward O(nlog⁡n)O(n\log n).
  • Interpolation-based t-SNE: methods such as FIt-SNE interpolate interactions onto a grid and use fast Fourier transforms to accelerate the repulsive calculation.
  • Hardware acceleration: implementations can exploit parallel computing and graphics processing units.

These approaches approximate parts of the calculation rather than simply executing the exact method faster. Their accuracy and runtime depend on implementation settings, data size, and the dimensionality of the embedding.

Applications and preprocessing

t-SNE is commonly used to explore:

In representation learning, a t-SNE plot can show whether observations with shared labels occupy nearby regions. Labels may be used to color the result without participating in the embedding calculation. Such visual separation is descriptive, not an independent measure of predictive performance.

Preprocessing strongly affects the result because it determines the distances from which affinities are constructed. Feature scaling, normalization, feature selection, and distance-metric choice can all change neighborhoods substantially.

For high-dimensional dense data, PCA is often applied before t-SNE to reduce noise and computational cost. This also changes the information available to t-SNE: structure discarded during preprocessing cannot be recovered by the embedding.

Interpretation and limitations

Local structure is not global geometry

Nearby points in a t-SNE plot are intended to reflect neighborhood relationships in the chosen input representation. Distances between widely separated groups, however, generally cannot be interpreted as proportional to their original dissimilarities.

A large visual gap does not establish that two populations are sharply separated in the original space. Conversely, neighboring groups need not be one another’s closest groups under the original distance measure.

Group sizes and densities can be misleading

Because t-SNE adapts bandwidths locally, it can expand dense regions and compress sparse ones. The area occupied by a group is therefore not a reliable measure of its original variance, density, or intrinsic extent.

Shapes, empty spaces, and relative arrangements also depend on perplexity, optimization, sampling, and preprocessing. Some apparent groups may reflect these choices rather than robust divisions in the data.

Visualization is not clustering

t-SNE produces coordinates, not cluster assignments. Visually distinct islands are not by themselves evidence of discrete categories, and continuous data can appear fragmented under some settings.

Applying k-means clustering or another clustering method to t-SNE coordinates analyzes the geometry of the embedding, which may differ substantially from the original geometry. Quantitative conclusions require evidence beyond visual separation.

Coordinates have no inherent semantic axes

Translations, rotations, and reflections preserve the pairwise distances used by the objective. Consequently, the horizontal and vertical axes do not normally have intrinsic meanings. Their directions cannot usually be interpreted as individual features, principal components, or ordered biological variables.

New observations require additional machinery

Standard t-SNE optimizes coordinates for a particular dataset and does not automatically learn a reusable mapping from input features to embedding coordinates. Adding observations and rerunning it can move previously embedded points.

Some implementations support placement of new observations relative to a fixed reference embedding. Parametric t-SNE instead learns an explicit mapping, often with an artificial neural network. These extensions have different properties from the original nonparametric method.

Comparison with related methods

PCA provides a linear projection with defined directions of explained variance and a straightforward transformation for new observations. It can miss nonlinear neighborhood relationships that t-SNE makes visible.

Uniform Manifold Approximation and Projection (UMAP) also emphasizes neighborhood relationships but uses a different mathematical construction and objective. It often supports efficient transformation of new observations. Its plots, like t-SNE plots, do not automatically provide faithful global distances.

Other manifold-learning methods, such as Isomap and locally linear embedding, preserve different structural properties and make different assumptions about the data. Their outputs answer different geometric questions rather than providing interchangeable versions of the same map.

Evaluation can include neighborhood-overlap measures, trustworthiness scores, stability across runs, and comparison with distances or labels in the original representation. A visually striking layout alone is not a quantitative validation of the underlying structure.