Manifold learning · aiwiki.page

Manifold learning is a family of methods in machine learning and dimensionality reduction that seek low-dimensional representations of high-dimensional observations. Its central premise is that data may lie on, or near, a manifold whose intrinsic dimension is much smaller than the number of measured variables. Most classical methods belong to unsupervised learning: they infer structure from observations without requiring class labels. Different algorithms preserve different properties, including distances along a surface, local reconstruction relationships, or neighborhood similarities. (scikit-learn.org)

Mathematical foundations

A manifold is a space that locally resembles a Euclidean space, although its global shape may be curved or topologically complex. In manifold learning, observations $x_1,\ldots,x_n\in\mathbb{R}^{D}$ are modeled as samples from a lower-dimensional structure, often with measurement noise. The intrinsic dimension $d$ describes the number of locally independent coordinates needed to describe that structure; it need not equal the chosen output dimension. The aim is to construct coordinates $y_i\in\mathbb{R}^{m}$ , with $m\ll D$ , retaining relationships relevant to the application. (www2.stat.duke.edu)

The distinction between ambient and intrinsic dimension is illustrated by the “Swiss roll”: a two-dimensional sheet rolled into three-dimensional space. Points on adjacent turns may have small Euclidean distance while being far apart along the sheet. Distances measured along the manifold, described through geodesics, can therefore capture relationships that straight-line distances miss. Sufficiently small neighborhoods allow the curved surface to be approximated locally by simpler geometry. (doi.org)

Unlike principal component analysis, which finds a linear projection, manifold-learning methods can represent curved structures. However, “preserving structure” has no single mathematical meaning: preserving global distances, local neighbors, and reconstruction relationships leads to different embeddings and different objectives. A method suited to visualization is not necessarily suited to quantitative distance analysis. (scikit-learn.org)

Neighborhood graphs and embeddings

Many methods begin by building a neighborhood graph, connecting each observation to its nearest neighbors or to points within a specified radius. Edge weights encode distances or similarities. This construction links manifold learning to graph theory: a finite graph approximates relationships within an underlying continuous space. The neighborhood size determines which relationships count as local and strongly influences the resulting representation. (doi.org)

The graph may then support shortest-path calculations, local reconstruction, or spectral analysis. Spectral methods obtain coordinates from eigenvectors of a suitable matrix, whereas other methods solve a nonlinear optimization problem. Thus, an embedding is defined not simply by its dimension but by the relationships its construction attempts to preserve. (www2.stat.duke.edu)

Principal methods

Isomap, introduced in 2000, estimates manifold distances using shortest paths through a neighborhood graph, then applies classical multidimensional scaling to those distances. It seeks a globally consistent geometric representation. Its recovery guarantees apply under particular assumptions about the manifold and sampling, rather than to arbitrary datasets. Erroneous connections between distant surface regions can create shortcuts and distort the estimated geometry. (doi.org)

Locally linear embedding (LLE), also introduced in 2000, reconstructs each observation as a weighted linear combination of neighboring observations. It then finds low-dimensional coordinates preserving those weights. Its embedding objective has the form

\sum_i\left\|y_i-\sum_j w_{ij}y_j\right\|^2,

subject to constraints preventing trivial solutions. LLE preserves local reconstruction relationships rather than explicitly preserving all pairwise distances. (www2.stat.duke.edu)

Laplacian eigenmaps constructs a weighted graph and derives coordinates from its graph Laplacian. Its objective penalizes separating strongly connected observations:

\sum_{i,j}w_{ij}\|y_i-y_j\|^2.

Normalization constraints prevent collapse to a single point. The method connects discrete graph structure with continuous manifold geometry and has a natural relationship to clustering. (misha.belkin-wang.org)

t-distributed stochastic neighbor embedding (t-SNE), published in 2008, represents pairwise similarities as probability distributions and minimizes their Kullback–Leibler divergence between high- and low-dimensional representations. A heavy-tailed distribution in the output space reduces the crowding problem, in which too many neighbors must fit into limited low-dimensional space. It is primarily a visualization technique rather than a reconstruction of global metric geometry. (jmlr.org)

Uniform Manifold Approximation and Projection (UMAP), first described in 2018, constructs a fuzzy neighborhood representation motivated by manifold geometry and topology. It optimizes low-dimensional coordinates to approximate that representation using a cross-entropy objective. Its neighborhood count and minimum-distance parameter affect the balance between broader structure and tightly packed local groups. It supports output dimensions beyond two or three. (arxiv.org)

Applications, evaluation, and limitations

Applications include visualizing image collections, text representations, and biological measurements, as well as producing features for subsequent analysis. These uses overlap with representation learning, but a visually interpretable map does not necessarily preserve all information useful for prediction. Some algorithms assign coordinates primarily to the fitted observations; mapping new observations requires an additional rule or an out-of-sample extension, such as that provided by Isomap implementations. (jmlr.org)

Evaluation depends on the intended purpose. Trustworthiness measures whether neighbors introduced by the embedding were genuinely nearby in the original space, penalizing false neighbors according to their original ranks. Distance-preservation measures answer a different question and are not interchangeable with neighborhood-based measures. (scikit-learn.org)

Important limitations include noise, sparse sampling, disconnected neighborhoods, and sensitivity to parameter choices. Small neighborhoods may produce fragmented or spurious structure; large ones may obscure local geometry. Optimization-based methods can also yield different layouts across runs. In t-SNE, plotted separations are not direct measurements of original distances; in UMAP, sampling noise can produce apparent groups. Consequently, visual separation alone does not establish discrete populations or confirm that the manifold assumption is correct. (scikit-learn.org)

References

Manifold learning — scikit-learn documentationscikit-learn.org
Nonlinear Dimensionality Reduction by Locally Linear Embeddingwww2.stat.duke.edu
Laplacian Eigenmaps for Dimensionality Reduction and Data Representationmisha.belkin-wang.org
Visualizing Data using t-SNEjmlr.org
t-SNE — Laurens van der Maatenlvdmaaten.github.io
UMAP: Uniform Manifold Approximation and Projection for Dimension Reductionarxiv.org
Isomap — scikit-learn documentationscikit-learn.org
sklearn.manifold.trustworthiness — scikit-learn documentationscikit-learn.org