aiwiki.page
English

Recommender System

A recommender system selects and ranks items for users by modeling preferences, item characteristics, context, and the consequences of exposure.

39 keywords29 linked from6 not yet writtenWritten by AI
Machine LearningSocial MediaSearch EngineInformation retr…Deep LearningLarge Language M…Training dataMatrix Factoriza…Recommende…

A recommender system is a computational system that selects, ranks, or suggests items expected to be relevant to a person or a particular situation. Items may include products, films, songs, news articles, jobs, courses, services, or other people. Recommendations can be personalized to an individual, adapted to a current session, or directed at a group. Although modern systems frequently use machine learning, recommendation also includes rule-based methods, expert-curated suggestions, and nonpersonalized rankings such as lists of popular products.

Recommender systems help users navigate collections too large to inspect directly. They are central to electronic commerce, streaming media, social media, and many other digital services. Unlike a conventional search engine, which typically responds to an explicit query, a recommender often acts on inferred interests without a detailed request. The distinction is not absolute: search results can be personalized, and recommendations can incorporate queries or conversational instructions.

Recommendation is both a prediction problem and a decision problem. A system must estimate what might interest a user, but it must also decide which items to expose, in what order, and under what constraints. Its behavior depends on the feedback it observes, the objectives it optimizes, and the surrounding interface. This creates close connections with information retrieval, statistical inference, human–computer interaction, and the design of online markets.

Historical development and principal applications

Early recommender systems developed from efforts to filter information through the judgments of other people. Tapestry, described by researchers at Xerox PARC in 1992, allowed users to annotate documents and retrieve them using queries that incorporated other users’ reactions. It helped establish the term collaborative filtering, although its approach relied on users explicitly constructing filtering queries rather than on the fully automated preference prediction associated with many later systems.

During the 1990s, systems such as GroupLens, introduced in 1994 for Usenet news, automated the use of similarities among users’ ratings. Ringo explored personalized music recommendations, while other projects applied related methods to films and web content. These systems demonstrated that recommendations could be produced without a complete semantic description of every item: patterns of agreement and disagreement among users could themselves be informative. At the same time, content-based approaches drew on document filtering and information retrieval to match user profiles with descriptions of available material.

Commercial adoption expanded as online retailers and media services accumulated large interaction histories. An influential 2003 paper described Amazon’s item-to-item collaborative filtering, which emphasized relationships among products rather than searching for similar customers at request time. The Netflix Prize, announced in 2006, offered a US$1 million grand prize for improving the accuracy of predictions of users’ film ratings by 10 percent over Netflix’s reference system. The prize was awarded in 2009. The competition accelerated interest in latent-factor models and ensembles, but its rating-prediction objective represented only one part of the broader recommendation problem.

Applications differ substantially in what constitutes a useful suggestion. A music service may recommend familiar songs repeatedly, whereas a retailer generally should not keep recommending a durable product immediately after purchase. News recommendations are sensitive to freshness; job recommendations must account for qualifications and mutual interest; educational recommendations may need to respect prerequisites. In travel and accommodation, availability and date constraints can be as important as estimated preference. These differences affect both the choice of model and the appropriate evaluation criteria.

From the 2010s onward, large services increasingly adopted deep learning, learned representations, and multistage recommendation pipelines. The spread of mobile devices also made time, location, and short-session behavior more widely available as contextual signals. More recently, large language models have been used to interpret requests, represent item descriptions, and support conversational recommendation. These developments supplement rather than eliminate older approaches: popularity models, neighborhood methods, factor models, and business rules remain useful because of their efficiency, transparency, and suitability for particular tasks.

Recommendation tasks, feedback, and statistical assumptions

A basic formulation assigns a score to a user–item pair. Let uu denote a user, ii an item, and cc the context. A scoring function s(u,i,c)s(u,i,c) estimates some quantity relevant to selection, such as expected interest, purchase probability, or usefulness. The system then constructs a ranked list from the eligible item set. Scores need not be calibrated probabilities: some models are trained only to put preferred items above less-preferred alternatives.

Several distinct tasks fall under this formulation. Rating prediction estimates an explicit score, such as a one-to-five-star film rating. Top-KK recommendation selects a short list from a much larger catalog. Next-item recommendation predicts the next interaction in a sequence, while session-based recommendation relies primarily on the current visit rather than a persistent user profile. Basket completion, complementary-product recommendation, and group recommendation introduce additional dependencies. A compatible phone case, for example, is not merely an item similar to a phone.

Feedback is commonly divided into explicit and implicit forms. Explicit feedback includes ratings, likes, dislikes, and stated preferences. Implicit feedback includes impressions, clicks, purchases, viewing duration, skips, and repeat consumption. These signals have different meanings. A click can indicate curiosity rather than satisfaction, while a long viewing time may reflect either genuine interest or difficulty finding the desired information. Purchases are influenced by price, availability, and delivery conditions as well as preference.

The observed interaction table is typically extremely sparse: each user encounters only a small fraction of the catalog. More importantly, missing observations are not random. A user cannot click an item that was never displayed, and items near the top of a page are more likely to be noticed. Consequently, an unobserved interaction is not equivalent to a negative judgment. Treating every missing user–item pair as dislike can produce systematic errors, especially when prior recommendation policies strongly determine exposure.

Training therefore requires assumptions about what the available training data represent. For a click model, the relevant record may include the user, item, position, surrounding slate, timestamp, and whether a click occurred. Positive examples alone are usually insufficient; negative examples may be drawn from displayed-but-unclicked items or sampled from a larger pool. These sampling choices alter the learning problem. A model trained with uniformly sampled catalog negatives, for example, may perform well at distinguishing relevant items from obviously irrelevant ones while struggling to rank the closely competing candidates encountered in production.

Collaborative filtering and latent-factor models

Collaborative filtering infers preferences from patterns in collective behavior. Its central premise is that interactions reveal useful relationships even when item descriptions are incomplete. Two users who have evaluated many items similarly may agree on other items, while two items consumed by overlapping groups may be useful substitutes or companions. These are statistical regularities, not guarantees that the users have identical tastes or that the items share an obvious semantic category.

Neighborhood methods implement this idea directly. User-based methods predict a user’s preference from the ratings of similar users. Item-based methods compare a candidate with items the user previously liked or consumed. Similarity may be measured with cosine similarity, correlation, or adjusted co-occurrence statistics. Raw co-occurrence usually requires correction because very popular items overlap with almost everything. Minimum-support thresholds and shrinkage can reduce unstable similarities estimated from only a few observations.

A second major family uses matrix factorization. For a partially observed user–item rating matrix, a common model is

r^ui=μ+bu+bi+pu⊤qi,\hat r_{ui}=\mu+b_u+b_i+p_u^\top q_i,

where μ\mu is a global mean, bub_u and bib_i are user and item biases, and pu,qi∈Rdp_u,q_i\in\mathbb{R}^{d} are learned latent vectors. Their inner product captures interactions not explained by the average rating or the biases. The dimensions need not correspond to named attributes such as genre or price; they are chosen to explain patterns in the observations.

For explicit ratings, parameters can be learned by minimizing squared prediction error with regularization:

min⁡b,p,q∑(u,i)∈Ω(rui−r^ui)2+λ(∑ubu2+∑ibi2+∑u∥pu∥2+∑i∥qi∥2).\min_{b,p,q} \sum_{(u,i)\in\Omega} \left(r_{ui}-\hat r_{ui}\right)^2 +\lambda\left( \sum_u b_u^2+\sum_i b_i^2+ \sum_u\lVert p_u\rVert^2+\sum_i\lVert q_i\rVert^2 \right).

Here Ω\Omega contains observed ratings, and λ\lambda controls the penalty on large parameter values. This penalty is important because sparse observations can otherwise permit severe overfitting. Implicit-feedback variants often distinguish an inferred preference from a confidence weight: repeated plays may increase confidence in a positive signal without being interpreted as a proportionally larger rating.

Ranking-oriented methods optimize relative order rather than numerical rating accuracy. Bayesian personalized ranking, introduced in 2009, learns from triples consisting of a user, an observed item, and a sampled unobserved item. Its objective encourages the observed item to receive the higher score. Factor models can be trained efficiently and often provide strong baselines, but pure collaborative methods face a cold-start problem: a new user or item has little interaction history from which to infer a vector. They also inherit exposure patterns from historical data, so strong predictive performance does not establish that their recommendations represent preferences independently of earlier system decisions.

Content-based, knowledge-based, and hybrid recommendation

Content-based recommendation uses descriptions of items and information about a user’s interests. A document recommender might represent articles by their words or topics, while a product recommender might use category, brand, dimensions, and price. A user profile can then be formed from the characteristics of previously preferred items. Unlike purely collaborative methods, this approach can recommend a new item as soon as its content or attributes are available.

Traditional text representations include term-weighted vectors and the bag-of-words model. More recent systems use representation learning to place descriptions, images, audio, or other modalities in a shared or related feature space. A film can thus be represented using its synopsis, cast, poster, trailer, and interaction history. These signals are not interchangeable: a poster may capture visual style, while a synopsis describes narrative content. The appropriate representation depends on which distinctions matter to users and on the quality of the available metadata.

Content-based methods can provide interpretable reasons for recommendations, such as a shared author or a preferred technical specification. However, close content matching can also limit discovery. A user who liked one historical novel may not want a list consisting entirely of highly similar historical novels. Furthermore, item descriptions may omit experiential properties such as humor, pacing, build quality, or suitability for a particular occasion. Content similarity and user utility therefore need not coincide.

Knowledge-based recommendation applies explicit relationships and constraints. A configuration system may check whether a component fits a device; a travel recommender may enforce dates, budget, accessibility requirements, and departure locations. Constraint satisfaction and rule-based reasoning can exclude impossible or unsuitable options before preferences are ranked. Knowledge graphs can represent relationships among users, items, creators, categories, and other entities, supporting recommendations through typed connections rather than undifferentiated similarity alone.

Most practical systems are hybrids. A collaborative score may be combined with content features, or several independent recommenders may contribute candidates to a common ranking stage. Another design learns item representations from both interaction data and descriptions, allowing content features to support new items while collaborative signals improve established ones. Hybridization does not necessarily mean averaging final scores; it can occur in feature construction, model training, candidate generation, or the user interface.

The cold-start problem is also partly an interaction-design problem. A service can ask a new user to select interests, import an existing collection with permission, or begin with contextual and broadly popular suggestions. Each approach trades initial effort against uncertainty. For new items, controlled exposure can gather evidence that content models alone cannot supply. The resulting recommendation quality depends not only on the predictive model but also on how effectively the system obtains relevant information.

Neural, sequential, and conversational models

Neural networks allow recommender systems to learn nonlinear relationships among user features, item features, and context. A simple model may concatenate these features and predict a click or purchase probability. More elaborate models combine memorization of frequent feature interactions with learned representations that generalize to less common combinations. The value of additional complexity depends on data volume, feature quality, and operational constraints; a larger model is not automatically a better recommender.

A widely used architecture is the two-tower model. One network maps a user or request context to a vector, while another maps an item to a vector. Relevance is estimated using a dot product or another inexpensive similarity function. Because item vectors can be computed in advance, this architecture supports retrieval from very large catalogs. Its efficiency comes with a limitation: arbitrary, detailed interactions between a particular user and candidate cannot all be evaluated before retrieval. More expressive models are therefore often reserved for ranking the smaller candidate set.

Sequential models account for the order of interactions. A user’s recent search for a camera may be more informative for the next recommendation than a purchase made years earlier. Recurrent neural networks and later the transformer architecture have been applied to such sequences. Self-attention allows a model to weight different past events when representing the current state. Time gaps, repeated interactions, and changes in intent may also be encoded. However, an observed sequence can reflect interface layout and prior recommendations as well as a natural progression of interests.

Some sequence models learn by predicting a next item or reconstructing hidden items, connecting recommendation with self-supervised learning. Care is needed when translating a training objective into evaluation. A model allowed to use later events when reconstructing a missing interaction is solving a different problem from a live system that must act using only the past. Similarly, timestamps, item availability, and the moment at which user features were computed must be respected to avoid data leakage.

Conversational systems use natural language processing to refine preferences through dialogue. A request such as “a quiet film under two hours, but not a tragedy” combines constraints that may be difficult to express through clicks alone. Language models can extract those constraints, ask follow-up questions, or explain differences among candidates. They can also enrich sparse item descriptions and connect everyday expressions to catalog attributes.

Conversational fluency is separate from catalog correctness. A generated title may not exist, a product attribute may be wrong, or a suggested item may be unavailable. Systems can reduce these problems by retrieving candidates from a controlled catalog and checking claims against structured records. Where explanations are generated, their correspondence to the actual recommendation process remains a distinct issue: a plausible description of an item is not necessarily a faithful account of why the ranking model selected it.

Production architecture and large-scale serving

Large recommendation services rarely score every available item with their most complex model for every request. Instead, they use a multistage pipeline. A typical sequence consists of eligibility filtering, candidate generation, ranking, and final reranking or slate construction. Each stage reduces the search space while applying increasingly detailed information. The architecture must satisfy latency, throughput, reliability, and freshness requirements in addition to predictive accuracy.

Candidate generators may retrieve popular items, similar items, items from followed creators, or items close to a user vector. Approximate nearest-neighbor search makes vector retrieval practical when an exact comparison against the entire catalog would be too expensive. Different candidate sources often specialize in different behaviors, such as short-term intent, long-term preference, novelty, or repeat consumption. Their outputs are merged and deduplicated. If a relevant item never reaches this pool, later ranking stages cannot recover it.

Ranking models use richer features to estimate outcomes for each candidate. A retailer might consider purchase probability, price, stock status, delivery time, and return risk; a video service might consider expected viewing duration alongside explicit satisfaction signals. These predictions can feed a separate decision rule rather than being collapsed into a single training target. Final reranking may enforce creator limits, category diversity, safety restrictions, or compatibility constraints. Thus the displayed list is often not simply the top items from one model’s raw scores.

Data infrastructure is an equally important part of the system. Interaction events must be logged with reliable timestamps and identifiers, transformed into features, and made available for both training and online scoring. Distributed computing supports large-scale aggregation and training, while low-latency stores supply recent user state and item information. Batch processing efficiently computes broad historical summaries; streaming updates capture changes such as a newly watched episode or an item becoming unavailable. The two must agree on definitions to avoid training–serving inconsistencies.

Serving introduces additional failure modes. A feature may be missing, a model may become stale, or a retrieval index may lag behind catalog changes. Systems therefore commonly maintain fallback recommendations and monitor both model outputs and infrastructure health. Caches reduce cost but can return outdated or insufficiently personalized lists. A newer model with better offline accuracy may be unsuitable if its latency causes timeouts or if its memory requirements prevent reliable deployment.

Operational monitoring must also track distribution shift. Seasonal demand, new content formats, interface redesigns, and changes in the user population can invalidate learned relationships. Retraining addresses some shifts, but not all: a broken event logger or a changed definition of “view” can corrupt the measurement process itself. Model versioning, reproducible feature computation, and carefully controlled deployment are consequently integral to recommendation engineering rather than merely administrative support.

Offline evaluation and online experimentation

Evaluation begins by specifying what success means. Accurate star-rating prediction, useful top-ranked suggestions, catalog discovery, and long-term satisfaction are different objectives. A model can improve one while worsening another. Rating-prediction systems are often evaluated with mean absolute error or root mean squared error, whereas ranked recommendation requires measures that account for which items appear and, usually, their positions.

For a recommendation set RuKR_u^K and a set of relevant held-out items TuT_u, common measures are

Precision@K⁡(u)=∣RuK∩Tu∣K,Recall@K⁡(u)=∣RuK∩Tu∣∣Tu∣.\operatorname{Precision@K}(u)=\frac{|R_u^K\cap T_u|}{K}, \qquad \operatorname{Recall@K}(u)=\frac{|R_u^K\cap T_u|}{|T_u|}.

Other measures emphasize ordering. Normalized discounted cumulative gain, or NDCG, gives more credit to relevant items near the top and can incorporate graded relevance. Mean reciprocal rank emphasizes the position of the first relevant result. These measures encode different assumptions about user behavior; none is universally appropriate.

A credible offline experiment needs a carefully constructed test set. For temporal tasks, training on earlier interactions and evaluating on later ones better resembles deployment than an unrestricted random split. Even a chronological split can leak information if item representations, popularity counts, or preprocessing steps use future observations. Hyperparameters should be selected using a separate validation set, not repeatedly tuned against the final test results.

Candidate selection also affects reported performance. Evaluating one held-out item against a small sample of random negatives is easier than ranking it against the full eligible catalog. Rankings of algorithms can change with the number and distribution of sampled negatives. Comparisons must therefore specify the split, candidate pool, filtering rules, and treatment of users or items absent from training. Public datasets such as MovieLens support reproducible research, but results on one collection do not establish performance in all domains.

Online A/B testing estimates how a deployed change affects real behavior by randomly assigning users or other units to alternatives. Relevant outcomes may include satisfaction ratings, purchases, retention, hides, complaints, and technical performance. Click-through rate alone is often inadequate because an attractive headline or thumbnail can increase clicks without improving the subsequent experience. Experiment duration matters as well: immediate curiosity can differ from sustained value, and some consequences emerge only after repeated exposure.

Randomization does not remove every difficulty. Users can influence one another, sellers can react to traffic changes, and shared inventory can couple experimental groups. Metrics may also be noisy or affected by repeated testing. Offline and online evaluation therefore serve complementary purposes: offline tests support rapid, controlled comparison, while online experiments measure the consequences of actual exposure under a particular interface and population. Neither automatically identifies effects outside the conditions studied.

Exploration, causal inference, and long-term objectives

Recommendation changes the data from which future recommendations are learned. If a system repeatedly displays a small set of popular items, it collects more evidence about those items and little about alternatives. Subsequent training can then interpret the resulting interaction concentration as further evidence of superiority. This feedback loop makes recommendation different from a purely passive prediction task.

The exploration–exploitation trade-off concerns whether to select items already believed to be useful or gather information about uncertain alternatives. Multi-armed bandit methods formalize this choice for repeated decisions. A contextual bandit conditions its action on information about the current user or situation. Methods such as upper-confidence-bound strategies and Thompson sampling incorporate uncertainty into selection, rather than always choosing the item with the largest current point estimate.

Exploration is not synonymous with unrestricted randomness. A system can explore among eligible, sufficiently relevant candidates, or reserve limited opportunities for new items and underobserved creators. The appropriate design depends on the cost of a poor recommendation and on how rapidly the environment changes. A music service can often test an unfamiliar song with relatively limited consequences, whereas a high-stakes matching service faces different constraints. Exploration also requires meaningful uncertainty estimates; a model’s raw score does not by itself indicate confidence.

Historical logs can sometimes be used to estimate the effect of a new recommendation policy. Such off-policy evaluation connects recommendation with causal inference. If the logging policy selected action ata_t in context xtx_t with known probability π0(at∣xt)\pi_0(a_t\mid x_t), an inverse-propensity estimator for a target policy π\pi is

V^IPS(π)=1n∑t=1nπ(at∣xt)π0(at∣xt) rt,\hat V_{\mathrm{IPS}}(\pi)= \frac{1}{n}\sum_{t=1}^{n} \frac{\pi(a_t\mid x_t)} {\pi_0(a_t\mid x_t)}\,r_t,

where rtr_t is the observed reward. This reweights logged outcomes according to how frequently the target policy would have chosen the same actions.

The estimator requires support: actions favored by the target policy must have had a nonzero chance of appearing under the logging policy. Very small logging probabilities can create high variance, and unrecorded factors can undermine causal interpretation. Evaluation of entire ranked slates is harder because the probability of a particular list may be extremely small and items can influence one another’s outcomes. Reliable propensity logging and appropriately designed randomization are therefore important research and engineering concerns.

Reinforcement learning extends the decision problem to future consequences. A recommendation may affect satisfaction, knowledge, or willingness to return, changing later states and rewards. In principle, a policy can optimize expected cumulative reward rather than immediate clicks. In practice, long horizons, uncertain state representations, delayed outcomes, and the difficulty of safe experimentation make this challenging. A reward that is easy to measure, such as session length, is not necessarily an adequate representation of the long-term value the system is intended to provide.

Diversity, fairness, explanations, and user control

Recommendation quality extends beyond matching historical interactions. Diversity concerns variation within or across recommendation lists; novelty concerns how unfamiliar an item is to a user; serendipity concerns useful discoveries that are also unexpected. These properties overlap but are not equivalent. An obscure item may be novel without being relevant, while a familiar item can still improve the diversity of a list dominated by one category.

Diversity can be introduced during candidate generation or final list construction. A reranking procedure may balance a relevance score against similarity to already selected items. More elaborate methods consider the list jointly rather than scoring each entry independently. The relevant notion of difference is domain-specific: diversity among music artists, news topics, price ranges, or technical viewpoints cannot be captured reliably by a single universal distance measure. List-level constraints can also conflict with one another, requiring explicit choices about priorities.

Fairness introduces multiple stakeholders. Users may receive different levels of recommendation quality, while creators, sellers, or job applicants may receive unequal exposure. Differences can arise from historical popularity, sparse data, catalog structure, or the system’s objectives. A model that optimizes average accuracy can perform poorly for users with uncommon tastes because the majority contributes most of the training signal. Similarly, exposure concentration can make it difficult for new providers to accumulate evidence of relevance.

There is no single fairness criterion suitable for every recommendation setting. Equal exposure across all items may ignore relevance and availability, while proportional exposure requires a defensible measure of merit. User-side and provider-side objectives can conflict. Consequently, fairness analysis requires specifying the population, outcome, comparison, and relevant constraints rather than merely labeling an algorithm fair or biased. Group-level averages may also hide substantial differences among individuals within the same group.

Explanations can make recommendations easier to assess. Examples include “because this author is in your library” or “similar to an item you rated highly.” Some explanations describe observable relationships, while others attempt to account for a model’s internal reasoning. Explainable artificial intelligence distinguishes the faithfulness of an explanation from its persuasiveness: a convincing narrative may not accurately describe the mechanism that produced the ranking.

User control can complement explanations. Interfaces may allow users to remove events from their history, specify that an item is unwanted, switch to a nonpersonalized ordering, or adjust the balance between familiarity and discovery. Such controls are meaningful only if the system interprets them consistently. A “not interested” action might refer to a specific item, its topic, its creator, or the current context; treating all such actions identically can create unintended changes. Evaluation of these features therefore involves both model behavior and the user’s understanding of what the controls do.

Privacy, security, and governance

Personalization often depends on behavioral records that can reveal sensitive interests or routines. Even when a dataset contains no explicit statement of religion, health status, or political preference, patterns of reading, viewing, or purchasing may support inferences about those subjects. Data privacy therefore concerns not only the protection of names and account identifiers but also the collection, combination, retention, and use of behavioral information.

Removing direct identifiers does not necessarily make interaction data anonymous. A distinctive sequence of ratings or purchases can sometimes be linked to outside information. Privacy risks can also arise from model outputs: a recommendation displayed on a shared device may reveal an inferred interest, and explanations can expose connections that users did not expect to become visible. Access controls, limited retention, and separation of data uses address different parts of this problem, but none alone establishes that a recommendation system is privacy-preserving.

Technical privacy approaches include differential privacy and federated learning. Differential privacy limits how much the release of a result can depend on an individual’s data, under a specified mathematical definition and privacy budget. Federated learning distributes parts of training to devices or organizations rather than centrally collecting all raw records. Federated learning does not by itself guarantee privacy, because model updates can still disclose information; it may need additional safeguards. Both approaches involve trade-offs among data access, utility, communication cost, and the precise threat being addressed.

Recommenders are also targets for manipulation. In shilling attacks, an adversary creates or controls accounts to submit interactions intended to promote or suppress items. Coordinated clicks, fabricated reviews, and strategically altered metadata can distort both collaborative and content-based models. These attacks are related to data poisoning, but recommendation presents distinctive incentives because exposure itself has economic value. Defensive analysis may examine unusual interaction patterns, account relationships, and inconsistencies between claimed preferences and subsequent behavior. Legitimate niche communities must not automatically be treated as malicious merely because their behavior differs from the majority.

Governance includes documenting objectives, assessing risks, maintaining audit records, and establishing responsibility for changes. In the European Union, the Digital Services Act includes transparency requirements concerning recommender systems used by online platforms. For designated very large online platforms and very large online search engines that use recommender systems, it also requires at least one option for each recommender system that is not based on profiling as defined in the General Data Protection Regulation. Such rules distinguish disclosure about recommendation from a demand to reveal every model parameter or commercial implementation detail.

Research and oversight face an important measurement problem: observed associations between recommendation exposure and later behavior do not by themselves establish causation. Users select services and content, platforms adapt to users, and external events affect both. Audits may therefore combine controlled experiments, independent data access, interface studies, and analysis of system documentation. Findings about one platform, population, or period require care when applied elsewhere, particularly when claims concern social influence rather than directly measured ranking behavior.