TF–IDF (term frequency–inverse document frequency) is a family of numerical weighting methods used in information retrieval and natural language processing to measure how strongly a term characterizes a document relative to a collection of documents. It combines a local factor, measuring a term’s occurrence within a document, with a collection-wide factor that reduces the weight of terms appearing in many documents. TF–IDF is widely used to represent text for document search, similarity measurement, keyword extraction, and machine learning. It is not a single standardized formula: implementations differ in their frequency scaling, smoothing, and normalization.
Definition and intuition
A term that occurs repeatedly in one document may help identify its subject. However, a term appearing in nearly every document usually provides little help in distinguishing one document from another. TF–IDF combines these observations:
- Term frequency (TF) measures the occurrence of a term within an individual document.
- Inverse document frequency (IDF) measures how uncommon that term is across the document collection.
The resulting weight is generally written as
where is a term, is a document, and is the reference collection, or corpus.
Here, a “term” need not be a complete word. Depending on tokenization, it may be a word stem, a character sequence, or an n-gram. Similarly, a “document” may be a complete article, a paragraph, a product description, or another chosen text unit. These choices affect both frequency statistics and the interpretation of the weights.
A high TF–IDF weight indicates that a term is prominent within a document and relatively uncommon in the reference collection. It does not, by itself, establish that the term is meaningful, factually important, or relevant to a particular user’s question.
Mathematical formulation
Term frequency
Let be the number of occurrences of term in document . Common TF definitions include:
Raw count
Relative frequency
where the denominator is the total number of term occurrences in the document.
Logarithmic frequency
Logarithmic scaling reduces the influence of repeated occurrences: a term appearing twenty times receives more weight than one appearing twice, but not ten times as much.
A binary variant uses when a term is present and otherwise. These alternatives encode different assumptions about how much additional evidence repeated occurrences provide.
Inverse document frequency
Let be the number of documents in the collection, and let
be the document frequency of term . A conventional IDF definition is
Document frequency counts documents containing a term, not the total number of its occurrences. A term appearing one hundred times in a single document contributes only one to its document frequency.
Under this formula:
- A term appearing in every document has IDF .
- A term appearing in few documents has a larger IDF.
- A term absent from the collection makes the formula undefined and requires separate handling.
The logarithm’s base changes all IDF values by a constant factor. This leaves cosine similarities unchanged when the rest of the weighting procedure is fixed, although it can matter when weights are used with other operations or fixed numerical thresholds.
A common smoothed and shifted variant is
Here, adding inside the ratio smooths the counts, while the additional outside the logarithm gives a term appearing in every document a weight of , rather than . Other conventions exist, so a specification of “TF–IDF” alone does not fully determine numerical results.
Vector normalization
Given a vocabulary of terms, a document can be represented as
This is a weighted form of the bag-of-words model. Most documents contain only a small fraction of the vocabulary, so their vectors are typically sparse.
An optional final step applies unit-length normalization:
This reduces the direct effect of vector magnitude when comparing documents of different lengths. It does not remove every possible length-related bias. A zero vector cannot be normalized by this expression and must be handled separately.
Worked example
Suppose a collection contains documents. The term “quasar” appears in documents, while “system” appears in . Using natural logarithms,
If a particular document contains “quasar” three times and “system” ten times, raw-count TF produces
Although “system” occurs more frequently within the document, “quasar” receives the higher weight because it is much rarer across the collection.
These are unnormalized weights. Logarithmic TF or a different IDF convention would change their values.
Document search and similarity
TF–IDF is closely associated with the vector-space approach to retrieval. A document and a search query are represented in a shared vector space, and a similarity function compares their vectors.
A common choice is cosine similarity:
For unit-length vectors, this reduces to their dot product. Terms shared by the query and document contribute to the score; terms with large IDF values can contribute more than widely distributed terms.
Query and document weighting need not be identical. For example, a system may use logarithmic TF for documents and a simpler presence-based representation for short queries. Query IDF values are normally derived from the document collection rather than from the query alone.
A search engine can calculate scores efficiently using an inverted index, which maps terms to the documents containing them. TF–IDF does not require constructing or storing a dense document-by-vocabulary array.
Historical development
TF–IDF developed from statistical approaches to automatic indexing and document retrieval. Term-frequency weighting was explored in early work on automated text processing, including the work of Hans Peter Luhn in the 1950s.
Karen Spärck Jones introduced inverse document frequency in a 1972 paper on term specificity. The central idea was that a term’s usefulness for distinguishing documents depends on how broadly it occurs across the collection, not simply on its frequency within an individual document.
Gerard Salton and collaborators developed and evaluated term-weighting methods within the SMART information retrieval system. Salton and Christopher Buckley’s 1988 study of term-weighting approaches helped establish widely used combinations of local frequency weighting, collection-frequency weighting, and document normalization.
The term TF–IDF consequently refers to a related family of schemes rather than one uniquely defined algorithm.
Applications and implementation
TF–IDF serves as a general-purpose feature extraction method for text. Beyond search, common applications include:
- Text classification: TF–IDF vectors can serve as inputs to a support vector machine or other classifier.
- Document clustering: Similarity between vectors can support methods such as k-means clustering.
- Keyword extraction: High-weight terms can identify expressions that distinguish a document from its reference collection.
- Document comparison: Lexical overlap can help detect related documents, repeated content, and near duplicates.
- Text summarization: Some automatic text summarization methods use term weights as part of sentence scoring.
Results depend substantially on preprocessing. Lowercasing, punctuation handling, stemming, stop-word removal, and the inclusion of word or character n-grams all change the feature space. IDF downweights common terms, but it does not make stop-word removal equivalent or universally unnecessary.
In supervised experiments, the vocabulary and IDF statistics are normally learned from the training data and then reused unchanged for validation and testing. Fitting them on held-out documents can introduce data leakage by allowing the representation to use information from outside the training partition. In retrieval evaluation, by contrast, computing IDF over the indexed document collection is generally part of the defined retrieval setting.
With a fixed vocabulary, previously unseen terms are usually ignored. Changes to the collection can also change IDF values, so an evolving index may require updated statistics. A reproducible implementation must specify the corpus, preprocessing, vocabulary rules, TF formula, IDF formula, and normalization procedure.
Limitations and related methods
TF–IDF measures lexical distribution rather than meaning. Its principal limitations include:
- Loss of linguistic structure: Ordinary unigram representations ignore word order and grammatical relationships.
- No inherent synonym handling: Documents about the same subject may receive low similarity if they use different words.
- No contextual disambiguation: Different meanings of a word generally share the same feature.
- Sensitivity to rare noise: Typographical errors, identifiers, and accidental strings may receive large IDF values.
- Corpus dependence: A term can be distinctive in one collection and ordinary in another. Weights from different corpora are not directly comparable without accounting for their construction.
- Imperfect repetition and length handling: Raw-count TF can overreward repeated terms, while simple normalization does not capture all relationships between document length and relevance.
BM25 is a related retrieval scoring family that explicitly models term-frequency saturation and document-length adjustment. Its scoring function is not simply a cosine similarity between conventional TF–IDF vectors.
Dense representations such as word embeddings and contextual representations from transformer models can capture relationships beyond exact term overlap. They require different computational resources and introduce their own modeling limitations. TF–IDF remains useful where sparse storage, transparent term contributions, inexpensive computation, and exact lexical matching are important.