{
  "id": "naive-bayes-classifier",
  "title": "Naive Bayes classifier",
  "lang": "en",
  "source_lang": "en",
  "summary": "A family of probabilistic classifiers that applies Bayes’ theorem using the assumption that features are conditionally independent given the class.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Naïve Bayes classifier",
    "naive Bayesian classifier",
    "simple Bayesian classifier",
    "independence Bayes classifier"
  ],
  "generated_at": "2026-10-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/naive-bayes-classifier/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/naive-bayes-classifier/",
      "json": "https://aiwiki.page/zh/naive-bayes-classifier.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "A naive Bayes classifier is a probabilistic supervised learning method that assigns an observation to a class using Bayes’ theorem and a simplifying assumption about its features. It treats features as conditionally independent once the class is known. Rather than denoting one specific implementation, the name covers a family of models with different feature distributions. These classifiers are used in machine learning, particularly for document classification and spam filtering. (scikit-learn.org)"
    },
    {
      "heading": "Mathematical formulation",
      "text": "Let \\(y\\) denote a class label and \\(x=(x_1,\\ldots,x_d)\\) an observed feature vector. Bayes’ theorem expresses the posterior probability of a class as\n\n\\[P(y\\mid x)=\\frac{P(y)P(x\\mid y)}{P(x)}.\\]\n\nHere, \\(P(y)\\) is the class prior and \\(P(x\\mid y)\\) is the class-conditional likelihood. The defining conditional independence assumption is\n\n\\[P(x\\mid y)=\\prod_{j=1}^{d}P(x_j\\mid y).\\]\n\nThis is stronger than pairwise independence: the entire joint conditional distribution must factorize. It does not require features to be independent in the population as a whole. The usual prediction rule is therefore\n\n\\[\\hat y=\\arg\\max_y \\left[P(y)\\prod_{j=1}^{d}P(x_j\\mid y)\\right].\\]\n\nThe denominator is identical for all candidate classes and can be omitted when selecting the largest score. (scikit-learn.org)\n\nImplementations commonly calculate log scores,\n\n\\[s_y=\\log P(y)+\\sum_j\\log P(x_j\\mid y),\\]\n\ninstead of multiplying many small probabilities. This reduces numerical underflow without changing which class wins. Normalized posterior estimates can be recovered from these scores, although accurate classification does not guarantee accurate probabilities. (nlp.stanford.edu)"
    },
    {
      "heading": "Learning and smoothing",
      "text": "Training estimates class frequencies and feature distributions from labeled training data. In count-based models, maximum likelihood estimation uses observed relative frequencies. An unseen feature–class combination then receives probability zero, potentially making the entire class score zero regardless of the other evidence. (nlp.stanford.edu)\n\nAdditive smoothing addresses this problem by adding pseudocounts. For a multinomial text model,\n\n\\[\\hat\\theta_{yj} =\\frac{N_{yj}+\\alpha} {\\sum_{k=1}^{V}N_{yk}+\\alpha V},\\]\n\nwhere \\(N_{yj}\\) counts occurrences of vocabulary term \\(j\\) in class \\(y\\), \\(V\\) is vocabulary size, and \\(\\alpha>0\\) controls smoothing. Setting \\(\\alpha=1\\) gives Laplace smoothing. This prevents vocabulary terms absent from one class’s training documents from automatically eliminating that class. (nlp.stanford.edu)\n\nThe selection of the highest-posterior class is a decision rule, not a requirement to estimate every parameter through fully Bayesian inference. Naive Bayes implementations may use frequency estimates or smoothed estimates rather than integrate over parameter uncertainty. (scikit-learn.org)"
    },
    {
      "heading": "Principal variants",
      "text": "Gaussian naive Bayes handles continuous features by assigning each feature a class-specific normal distribution. Training estimates a mean and variance for each feature within each class. The model does not estimate cross-feature correlations. (scikit-learn.org)\n\nMultinomial naive Bayes models category counts, especially word frequencies. In document classification, its score includes \\(x_j\\log\\theta_{yj}\\), so repeated occurrences contribute repeatedly. Its text interpretation is a sequence of independent word draws given the class and document length—not independent multinomial count coordinates, whose total is constrained. (cmi.ac.in)\n\nBernoulli naive Bayes represents each feature as a binary indicator governed by a Bernoulli distribution. In text applications, it records whether a word occurs, not how often. Both presence and absence contribute to its likelihood, unlike the usual multinomial document score. (aaai.org)\n\nCategorical naive Bayes models each discrete feature through its own categorical distribution. It is appropriate when features have several unordered values, rather than representing repeated event counts. (scikit-learn.org)\n\nComplement naive Bayes modifies multinomial classification by estimating weights from documents outside each target class. Introduced in research on text classification, it addresses weaknesses involving class imbalance and feature weighting. It is an adaptation rather than simply another likelihood distribution within the basic model. (people.csail.mit.edu)"
    },
    {
      "heading": "Text representation and computational properties",
      "text": "In natural language processing, documents commonly undergo tokenization and conversion to a bag-of-words representation. Multinomial models preserve token counts but discard word order; Bernoulli models additionally discard repetition. These choices distinguish the statistical events each model treats as evidence. McCallum and Nigam’s 1998 comparison demonstrated that the two event models can produce materially different classification results. (cmi.ac.in)\n\nThe independence assumption substantially reduces the number of parameters compared with an unrestricted joint distribution. Training mainly requires accumulating statistics rather than solving a large iterative optimization problem. Text-model training and prediction can consequently be efficient even with large vocabularies. Some implementations also support incremental updates, allowing online learning or processing datasets in batches that need not fit entirely in memory. (nlp.stanford.edu)"
    },
    {
      "heading": "Limitations and evaluation",
      "text": "Real features frequently remain dependent within a class. Correlated words, for example, can cause evidence to be counted repeatedly. Nevertheless, incorrect probability estimates may preserve the correct ordering of classes. Domingos and Pazzani showed in 1997 that naive Bayes can be optimal under zero–one loss even when conditional independence is violated. This does not establish that it will perform well on every dependent dataset. (gwern.net)\n\nProbability quality therefore requires separate attention from classification accuracy. A model may choose useful labels while producing poorly calibrated confidence estimates. Its restricted representation also limits which relationships it can capture; increasing training data does not remove an inappropriate independence assumption. (gwern.net)\n\nEvaluation must keep learned preprocessing within the training partition. Feature selection, vocabulary construction, and other fitted transformations can introduce data leakage when they use held-out observations. Cross-validation can compare model variants and smoothing hyperparameters, while an independent test set measures performance after those choices have been made. (scikit-learn.org)"
    }
  ],
  "links": [
    "supervised-learning",
    "bayes-theorem",
    "machine-learning",
    "probability",
    "conditional-independence",
    "training-data",
    "maximum-likelihood-estimation",
    "bayesian-inference",
    "normal-distribution",
    "bernoulli-distribution",
    "natural-language-processing",
    "tokenization",
    "bag-of-words-model",
    "online-learning",
    "loss-function",
    "feature-selection",
    "data-leakage",
    "cross-validation",
    "hyperparameter",
    "test-set"
  ],
  "sources": [
    {
      "title": "scikit-learn.org/stable/modules/naive_bayes.html",
      "url": "https://scikit-learn.org/stable/modules/naive_bayes.html"
    },
    {
      "title": "nlp.stanford.edu/IR-book/html/htmledition/naive-bayes-text-classification-1.html",
      "url": "https://nlp.stanford.edu/IR-book/html/htmledition/naive-bayes-text-classification-1.html"
    },
    {
      "title": "cmi.ac.in/~madhavan/courses/dmml2019aug/literature/mccallum-nigam.pdf",
      "url": "https://www.cmi.ac.in/~madhavan/courses/dmml2019aug/literature/mccallum-nigam.pdf"
    },
    {
      "title": "aaai.org/papers/041-ws98-05-007/",
      "url": "https://aaai.org/papers/041-ws98-05-007/"
    },
    {
      "title": "scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.CategoricalNB.html",
      "url": "https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.CategoricalNB.html"
    },
    {
      "title": "people.csail.mit.edu/jrennie/papers/icml03-nb.pdf",
      "url": "https://people.csail.mit.edu/jrennie/papers/icml03-nb.pdf"
    },
    {
      "title": "nlp.stanford.edu/IR-book/html/htmledition/properties-of-naive-bayes-1.html",
      "url": "https://nlp.stanford.edu/IR-book/html/htmledition/properties-of-naive-bayes-1.html"
    },
    {
      "title": "gwern.net/doc/ai/1997-domingos.pdf",
      "url": "https://gwern.net/doc/ai/1997-domingos.pdf"
    },
    {
      "title": "scikit-learn.org/stable/common_pitfalls.html",
      "url": "https://scikit-learn.org/stable/common_pitfalls.html?highlight=linearregression"
    }
  ],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}