CatBoost is an open-source machine-learning library and family of gradient-boosting methods developed by Yandex. It builds ensembles of decision trees for classification, regression, ranking, and related prediction tasks. Its distinguishing techniques include ordered processing of categorical features, ordered boosting to reduce a particular source of training-related prediction bias, and, by default, symmetric decision trees. CatBoost is especially associated with tabular datasets containing mixtures of numerical and categorical variables.
Development and scope
CatBoost was publicly released in 2017. Its name derives from “categorical boosting,” reflecting its emphasis on learning from categorical variables without requiring users to construct all numerical encodings manually. The 2018 research paper CatBoost: unbiased boosting with categorical features described the statistical motivation for its ordered methods.
The project combines a research approach with a production-oriented software library. Its core implementation is written in C++, with interfaces including Python and R, command-line tools, and facilities for model deployment. The software is distributed under the Apache License 2.0.
CatBoost belongs to the same broad family of tree-boosting systems as XGBoost and LightGBM. Their capabilities overlap substantially, but they differ in tree construction, categorical-feature processing, optimization strategies, and default settings.
Learning method
Additive tree ensembles
CatBoost uses decision trees as the components of an ensemble. For a scalar prediction task, the model can be represented as
where is an initial prediction, is the tree added at iteration , and controls its contribution.
Each new tree is fitted using information derived from the current model’s errors, typically gradients of a chosen loss function. Leaf values may be estimated using gradient- or second-order optimization methods, depending on the objective and configuration. The learning rate, number of iterations, and tree depth jointly influence model capacity and training behavior.
Supported tasks include binary classification, multiclass classification, numerical regression, and learning to rank. The model’s numerical output is interpreted according to the task: for example, a binary classifier can transform its raw score into a class probability.
Symmetric trees
By default, CatBoost grows symmetric trees, also called oblivious decision trees. Every internal node at the same depth uses the same split condition. A path can therefore be represented by a sequence of binary split outcomes, which identifies the leaf directly.
A complete symmetric tree of depth has leaves. This regular structure supports efficient prediction and can act as a structural constraint on model complexity. It also restricts the partitions a single tree can express compared with an unrestricted tree of the same depth.
Symmetric trees are a default rather than a universal requirement: CatBoost also provides non-symmetric growth policies. Available analysis and export features can depend on the selected policy.
Categorical-feature processing
A categorical feature represents membership in a discrete category, such as a product identifier, country code, or device type. Treating arbitrary category identifiers as ordinary numbers introduces a potentially misleading numerical ordering. One-hot encoding avoids that ordering but can create many variables when a feature has numerous distinct values.
CatBoost supports one-hot encoding for some categorical features and statistical transformations for others. The choice depends on settings, task, and execution mode.
Ordered target statistics
A common target-encoding method replaces a category with a statistic of the target values associated with it. A naïve implementation that includes an observation’s own target in its encoded feature can introduce target leakage, particularly for rare categories.
CatBoost addresses this problem using ordered statistics. Conceptually, the training observations are arranged in a random permutation. The encoded value for an observation uses target information only from observations preceding it in that permutation.
For a numerical target, a simplified smoothed category statistic is
where is the category of observation , is an earlier target value, is its position in the permutation, is a prior value, and controls smoothing.
This formula illustrates the principle rather than every transformation implemented by CatBoost. Depending on the task, the library can use target discretization, class-dependent statistics, category frequencies, and other constructions.
For prediction on new data, the required categorical statistics are derived from training data and retained with the model; the new observation’s target is not required. Smoothing and fallback values allow previously unseen categories to receive defined representations, though they do not guarantee accurate predictions for unfamiliar categories.
Feature combinations
CatBoost can also form combinations of categorical features and compute statistics for those combinations. A category pair such as “product type × region” may contain information absent from either feature alone.
This automates part of feature engineering, but larger combinations can increase memory use and the risk of fitting sparse patterns. Their construction and complexity are controlled by training settings.
Ordered boosting
Ordered boosting addresses a related but distinct problem from categorical encoding. In conventional boosting, the gradient calculated for a training observation depends on a model already fitted using that observation’s target. This dependence can make the distribution of training-time gradients differ from the corresponding behavior on unseen observations. The CatBoost research describes this as prediction shift.
Conceptually, ordered boosting estimates an observation’s gradient using a model trained only on observations earlier in a permutation. This reduces the direct influence of that observation’s target on the model used to calculate its update. Practical implementations share computations rather than independently training a complete model for every possible prefix.
Two distinctions are important:
- Ordered categorical statistics concern how categorical inputs are represented.
- Ordered boosting concerns how training-time predictions and gradients are constructed.
CatBoost supports both Ordered and Plain boosting modes. Use of the library does not imply that every model uses ordered boosting; defaults and availability depend on the configuration. Neither ordered mechanism eliminates every form of leakage or overfitting.
Software and execution
CatBoost supports training on CPUs and, for supported configurations, GPUs. GPU training can accelerate suitable workloads, although supported objectives, parameters, and numerical behavior are not identical across execution modes.
The library accepts numerical and categorical features and also provides processing facilities for text and embedding features. Numerical variables are quantized into discrete intervals for split selection. Missing numerical values can be handled according to a specified policy. Categorical inputs have separate representation requirements; their missing values are not necessarily treated in the same way as numerical missing values.
Common controls include tree depth, iteration count, learning rate, sampling settings, and leaf-value regularization. A validation set can be used for monitoring and early stopping. Methods for feature importance and SHAP values support model inspection, although their availability can depend on the model type.
Deployment options include native model loading and export facilities. Export compatibility varies with the model’s features, tree structure, and destination format.
Applications and limitations
CatBoost is used for structured prediction problems such as customer-response modeling, demand estimation, fraud detection, and ranking in search or recommendation systems. It is particularly relevant when categorical variables are numerous or have high cardinality.
Its practical strengths include integrated categorical processing, the ability to model nonlinear interactions, and relatively little need for feature scaling. These properties do not establish a universal accuracy advantage over other tree ensembles or neural methods.
Important limitations include:
- Computational cost. Large category-statistic tables, feature combinations, and deeper trees can consume substantial memory and training time.
- Limited extrapolation. Tree ensembles generally produce piecewise-constant predictions and do not naturally extend numerical trends beyond observed ranges.
- Dependence on data design. Ordered processing does not repair future information embedded in features, duplicated entities across evaluation splits, or inappropriate sampling.
- Restricted guarantees under changing conditions. Previously unseen categories can be processed, but distribution shift can still degrade performance.
- Interpretation limits. Feature importance and attribution values describe aspects of model behavior; they do not establish causal relationships.
For chronological data, a random permutation is not equivalent to a time-respecting evaluation. For grouped observations, such as repeated records from the same customer, ordered learning likewise does not prevent leakage between groups. Reliable assessment still depends on a suitable test set and an evaluation design matching the intended prediction setting.