{
  "id": "deep-q-network",
  "title": "Deep Q-Network",
  "lang": "en",
  "source_lang": "en",
  "summary": "A deep Q-network is a reinforcement-learning method that uses a deep neural network, experience replay, and a target network to estimate action values and learn decisions from experience.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "DQN",
    "Deep Q-Networks",
    "Deep Q-Learning Network"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/deep-q-network/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/deep-q-network/",
      "json": "https://aiwiki.page/zh/deep-q-network.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "A deep Q-network (DQN) is a reinforcement-learning method that combines Q-learning with a deep neural network to estimate the long-term value of taking particular actions. It allows an agent to learn from high-dimensional observations, such as images, without maintaining a separate value estimate for every possible state–action pair. The standard DQN algorithm stabilizes learning through experience replay, which reuses stored interactions, and a target network, which provides more slowly changing learning targets. DQN is primarily designed for problems with a finite, discrete set of actions."
    },
    {
      "heading": "Historical development",
      "text": "Q-learning was originally formulated using a table of action values. This approach becomes impractical when an environment has very many states or provides observations such as camera images. Earlier work explored neural-network approximations to value functions, but combining nonlinear approximation with reinforcement-learning updates often produced unstable learning.\n\nResearchers at DeepMind presented a prominent deep Q-learning system in the 2013 paper Playing Atari with Deep Reinforcement Learning. It learned to play several Atari 2600 games using screen images and rewards. A subsequent paper, Human-level control through deep reinforcement learning, published in Nature in 2015, described the influential DQN architecture and training procedure incorporating a separate target network, and evaluated it across 49 Atari games.\n\nThe Atari results demonstrated that a shared network architecture and broadly shared training procedure could learn many distinct tasks from visual observations. A separate agent was trained for each game: the results did not represent one network simultaneously mastering all games or learning general human-level intelligence. Performance also varied substantially between games, particularly where rewards were sparse or success required extended planning.\n\nDQN became an important foundation for deep reinforcement learning by showing how representation learning and value-based control could be trained together."
    },
    {
      "heading": "Mathematical formulation",
      "text": "States, actions, and returns\n\nDQN is usually formulated within a Markov decision process. At time \\(t\\), an agent observes a state \\(s_t\\), chooses an action \\(a_t\\), receives a reward \\(r_t\\), and transitions to \\(s_{t+1}\\).\n\nThe objective is to maximize the expected discounted return,\n\n\\[G_t=\\sum_{k=0}^{\\infty}\\gamma^k r_{t+k},\\]\n\nwhere \\(0\\leq\\gamma<1\\) is a discount factor. Episodic formulations may also use \\(\\gamma=1\\) when returns are well defined.\n\nFor a policy \\(\\pi\\), the action-value function is\n\n\\[Q^\\pi(s,a) = \\mathbb{E}_\\pi[G_t\\mid s_t=s,\\ a_t=a].\\]\n\nThe optimal action-value function \\(Q^*\\) satisfies the Bellman optimality equation,\n\n\\[Q^*(s,a) = \\mathbb{E}\\left[ r_t+\\gamma\\max_{a'}Q^*(s_{t+1},a') \\mid s_t=s,\\ a_t=a \\right].\\]\n\nThis equation expresses an action’s value as its immediate reward plus the discounted value of the best subsequent action.\n\nNeural approximation\n\nInstead of storing a table, DQN represents action values with an artificial neural network, written \\(Q(s,a;\\theta)\\), where \\(\\theta\\) denotes its trainable parameters.\n\nIn the standard architecture, the network takes a state representation as input and outputs one value for each available action. The greedy action is\n\n\\[a^*(s)=\\arg\\max_a Q(s,a;\\theta).\\]\n\nFor image observations, a convolutional neural network extracts spatial features before fully connected layers produce the action values. For low-dimensional numerical observations, a multilayer perceptron may be sufficient.\n\nThe network does not directly predict the best action as a class label. It estimates expected returns, from which actions are selected."
    },
    {
      "heading": "Learning mechanism",
      "text": "DQN uses temporal-difference learning: an estimate is updated toward a target that itself contains an estimated future value. This process is called bootstrapping.\n\nExperience replay\n\nDuring interaction, the agent stores transitions\n\n\\[(s_t,a_t,r_t,s_{t+1},d_t)\\]\n\nin an experience replay buffer, where \\(d_t\\) indicates whether the transition ends the underlying task.\n\nTraining uses randomly sampled minibatches of stored transitions rather than only the latest interaction. This has several effects:\n\n- It reduces the strong temporal correlations present in consecutive observations.\n- It allows an interaction to contribute to several updates.\n- It mixes experience collected under different versions of the agent’s behavior.\n\nReplay does not make the data fully independent or the training distribution stationary. The buffer’s contents continue to change, and important experiences can be rare or overwritten.\n\nDQN is off-policy: it can learn about a greedy target policy from transitions generated by a different behavior policy. In standard DQN, that behavior policy is commonly an exploratory version of the agent’s current policy.\n\nTarget network\n\nDQN maintains an online network with parameters \\(\\theta\\) and a target network with parameters \\(\\theta^-\\). For a sampled transition, a standard target is\n\n\\[y_t= r_t+\\gamma(1-d_t) \\max_{a'}Q(s_{t+1},a';\\theta^-).\\]\n\nThe target-network parameters are held fixed between periodic copies from the online network. This slows changes in the quantities being predicted and helps reduce destabilizing feedback.\n\nWhen \\(d_t=1\\), the target is simply \\(r_t\\): there is no continuation value after a genuine terminal state. An externally imposed time limit is not necessarily a terminal state of the underlying task, so treating every truncation as terminal can introduce bias.\n\nLoss and parameter updates\n\nA basic training objective is the expected squared temporal-difference error,\n\n\\[L(\\theta) = \\mathbb{E}_{(s,a,r,s',d)\\sim\\mathcal D} \\left[ \\left( r+\\gamma(1-d)\\max_{a'}Q(s',a';\\theta^-) - Q(s,a;\\theta) \\right)^2 \\right],\\]\n\nwhere \\(\\mathcal D\\) denotes the replay buffer’s sampling distribution.\n\nParameters are updated using minibatch gradient optimization and backpropagation. The target is treated as constant during the update; gradients do not propagate through the target network. The update is therefore commonly described as a semi-gradient update.\n\nA Huber loss is also widely used. It is quadratic for small errors and linear for large errors, limiting the influence of unusually large temporal-difference errors. Learning rate, replay capacity, update frequency, and target-copy interval are important hyperparameters.\n\nExploration\n\nDQN commonly uses an \\(\\varepsilon\\)-greedy behavior policy:\n\n- With probability \\(\\varepsilon\\), choose a random action.\n- Otherwise, choose an action maximizing the online network’s estimated value.\n\nThe exploration probability often decreases during training. This mechanism is simple, but it does not provide systematic exploration over long action sequences. An agent may fail to discover useful behavior when rewards require many coordinated actions before any positive feedback occurs."
    },
    {
      "heading": "Canonical Atari implementation",
      "text": "The influential Atari implementation transformed raw game frames into a more manageable input representation. It used grayscale images resized to \\(84\\times84\\) pixels and stacked four recent frames. Frame stacking provided limited information about motion that a single image could not reveal.\n\nA typical action was repeated for several emulator frames. Image preprocessing also reduced flickering effects. Rewards were clipped to a small range, helping make reward scales more comparable across games.\n\nThese choices were important components of the experimental system, but they are not the definition of DQN. Other applications can use different observations, reward scales, network architectures, and interaction schedules.\n\nReward clipping also changes the optimization objective: an agent trained on clipped rewards is maximizing a transformed return, not necessarily the original score. Likewise, a short stack of frames does not guarantee that the input contains all information needed to predict future outcomes."
    },
    {
      "heading": "Major extensions",
      "text": "Double DQN\n\nStandard DQN uses the same target-network values both to select the maximizing action and to evaluate it. When estimates contain noise, maximizing over them can favor overestimated values.\n\nDouble DQN separates action selection from evaluation:\n\n\\[a^*=\\arg\\max_{a'}Q(s_{t+1},a';\\theta),\\]\n\n\\[y_t^{\\mathrm{Double}} = r_t+\\gamma(1-d_t) Q(s_{t+1},a^*;\\theta^-).\\]\n\nThe online network selects the action, and the target network evaluates it. This reduces a source of overestimation, although it does not guarantee unbiased estimates or stable learning in every setting.\n\nPrioritized experience replay\n\nPrioritized replay samples some transitions more frequently, often using the magnitude of their temporal-difference errors as a priority. Such transitions may provide more learning progress than uniformly sampled ones.\n\nBecause prioritization changes the sampling distribution, importance-sampling weights can compensate for the resulting bias. Large errors do not always indicate useful experience: stochastic rewards or irreducible uncertainty can also generate them.\n\nDueling networks\n\nA dueling architecture separates the estimation of state value from the relative advantages of individual actions. A common combination is\n\n\\[Q(s,a) = V(s)+A(s,a) - \\frac{1}{|\\mathcal A|} \\sum_{b\\in\\mathcal A}A(s,b).\\]\n\nHere, \\(V(s)\\) estimates state value and \\(A(s,a)\\) represents an action’s relative advantage. Subtracting the mean advantage resolves ambiguity in how the two components combine. This architecture can help when many actions have similar consequences.\n\nDistributional and combined methods\n\nDistributional reinforcement learning models a distribution of possible returns rather than only their expectation. Distributional DQN variants can still select actions by expected return while learning a richer prediction target.\n\nOther extensions include multi-step returns, parameterized exploration noise, and recurrent architectures for partially observed environments. Rainbow combines several DQN improvements, including Double DQN, prioritized replay, dueling networks, multi-step learning, distributional value estimation, and noisy networks."
    },
    {
      "heading": "Applications and relationship to other methods",
      "text": "DQN is particularly suited to environments with a manageable discrete action set and sufficiently informative experience. It has been used in game-playing research and investigated for simulated scheduling, resource allocation, and other sequential decision problems.\n\nIts central distinction from supervised learning is that correct action labels are not supplied. Rewards may be delayed, and the agent’s actions influence the observations it later receives. Nevertheless, each network update resembles regression against a constructed numerical target.\n\nDQN is value-based rather than a policy-gradient method: it learns action values and derives a policy from them. Actor–critic approaches instead train an explicit policy alongside a value estimator.\n\nStandard DQN is poorly suited to continuous action spaces because its target requires maximizing over possible actions. With finitely many actions, their values can be enumerated; with continuously varying actions, that maximization generally requires another mechanism."
    },
    {
      "heading": "Limitations and evaluation",
      "text": "Stability and convergence\n\nExperience replay and target networks improve practical stability but do not establish general convergence for deep nonlinear value approximation. DQN combines function approximation, bootstrapping, and off-policy learning—a combination often called the deadly triad. In some settings, these ingredients can produce inaccurate or divergent estimates.\n\nSample efficiency and computational cost\n\nDQN can require extensive environment interaction. Replay improves reuse of collected transitions, but learning may still be expensive when each interaction requires physical operation or a complex simulation. Training cost depends on network size, observation storage, replay sampling, and the number of updates per interaction.\n\nPartial observability and generalization\n\nIf identical observations can correspond to different underlying situations, a feedforward DQN may be unable to choose correctly without additional history. Frame stacking only partly addresses this problem. A recurrent neural network can incorporate longer histories, though it introduces additional training challenges.\n\nStrong performance in one environment also does not imply robust generalization. Changes in visual appearance, transition dynamics, or available actions can cause substantial distribution shift.\n\nReward and benchmark dependence\n\nDQN optimizes the reward signal it is given, not an independently specified notion of task success. Poorly designed rewards can encourage unintended behavior, including reward hacking.\n\nReported performance depends on evaluation details such as random seeds, interaction budgets, exploration during evaluation, episode termination rules, and whether learning continues during testing. Human-normalized game scores are useful comparative measures, but aggregate scores can obscure large differences between individual tasks."
    }
  ],
  "links": [
    "reinforcement-learning",
    "q-learning",
    "deep-learning",
    "deepmind",
    "representation-learning",
    "markov-decision-process",
    "policy-reinforcement-learning",
    "artificial-neural-network",
    "convolutional-neural-network",
    "multilayer-perceptron",
    "temporal-difference-learning",
    "backpropagation",
    "loss-function",
    "hyperparameter",
    "supervised-learning",
    "policy-gradient-method",
    "recurrent-neural-network",
    "generalization",
    "distribution-shift",
    "reward-hacking"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}