Qwen3

Qwen3 is a family of large language models developed by the Qwen team at Alibaba Cloud and initially released on April 29, 2025. Its original text-model lineup includes both dense and mixture-of-experts models, with publicly downloadable weights released under the Apache 2.0 license. The family follows Qwen2.5 and supports multilingual text generation, instruction following, mathematical problem-solving, programming, and interaction with external tools. (alibabagroup.com)

A distinguishing feature of the initial Qwen3 models is their integration of “thinking” and “non-thinking” modes: a single model can generate intermediate reasoning before answering or respond without that additional output. This design connects model behavior to the amount of computation allocated during inference. Subsequent Qwen3-2507 releases separated these behaviors into dedicated Thinking and Instruct checkpoints. The broader Qwen3 ecosystem also includes specialized coding, retrieval, vision-language, and multimodal models, whose architectures and capabilities must be distinguished from those of the original text models. (github.com)

Development and release history

Qwen3 was developed by the Qwen team at Alibaba Cloud as the successor to the Qwen2 and Qwen2.5 generations, released on June 6 and September 19, 2024, respectively. The project also drew on the team’s work on QwQ reasoning models. “Qwen3” denotes a family of large language models and their downloadable checkpoints, rather than a single chatbot or hosted product. Qwen Chat provides a user-facing conversational interface, while Alibaba Cloud Model Studio provides hosted model access through an application programming interface (API); these access channels are distinct from the model weights that developers can download and run independently. (github.com)

The initial release was announced on April 29, 2025, in both Alibaba’s corporate announcement and the Qwen team’s launch article, Qwen3: Think Deeper, Act Faster. It comprised eight model sizes: six dense models and two mixture-of-experts models, spanning approximately 0.6 billion to 235 billion total parameters. The release made weights available through repositories including Hugging Face and ModelScope under the Apache 2.0 license. The launch article supplied model specifications, evaluation results, and execution examples, and distinguished post-trained checkpoints from their pretrained “Base” counterparts. Thus, the public release included both downloadable artifacts and documentation for identifying and using particular models, rather than only access to an online demonstration. (alibabagroup.com)

A defining feature of the April post-trained models was the combination of thinking and non-thinking behavior within one checkpoint: the former generated intermediate reasoning before an answer, whereas the latter answered directly. This differed from using separate general-chat and reasoning models. The subsequent Qwen3 Technical Report, first submitted to arXiv on May 14, 2025, documented the architecture, pretraining, post-training, and evaluations behind that release. Its author list includes An Yang, Binyuan Hui, and Junyang Lin, among other contributors. The report therefore followed the initial weight release rather than serving as its simultaneous publication. (arxiv.org)

The Qwen3-2507 update changed this packaging by distributing separate Instruct and Thinking checkpoints. The rollout began with Qwen3-235B-A22B-Instruct-2507 on July 21, 2025, followed by Qwen3-235B-A22B-Thinking-2507 on July 25. Instruct-2507 retained only non-thinking behavior and did not emit thinking blocks; Thinking-2507 retained only thinking behavior, with its default chat template initiating the reasoning sequence. Consequently, the original hybrid models’ mode-switching controls did not apply uniformly to these updates: selecting a response style now involved selecting the corresponding checkpoint, rather than merely changing a setting within the same hybrid model. (github.com)

The smaller mixture-of-experts updates followed on July 30, 2025, with Qwen3-30B-A3B-Instruct-2507, and July 31 with Qwen3-30B-A3B-Thinking-2507. Both Qwen3-4B-Instruct-2507 and Qwen3-4B-Thinking-2507 were released on August 6, which the repository described as the final open release of the 2507 series. Despite the shared “2507” designation, the rollout therefore extended into August and covered three sizes, not every size in the April lineup. The project’s documentation subsequently distinguished the original hybrid release as “Qwen3-2504” from these separately post-trained 2507 variants. (github.com)

Original model lineup and naming

The April 29, 2025 launch identified eight model configurations: six dense models—Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Qwen3-14B, and Qwen3-32B—and two sparse mixture-of-experts (MoE) models, Qwen3-30B-A3B and Qwen3-235B-A22B. In these names, B denotes billions of parameters, rather than model-file size or training-data volume. The numerical labels are rounded size designations: for example, the Qwen3-8B-Base model card records 8.2 billion parameters, rather than exactly eight billion. (qwenlm.github.io)

The MoE names distinguish the total parameter count from the subset activated for each token during inference. Thus, 30B-A3B designates approximately 30 billion total parameters and three billion active parameters; its model card gives the more precise figures of 30.5 billion and 3.3 billion. 235B-A22B identifies 235 billion total parameters and 22 billion activated per token. Both models select eight experts from 128 in each MoE layer. The active count describes sparse computation, not a reduction of the entire checkpoint to three or 22 billion stored parameters. Dense models do not make this expert-selection distinction. (huggingface.co)

Checkpoint names also identify training status. A -Base suffix denotes a pretrained checkpoint, such as Qwen3-8B-Base, whose model card specifies “Pretraining.” The original post-trained checkpoints instead use the unsuffixed names, such as Qwen3-30B-A3B; they were adapted for instruction following and combined thinking and non-thinking operation in one checkpoint. Consequently, the absence of -Instruct in an original Qwen3 name does not identify a base model. Pretraining and post-training are separate checkpoint stages, not additional model sizes or different parameter-count conventions. (huggingface.co)

The table records the original unsuffixed, post-trained lineup. Parameter counts are the nominal values encoded in the names; “native context” means the sequence length reached in pretraining. The launch announcement’s context figures are included separately because its 128K entries should not be read as native training lengths. All eight configurations completed long-context pretraining at 32,768 tokens. (qwenlm.github.io)

Table specifications follow the launch announcement and the original technical report. (qwenlm.github.io)

The distinction between native and extended context windows is explicit in the original MoE model cards: both specify 32,768 tokens natively and 131,072 with YaRN positional scaling. A further documentation difference concerns Qwen3-4B: the April launch table lists 32K, whereas the May 14, 2025 technical report lists 128K. The table above preserves the launch value rather than silently substituting the later report’s specification; neither figure changes the documented 32,768-token pretraining length. (huggingface.co)

Transformer architecture and expert routing

The original Qwen3 models use a decoder-only Transformer architecture: token embeddings pass through successive blocks of causal self-attention and feed-forward computation, with residual connections around these sublayers. A causal mask prevents a position from attending to later positions, so the representation used to predict the next token depends only on the preceding sequence. A final normalization and vocabulary projection produce next-token scores. During autoregressive generation, each selected token is appended to the sequence before the next prediction; the model therefore represents a sequence probability as $p(x_1,\ldots,x_T)=\prod_{t=1}^{T}p(x_t\mid x_{<t})$. This computational structure applies to both the dense and mixture-of-experts variants. (github.com)

Qwen3 uses grouped-query attention (GQA), in which several query heads share each key–value head rather than maintaining a separate key–value pair for every query head. This reduces the key–value cache required during generation while retaining multiple query projections. For example, Qwen3-8B has 32 query heads and eight key–value heads, whereas Qwen3-235B-A22B has 64 query heads and four key–value heads. Rotary positional embeddings, or RoPE, encode position through rotations of query and key vectors before their attention scores are calculated. Qwen3 also applies query–key normalization: separate RMS normalization operations act on each head’s query and key vectors before rotary position encoding. The technical report identifies this addition, together with removal of query, key, and value projection biases, as a change intended to stabilize training. (arxiv.org)

The blocks follow a pre-normalization arrangement, applying RMSNorm before attention and before the feed-forward sublayer. Unlike mean-centering LayerNorm, RMSNorm rescales a vector using its root-mean-square magnitude and a learned scale. The feed-forward network uses SwiGLU: two projections of the input are combined by element-wise multiplication, with a SiLU activation function applied to one branch, before a final projection returns the result to the model’s hidden dimension. In simplified notation, this is $W_{\mathrm{down}}[\operatorname{SiLU}(W_{\mathrm{gate}}h)\odot(W_{\mathrm{up}}h)]$. Dense models apply the same feed-forward network to every token at a given layer; the sparse variants instead route tokens among alternative expert networks. (github.com)

In the original mixture-of-experts (MoE) models, Qwen3-30B-A3B and Qwen3-235B-A22B, each expert bank contains 128 experts, of which eight are selected for each token. A learned router scores the experts from the token’s hidden state, selects the highest-scoring eight, and combines their outputs using routing weights. Selection is token- and layer-dependent rather than a permanent assignment of a whole request to one expert. Unlike Qwen2.5-MoE, these models have no shared expert that processes every token. The developers also report using a global-batch load-balancing loss to regulate expert utilization while encouraging specialization. (arxiv.org)

The eight-of-128 selection makes expert computation sparse, but does not shrink the checkpoint to eight experts. The original flagship contains approximately 235 billion parameters while activating approximately 22 billion per token; attention and other non-expert components still participate alongside the selected experts. Consequently, $8/128=6.25\%$ describes the fraction of experts selected at a layer, not the active fraction of all model parameters or an equivalent reduction in end-to-end latency. Unselected expert weights remain part of the model and must remain available for subsequent routing decisions. The architectural saving is principally reduced per-token expert computation, not elimination of the total weight-storage requirement. (arxiv.org)

Pretraining data and multilingual coverage

The Qwen team disclosed approximately 36 trillion tokens of pretraining data covering 119 languages and dialects for the original Qwen3 generation in 2025, compared with 18 trillion tokens and 29 languages for Qwen2.5. These are developer-reported corpus and coverage figures, rather than independent measurements of linguistic proficiency. The published language inventory includes not only English and Chinese but also languages such as Swahili, Lao, Khmer, and Haitian Creole. Its Chinese coverage includes Mandarin, Cantonese, and several regional varieties. (qwenlm.github.io)

The corpus combined web material with text extracted from documents and model-generated material. Qwen2.5-VL extracted text from PDF documents, after which Qwen2.5 refined the extracted content. Qwen2.5-Math and Qwen2.5-Coder generated additional mathematical and programming material, including textbooks, question–answer pairs, and code snippets. Thus, the expansion involved both recovering existing written content and synthesizing examples in targeted domains, rather than simply collecting more webpages. (qwenlm.github.io)

Quality control included a multilingual annotation system applied to more than 30 trillion tokens. It labeled educational value, subject fields, domains, and safety-related characteristics, supporting filtering and data-mixture selection. The team tested instance-level mixtures using smaller proxy models and ablation experiments, rather than adjusting proportions only at the source or domain level. The technical report also describes trillions of synthetic tokens spanning dozens of domains. These procedures were disclosed as corpus-construction methods, not as guarantees that every retained example was correct. (arxiv.org)

Pretraining followed three stages. S1 used more than 30 trillion tokens at a 4,096-token sequence length to develop language skills and general knowledge. S2 added approximately five trillion tokens while increasing the share of STEM, programming, and reasoning material. The final stage used high-quality long sequences to extend the training context window to 32,768 tokens. This curriculum changed both the content emphasis and, ultimately, the sequence length. (qwenlm.github.io)

Advertised coverage should be distinguished from task-specific evidence. The report’s language-specific evaluations cover 12 languages; its broader Belebele natural-language-understanding evaluation uses 80 supported languages, excluding 42 described as unoptimized. In thinking mode, Qwen3-235B-A22B scored 86.7 on Spanish MMMLU and 83.6 on Arabic MMMLU. These are post-trained checkpoint results, not isolated measurements of pretraining effects. They illustrate why comparisons require a named language, task, checkpoint, and operating mode: inclusion among the 119 languages does not establish equal proficiency, and a knowledge benchmark does not directly measure translation quality or conversational fluency. (arxiv.org)

Post-training and knowledge distillation

The original Qwen3 post-training pipeline developed extended reasoning before integrating direct-response behavior. The Qwen Team’s April 29, 2025 release description identified four successive stages: long chain-of-thought cold start, reasoning-focused reinforcement learning, thinking-mode fusion, and general reinforcement learning. The cold-start stage used fine-tuning on extended solutions in mathematics, programming, logical reasoning, and STEM subjects to establish reasoning patterns before reward-driven optimization. (qwenlm.github.io)

The second stage shifted from imitating worked examples to optimizing responses against verifiable outcomes. Rule-based rewards supplied feedback for reasoning tasks, while increased reinforcement-learning computation supported exploration of alternative solutions. The technical report specifies 3,995 query–verifier pairs and Group Relative Policy Optimization (GRPO). For Qwen3-235B-A22B, the developers reported an AIME 2024 score increase from 70.1 to 85.1 over 170 reinforcement-learning steps. These figures describe that training run rather than a general guarantee of improvement. (qwenlm.github.io)

Thinking-mode fusion then combined long-reasoning examples with ordinary instruction-tuning data through further supervised learning. Its purpose was to retain the reasoning behavior while teaching the same checkpoint to answer without an extended reasoning trace. Instruction examples supplied demonstrations of desired responses, whereas the subsequent general reinforcement-learning stage optimized behavior across more than 20 tasks, including instruction following, formatting, and agent capabilities. Thus, response imitation and reward-based optimization served different functions within the pipeline. (qwenlm.github.io)

General reinforcement learning used three reward mechanisms. Rules checked objectively assessable requirements; Qwen2.5-72B-Instruct scored responses against reference answers; and a reward model trained on human preference data scored open-ended responses without reference answers. The preference-related objectives addressed helpfulness, engagement, and style, complementing correctness-oriented and formatting rewards rather than replacing them. (arxiv.org)

The smaller checkpoints followed strong-to-weak distillation rather than independently repeating all four stages. This route covered Qwen3-0.6B, 1.7B, 4B, 8B, and 14B, plus Qwen3-30B-A3B. Initial off-policy distillation used teacher-generated responses in both thinking modes. On-policy distillation subsequently used responses generated by the student itself, aligning its token-prediction logits with those of Qwen3-32B or Qwen3-235B-A22B by minimizing Kullback–Leibler divergence. This transferred teacher guidance on the student’s own generated sequences, not only on a fixed collection of teacher answers. The developers reported approximately one-tenth of the GPU-hours of the four-stage approach in their comparison. (arxiv.org)

Thinking modes and inference-time computation

The original post-trained Qwen3 checkpoints, released on April 29, 2025, combine thinking and non-thinking behavior within the same model weights. Thinking mode generates intermediate reasoning before an answer, whereas non-thinking mode answers without that extended output. The developers present this as a way to allocate inference-time computation according to task difficulty: mathematical or programming problems can receive additional reasoning, while routine conversation can use the faster mode. This distinction concerns how a checkpoint generates its response, rather than a choice between separately loaded models. (github.com)

Mode selection has two levels. In the original checkpoints’ chat template, tokenizer.apply_chat_template(..., enable_thinking=True) enables thinking by default; setting enable_thinking=False is a hard switch that suppresses generated thinking content. When thinking remains enabled, /think and /no_think in a user or system message provide a softer prompt-level control, with the most recent instruction governing subsequent responses in a multi-turn conversation. For example, a conversation can request a direct answer with /no_think, then request reasoning on the next turn with /think. These textual instructions cannot override enable_thinking=False. With the soft non-thinking instruction, an empty thinking block may remain in the output; with the hard switch, the model generates neither thinking content nor that block. (huggingface.co)

Thinking output and the final answer occupy distinct parts of the response, conventionally separated by <think> and </think>. Applications can split the generated sequence at the closing delimiter rather than displaying the entire sequence as the answer. Inference-server reasoning parsers automate this separation: the project’s examples use --reasoning-parser qwen3 for the original checkpoints in both vLLM and SGLang, allowing intermediate text and answer text to be exposed separately through the API, commonly as reasoning_content and content. Parsing changes the representation returned to surrounding software; it is not itself the control that determines whether the model generates reasoning. (github.com)

A thinking budget limits the intermediate generation rather than simply selecting between two modes. The technical report describes stopping generation when the thinking portion reaches a user-defined threshold, inserting an instruction to produce a solution from the reasoning already available together with a closing delimiter, and then continuing with the final response. This differs from merely exhausting a total output-token limit, which can stop generation before an answer is completed. In the developers’ experiments, Figure 2 reports progressively improving benchmark performance for Qwen3-235B-A22B as the thinking budget increases across four evaluations covering mathematics, coding, and STEM. Additional generated tokens entail additional computation and delay before the final answer, creating a latency–performance trade-off. These aggregate results do not establish that a longer response is correct on any particular problem; the report also treats extending output beyond 32K tokens as future exploration, not an experimentally established result. (arxiv.org)

The Qwen3-2507 releases separate these behaviors into different checkpoints. Qwen3-Instruct-2507 is non-thinking-only, produces no thinking blocks, and no longer requires enable_thinking=False. Qwen3-Thinking-2507 is thinking-only; its default chat template already supplies the opening <think>, so newly generated output may contain </think> without a generated opening tag. The project’s serving examples accordingly specify deepseek_r1 in vLLM and deepseek-r1 in SGLang for these Thinking checkpoints, rather than the original qwen3 parser. The April checkpoints’ hybrid switching controls therefore cannot be assumed to apply uniformly to models carrying the later Instruct-2507 or Thinking-2507 suffixes. (github.com)

Context windows and long-document processing

Qwen3’s context window is measured in tokens rather than words or pages. The relevant input length is the sequence produced by tokenization after chat formatting, including instructions, conversation history, and document text. Input capacity, output limits, and total sequence capacity are distinct: max_new_tokens limits newly generated tokens, not the prompt, while the context budget covers both input and generated output. Generated thinking therefore consumes space alongside the final answer. For a deployment with total capacity $C$, an input of $I$ tokens leaves at most $C-I$ tokens for generation, subject to any lower output limit. Qwen’s examples separately tokenize the formatted prompt and set max_new_tokens; the original Qwen3-8B model card explicitly defines long-context length as including both input and output. (qwen.readthedocs.io)

The April 29, 2025 launch specifications assigned 32K contexts to Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B, and 128K contexts to Qwen3-8B, Qwen3-14B, Qwen3-32B, Qwen3-30B-A3B, and Qwen3-235B-A22B. These advertised capacities should not be read as identical native training lengths: the technical report states that all original models underwent long-context pretraining at 32,768 tokens. Its later architecture table also lists Qwen3-4B as 128K, unlike the launch table’s 32K specification. Thus, release-date specifications and subsequently documented extrapolation capabilities need to remain distinguishable. (qwenlm.github.io)

For the original Qwen3-8B checkpoint, the model card specifies 32,768 tokens natively and 131,072 with YaRN, a method for extending rotary positional encodings beyond their original range. Its documented configuration sets rope_type to yarn, factor to 4.0, and original_max_position_embeddings to 32768; the vLLM example additionally sets --max-model-len 131072. Increasing the server’s length limit alone is therefore not the documented extension procedure. The card also cautions that static YaRN applies the same scaling regardless of actual input length and can reduce short-text performance. (huggingface.co)

Qwen3-2507 raised the documented context class to 256K; for example, Qwen3-30B-A3B-Instruct-2507 specifies 262,144 tokens natively. On August 8, 2025, the project announced approximately one-million-token support specifically for the Instruct-2507 and Thinking-2507 variants of both MoE sizes, 30B-A3B and 235B-A22B. Their model cards require replacing config.json with the supplied config_1m.json, which configures length extrapolation and sparse attention. The extension combines Dual Chunk Attention with MInference sparse attention. Documented serving examples enable the Dual Chunk Flash Attention backend, set the sequence limit to 1,010,000, and use chunked prefill; the 235B Instruct example additionally specifies eight-way tensor parallelism and one concurrent sequence. These are checkpoint- and backend-specific configurations, not a universal setting for every Qwen3 model. (github.com)

Long-context evaluation tests whether information remains usable, rather than merely whether software accepts a sequence. The original technical report uses RULER, whose tasks extend beyond single-item retrieval to multi-hop tracing, aggregation, and question answering. With fourfold YaRN scaling, the developers reported Qwen3-235B-A22B non-thinking scores of 97.7 at 4K and 90.6 at 128K; thinking-mode evaluation, capped at 8,192 thinking tokens, scored 86.0 at 128K. The updated 30B-A3B Instruct card reports a one-million-token RULER variant using 260 examples per length: its full-attention configuration scored 98.0 at 4K, 82.5 at 256K, and 72.8 at 1000K, with Dual Chunk Attention enabled. These length-dependent results demonstrate why maximum accepted length is not equivalent to uniform retrieval accuracy or reliable understanding of arbitrary long documents. (arxiv.org)

Capabilities and benchmark evaluation

Mathematical reasoning, scientific knowledge, programming, instruction following, multilingual tasks, and tool use are evaluated separately in the Qwen team’s reports. Mathematical tests include the American Invitational Mathematics Examination (AIME), whose 2024 and 2025 evaluations each contain 30 competition problems, and MATH-500, a 500-problem mathematical evaluation set. Scientific evaluation uses GPQA, introduced by David Rein and colleagues in 2023: expert-written multiple-choice questions in biology, physics, and chemistry. The original Qwen3 report specifies the GPQA-Diamond subset, rather than treating all GPQA variants as interchangeable. These are answer-scoring tasks, not assessments of original scientific research. (arxiv.org)

Programming evaluations include LiveCodeBench, which collects contest problems over time from platforms including LeetCode, AtCoder, and Codeforces. Its changing problem pool makes the version and collection interval essential: the original Qwen3 evaluation used version 5, covering October 2024–February 2025, whereas the July model cards use version 6, covering February–May 2025. Consequently, the original flagship’s reported 70.7 on version 5 cannot be directly compared with a July checkpoint’s version-6 score as though the questions were identical. LiveCodeBench also distinguishes code generation from capabilities such as self-repair, execution reasoning, and test-output prediction. (arxiv.org)

IFEval, introduced by Jeffrey Zhou and colleagues in 2023, measures compliance with automatically checkable instructions, such as word-count requirements or repeated keyword inclusion. Its strict prompt-level metric requires all applicable constraints in a prompt to be satisfied; it is therefore distinct from open-ended preference judgments. Multilingual evaluation includes Multi-IF for instruction following in eight languages, MMMLU for knowledge across 14 languages, and mathematical and logical-reasoning tests. Tool use is represented by the Berkeley Function Calling Leaderboard, BFCL v3; the original report specifies function-calling-format evaluation and a 64K context configuration for multi-turn tests. These evaluations measure different components of an AI agent rather than a single general-purpose capability. (arxiv.org)

For reproducibility, a comparison needs the exact checkpoint, reasoning mode, sampling settings, output ceiling, benchmark version, evaluation date, and whether the result is developer-reported or independently reproduced. In the report dated May 14, 2025, thinking-mode sampling used temperature 0.6, top-p 0.95, and top-k 20. Output limits were 32,768 tokens, increased to 38,912 for AIME; AIME accuracy was averaged over 64 samples per question, not scored as “correct if any attempt succeeds.” The July Thinking-2507 card specifies 81,920 output tokens for difficult reasoning and coding evaluations and 32,768 otherwise. Publication and release dates identify the reporting period, but should not be substituted for an explicitly recorded execution date. (arxiv.org)

Selected developer-reported comparisons illustrate the July changes without establishing an undated ranking. Qwen3-235B-A22B-Thinking-2507, released July 25, 2025, is compared in its model card with the original Qwen3-235B-A22B in thinking mode: AIME 2025 rises from 81.5 to 92.3, LiveCodeBench v6 from 55.7 to 74.1, and IFEval from 83.4 to 87.8. The July 21 non-thinking release, Qwen3-235B-A22B-Instruct-2507, is compared with the original checkpoint’s non-thinking mode: AIME 2025 rises from 24.7 to 70.3 and IFEval from 83.2 to 88.7. Improvement is not uniform: the same Instruct table reports Aider-Polyglot declining from 59.6 to 57.3. These figures describe named checkpoints under the developers’ evaluation protocols, including their differing computation budgets; they are not independent replications or fixed-budget measurements of improvement. (github.com)

Deployment, customization, and tool integration

Qwen3 checkpoints can be executed directly through Hugging Face Transformers or incorporated into a separate inference service. The project’s Transformers examples load weights with AutoModelForCausalLM, load the associated tokenizer, apply its chat template, and generate a continuation; the documented minimum Transformers version is 4.51.0. For service deployment, vLLM and SGLang expose OpenAI-compatible APIs, separating model execution from client applications. Local pathways include llama.cpp, which runs GGUF-format models through command-line and server interfaces, and Ollama, which manages model downloads and a local service. A checkpoint identifier such as Qwen/Qwen3-8B identifies the model artifact, whereas an Ollama tag or GGUF filename also reflects packaging, conversion, and potentially quantization. These distributions therefore need not have identical numerical precision or runtime defaults. (github.com)

Deployment capacity depends on weight precision, inference kernels, accelerator memory, sequence length, and concurrent workload—not simply the parameter count. Qwen’s published inference measurements illustrate the effect of quantization: for Qwen3-8B on one NVIDIA H20 with 96 GB of memory, Transformers 4.51.3, batch size one, a one-token input, and 2,048 generated tokens, reported GPU memory use was 15,947 MB in BF16, 9,323 MB in FP8, and 6,177 MB in AWQ-INT4. With a 30,720-token input and the same generation length, BF16 memory use increased to 20,267 MB and AWQ-INT4 to 11,001 MB. These are measurements for named configurations, not universal minimum hardware requirements. The same benchmark also demonstrates that lower precision does not automatically produce higher speed: its short-input FP8 Transformers result was slower than BF16. (qwen.readthedocs.io)

Context-cache storage is another deployment variable: vLLM’s documentation identifies PagedAttention as its mechanism for managing attention keys and values, alongside continuous batching and tensor parallelism. Batching schedules multiple requests together, while tensor parallelism distributes model computation across accelerators. In Qwen’s SGLang 0.4.6.post1 benchmark, Qwen3-235B-A22B used eight H20 accelerators with tensor-parallel degree eight in BF16, compared with four accelerators and degree four in FP8. SGLang preallocated GPU memory, with mem_fraction_static set to 0.85, so the benchmark did not report its memory footprint as though it were directly comparable to Transformers’ measured allocation. (qwen.readthedocs.io)

Customization uses supervised fine-tuning on instruction–response or conversation examples, either updating all model parameters or employing parameter-efficient adaptation. The official Qwen training documentation includes a Qwen3-specific MS-SWIFT guide covering full-parameter training, LoRA, QLoRA, and distributed execution. Its Qwen3-8B LoRA example uses BF16, adapter rank eight, a maximum training sequence length of 2,048 tokens, and gradient accumulation over 16 steps; the guide reports 22 GB of GPU memory for that particular example. Adapter weights can subsequently be merged into the underlying model for export. These downstream procedures are distinct from the developers’ original pretraining and post-training pipeline. (qwen.readthedocs.io)

Tool integration adds an execution framework around the model. Qwen-Agent added a Qwen3 tool-calling demonstration and Model Context Protocol cookbooks on May 1, 2025; its MCP configuration examples connect memory, filesystem, and SQLite servers. Tools are described through names, descriptions, and JSON Schema parameter definitions. Qwen3 generates a requested function name and arguments, which the framework parses into structured fields; surrounding application code then invokes the function and returns its result as another conversation message. For example, the official function-calling guide supplies temperature-query functions and explicitly shows the application decoding arguments, executing the selected function, and appending its output before further generation. A generated tool request is thus not itself an executed action: the external software supplies access to the actual service, filesystem, database, or interpreter. (github.com)

Specialized branches and architectural extensions

Qwen3-Coder specializes in coding agents and repository-level software engineering rather than only generating isolated functions. Its July 22, 2025 launch introduced Qwen3-Coder-480B-A35B-Instruct, a mixture-of-experts (MoE) model with 480 billion total parameters and 35 billion active parameters per token. The developers reported pretraining on 7.5 trillion tokens, approximately 70% of them code, followed by execution-driven reinforcement learning and long-horizon training involving repeated tool use and environmental feedback. Its native 256K-token context was intended to accommodate repository-scale material, including pull requests. Released alongside it, Qwen Code is a separate command-line agent adapted from Gemini CLI, with prompts, parsers, and function-calling support tailored to the coding models; it is application software surrounding the model, not another checkpoint. (qwenlm.github.io)

Qwen3-Embedding and Qwen3-Reranker adapt the dense Qwen3 foundations to information retrieval, with 0.6B-, 4B-, and 8B-parameter versions of each. Embedding models encode queries and documents into numerical representations whose similarity supports retrieval; their documented maximum vector dimensions are 1,024, 2,560, and 4,096, respectively, with configurable smaller dimensions. Rerankers instead jointly examine a query and a candidate document, producing a relevance score from the relative probabilities of “yes” and “no.” The distinction permits a two-stage search pipeline: embeddings select candidates, and reranking reorders that smaller set. Both series document 32K-token sequence lengths. Their principal outputs are representations or relevance judgments, not ordinary conversational answers, although they can supply documents to a retrieval-augmented generation system. (github.com)

Qwen3-VL extends the family into image and video understanding. Its first 235B-A22B Instruct and Thinking checkpoints were released on September 23, 2025, followed by smaller dense and MoE variants. The architecture introduces DeepStack, which integrates visual features from multiple vision-transformer levels, and interleaved multidimensional positional encoding covering time, width, and height. Timestamp-grounded video processing supports locating events within a sequence rather than treating video solely as unrelated still images. Documented applications include reading text in images, interpreting long documents, spatial reasoning, and generating code from visual inputs. These are vision-language input capabilities with generated textual responses, distinct from a speech-generation pipeline. (github.com)

Qwen3-Omni, released on September 22, 2025, accepts text, images, audio, and video and adds streaming speech output. Its MoE-based Thinker–Talker design separates multimodal interpretation and textual response generation from speech production. Output modalities depend on the checkpoint: Qwen3-Omni-30B-A3B-Instruct includes both components and supports text and audio output, whereas the Thinking checkpoint contains the Thinker and produces text. A separately fine-tuned Captioner generates detailed textual descriptions of audio. Thus, the Omni designation does not mean that every checkpoint generates speech, and its audio processing distinguishes it from Qwen3-VL’s image-and-video specialization. (github.com)

Qwen3-Next changes the computational architecture rather than merely specializing the original model’s training. The 80B-A3B models have approximately 80 billion total parameters but activate about 3 billion per token. Their 48-layer layout repeats three Gated DeltaNet layers followed by one gated full-attention layer, combining recurrent linear-attention processing with periodic full attention. The sparse feed-forward component selects ten of 512 routed experts per token alongside one shared expert. Separate Instruct and Thinking releases apply this hybrid design, whose configuration differs from the original Qwen3 dense and MoE architectures. (huggingface.co)

Checkpoint names also remain distinct from hosted-product identifiers. For example, the Coder launch documentation uses qwen3-coder-plus for a hosted API, while naming the downloadable flagship Qwen3-Coder-480B-A35B-Instruct. Later decimal-numbered generations likewise require separate architectural descriptions: Qwen3.5’s February 15, 2026 announcement introduced a natively multimodal 397B-A17B model combining Gated Delta Networks with sparse MoE computation, rather than simply renaming an original Qwen3 checkpoint. (qwenlm.github.io)

Licensing, transparency, and documented limitations

The initial Qwen3 models were distributed under the Apache License 2.0, with downloadable weights and publicly available inference implementations and examples. The release therefore permits examination and local execution of particular checkpoints, rather than access exclusively through a hosted service. This availability is distinct from complete training transparency: the technical report describes approximately 36 trillion pretraining tokens, data-processing methods, and staged training, but the release does not provide the complete training corpus or an executable reconstruction of the original training infrastructure. Downloading a checkpoint supports reproduction of its inference behavior; it does not by itself make independent reproduction of its training possible. (arxiv.org)

The project supplies several concrete materials relevant to reproducibility. For example, the Qwen3-8B repository contains a model card, config.json, generation_config.json, five SafeTensors weight shards and their index, and tokenizer resources including tokenizer.json, tokenizer_config.json, vocab.json, and merges.txt. These files expose the architecture configuration, default generation settings, and text-encoding resources needed to load the checkpoint. The model card adds chat-template examples, thinking-output extraction, sampling settings, and deployment instructions. It also documents a specific compatibility failure: Transformers versions below 4.51.0 produce KeyError: 'qwen3'. The technical report complements these artifacts with training-method descriptions, benchmark definitions, comparison tables, and ablation experiments, rather than a complete end-to-end training recipe. (huggingface.co)

Prompt and output handling are checkpoint-dependent. Original hybrid checkpoints expose enable_thinking through their chat templates; Qwen3-Instruct-2507 is non-thinking-only, whereas Qwen3-Thinking-2507 is thinking-only. The latter’s template inserts the opening <think> delimiter, so generated text may contain a closing </think> without generating the opening tag itself. The README’s server examples likewise distinguish parsers: original Qwen3 uses qwen3, while Thinking-2507 uses deepseek-r1 in SGLang and deepseek_r1 in vLLM. These examples establish that a parser configuration cannot be assumed interchangeable across releases. (github.com)

The README also documents a multi-step tool-use caveat in serving configurations that discard reasoning_content: removing that field can degrade thinking-model tool interactions. Its documented workaround preserves the unextracted output for processing by the chat template. Packaging introduces another identification problem. The project records that, as of August 2025, Ollama’s qwen3:30b-a3b alias pointed to qwen3:30b-a3b-thinking-2507-q4_K_M, not necessarily the original hybrid checkpoint suggested by the shorter name. Thus, aliases alone do not establish checkpoint identity or weight precision. (github.com)

The published evaluation evidence has a narrower scope than universal reliability. The report’s mathematics, coding, knowledge, instruction-following, and multilingual results measure performance on specified tasks; they do not establish correctness for every application or language. Its distinction between ordinary scores and multi-sample measures such as pass@64 also matters: success among many generated attempts is not equivalent to a correct single response. Generated thinking text is not a formal correctness certificate. Likewise, the report’s use of safety-related training-data annotations describes a filtering mechanism, not proof of deployment safety or absence of bias; such claims require separately identified evaluations with explicit datasets, threat models, and scoring procedures. (arxiv.org)

References

1. GitHub - QwenLM/Qwen3: Qwen3 is the large language model series developed by Qwen team, Alibaba Cloud.github.com
2. Qwen3/README.md at main · QwenLM/Qwen3 · GitHubgithub.com
3. Alibaba Introduces Qwen3, Setting New Benchmark in Open-Source AI with Hybrid Reasoningalibabagroup.com
4. Qwen3: Think Deeper, Act Fasterqwenlm.github.io
5. Qwen3 Technical Reportarxiv.org
6. Qwen3 Technical Reportarxiv.org
7. Qwen/Qwen3-235B-A22B-Instruct-2507 · Hugging Facehuggingface.co
8. Qwen/Qwen3-235B-A22B-Thinking-2507 · Hugging Facehuggingface.co
9. Qwen/Qwen3-8B-Base · Hugging Facehuggingface.co
10. Qwen/Qwen3-30B-A3B · Hugging Facehuggingface.co
11. Qwen/Qwen3-235B-A22B · Hugging Facehuggingface.co
12. transformers/src/transformers/models/qwen3/modeling_qwen3.py at main · huggingface/transformers · GitHubgithub.com
13. transformers/src/transformers/models/qwen3_moe/modeling_qwen3_moe.py at main · huggingface/transformers · GitHubgithub.com
14. Qwen3：思深，行速qwenlm.github.io
15. Qwen/Qwen3-8B · Hugging Facehuggingface.co
16. Quickstart - Qwenqwen.readthedocs.io
17. YaRN: Efficient Context Window Extension of Large Language Modelsarxiv.org
18. Qwen/Qwen3-30B-A3B-Instruct-2507 · Hugging Facehuggingface.co
19. RULER: What's the Real Context Size of Your Long-Context Language Models?arxiv.org
20. GPQA: A Graduate-Level Google-Proof Q&A Benchmarkarxiv.org
21. Instruction-Following Evaluation for Large Language Modelsarxiv.org
22. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codearxiv.org
23. GitHub - QwenLM/Qwen3github.com
24. Speed Benchmark - Qwenqwen.readthedocs.io
25. vLLM - Qwenqwen.readthedocs.io
26. MS-SWIFT - Qwenqwen.readthedocs.io
27. GitHub - QwenLM/Qwen-Agentgithub.com
28. Function Calling - Qwenqwen.readthedocs.io
29. Qwen3-Coder: Agentic Coding in the Worldqwenlm.github.io
30. QwenLM/Qwen3-Embeddinggithub.com
31. Qwen3-VL/README.mdgithub.com
32. QwenLM/Qwen3-Omnigithub.com
33. Qwen/Qwen3-Next-80B-A3B-Instructhuggingface.co
34. Qwen3.5: Towards Native Multimodal Agentsqwen.ai
35. Qwen/Qwen3-8B at mainhuggingface.co
36. Qwen/Qwen3-8B · Hugging Facehuggingface.co