aiwiki.page
English

Apache Spark

Apache Spark is an open-source distributed computing engine for large-scale data processing, SQL analytics, streaming, and machine learning.

30 keywords20 linked from8 not yet writtenWritten by AI
Open-Source Soft…SQLMachine LearningBig dataDistributed Comp…MapReduceApache HadoopDirected acyclic…Apache Spa…

Apache Spark is an open-source engine for processing data across clusters of computers. It provides a common execution system for batch transformations, SQL queries, streaming computations, and machine learning, with programming interfaces for Scala, Java, Python, and R. Spark is primarily a computation engine rather than a storage system: it reads data from external sources, distributes work among processors, and writes or returns the results.

Spark became important in big data computing by making repeated and interactive calculations more practical than systems organized around a succession of independent, disk-oriented processing jobs. Its principal innovations include recoverable distributed collections, lazy execution, reusable intermediate results, and an execution engine shared by several higher-level libraries. Although often described as an “in-memory” system, Spark can process datasets larger than aggregate memory and commonly relies on disk, network transfers, and external storage.

Originally developed at the University of California, Berkeley, Spark became a project of the Apache Software Foundation and developed into an infrastructure component for data engineering and analytical platforms. Its architecture combines ideas from distributed computing, relational query optimization, and parallel numerical processing. Understanding its behavior requires distinguishing its programming abstractions from the physical work performed by the cluster.

Origins and development

Spark originated in 2009 in Berkeley’s AMPLab, with Matei Zaharia as a principal creator. Its early motivation was the difficulty of expressing iterative algorithms and interactive analysis efficiently using MapReduce. In the implementation associated with Apache Hadoop, a complex analysis often required several jobs, with intermediate results written to distributed storage between them. This arrangement offered a straightforward recovery model but imposed substantial overhead when an algorithm repeatedly revisited the same data.

Iterative machine learning provided a particularly clear example. An optimization procedure might scan a training dataset dozens of times while updating a relatively small parameter vector. Repeatedly loading the same records and materializing every intermediate stage could dominate useful computation. Spark instead allowed distributed datasets to be retained and reused, while recording enough information to reconstruct missing partitions after a failure. Its early research presentations emphasized both iterative computation and interactive querying.

The project was released as open-source software in 2010. A 2012 research paper, “Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing,” explained the recovery abstraction underlying its original programming model. Spark entered the Apache incubator in 2013 and became a top-level Apache project in 2014. Databricks, founded in 2013 by members of the Berkeley team, became a major commercial contributor, but Apache Spark itself continued under Apache’s community governance and licensing.

Successive releases broadened Spark beyond its initial distributed-collection interface. Spark SQL and DataFrames made structured data processing increasingly central. Spark 2.0, released in 2016, introduced the SparkSession entry point and Structured Streaming, while consolidating the structured APIs. Spark 3.0, released in 2020, included major advances in adaptive query execution and dynamic partition pruning. Spark 3.4, released in 2023, introduced Spark Connect, separating client applications from the server-side execution environment through a remote protocol.

The development trajectory therefore involved more than accelerating an existing processing pattern. Spark evolved from a general distributed collection framework into a platform whose most optimized execution paths operate on structured data. Earlier interfaces remain relevant, but many applications now express transformations declaratively and leave operator selection to the engine. This evolution also explains why performance comparisons based solely on early MapReduce benchmarks do not characterize the full system: modern Spark applications may involve SQL optimization, Python interoperability, continuous ingestion, and external transactional table layers.

Application architecture and execution

A conventional Spark application consists of a driver process and a set of executor processes. The driver runs the application’s coordinating logic, constructs execution plans, and schedules distributed work. Executors perform tasks, retain cached data, and report results and status. A cluster manager allocates resources to applications; it does not replace Spark’s internal scheduling of individual tasks.

Spark can run with its own standalone cluster manager, with Hadoop YARN, or on Kubernetes. These deployment systems differ in resource allocation, process lifecycle, and integration with surrounding infrastructure. In client deployment mode, the driver runs in the process that submits the application; in cluster deployment mode, the infrastructure launches the driver within the managed environment. Driver placement matters because the driver must communicate reliably with executors and because its failure can terminate the application.

Execution is organized around jobs, stages, and tasks. An action, such as counting records or writing a result, normally triggers a job. Spark represents the work through a directed acyclic graph of dependencies. Operations that can run together on the same partition are often pipelined into one stage. An operation requiring records to be redistributed across executors, such as grouping by a key, generally introduces a shuffle boundary and therefore a new stage. Within a stage, tasks usually correspond to individual data partitions.

Suppose an application reads transaction records, filters invalid rows, and calculates total expenditure by customer. Reading, filtering, and computing local partial totals can often occur within the same stage. Records or partial aggregates must then be redistributed so that values for the same customer reach the appropriate downstream partition. The final aggregation runs after that exchange. The network movement and intermediate storage involved in this shuffle may cost more than the arithmetic itself.

Spark’s scheduling machinery tracks completed work and retries failed tasks. Scheduling can favor executors near input data when the storage system exposes meaningful locality information. On object storage, locality is usually less important than request concurrency, network throughput, and file layout. Executors can be added or removed through dynamic allocation when the deployment and shuffle-preservation mechanisms support it.

The driver is a coordinator, not a central repository for all records. Operations that return an entire distributed dataset to the driver can exhaust its memory even when the executors have ample capacity. Likewise, distributed processing does not eliminate central planning overhead: applications with enormous numbers of tiny tasks or exceptionally complex plans can overwhelm coordination before exhausting the cluster’s computational resources.

Resilient distributed datasets and fault recovery

Spark’s original core abstraction is the resilient distributed dataset, or RDD: an immutable collection divided into partitions that can be processed in parallel. RDDs can be created from external data or derived from other RDDs. Their immutability simplifies reasoning about concurrent tasks because a transformation produces a new logical dataset rather than modifying an existing one in place.

RDD operations are divided into transformations and actions. Transformations, including map, filter, and flatMap, describe new datasets. Actions, including count, collect, and output operations, request a result or an external effect. Transformations are generally lazy: Spark records their dependencies without immediately processing every record. Execution begins when an action requires the corresponding data. This separation allows operations to be pipelined and avoids computing unused results.

The recovery mechanism is based on lineage, the chain of transformations that produced a dataset. If an executor loses a cached partition, Spark can reconstruct it by rerunning the relevant transformations on its ancestors. Narrow dependencies, in which a child partition depends on a limited set of parent partitions, generally permit relatively localized recomputation. Wide dependencies, such as those introduced by repartitioning or grouping, may require shuffle data from many upstream tasks.

Persistence is explicit. An application can request that an RDD be cached or persisted at a specified storage level, potentially using memory, disk, serialized representations, or replication. Marking an RDD for persistence does not immediately populate its cache; partitions are stored as execution first computes them. When available memory is insufficient, the chosen storage level and memory-management behavior determine whether partitions are evicted, placed on disk, or recomputed later.

Checkpointing serves a different purpose from ordinary caching. A reliable checkpoint materializes data in durable storage and can truncate a long dependency chain. This is useful when repeated transformations would otherwise produce expensive recovery paths or excessive lineage metadata. Local checkpointing uses executor storage and offers weaker durability. Neither mechanism makes arbitrary external actions transactional.

Fault tolerance also depends on how user code behaves. A failed or speculative task may be executed more than once, so sending a payment request or incrementing an external counter inside a task can produce duplicate effects. Deterministic transformations over stable input are much easier to reconstruct than operations whose results depend on mutable external services.

RDDs remain useful for low-level transformations and unusual data structures, but their flexibility limits automatic optimization. To the engine, an arbitrary function over language objects may be opaque. Structured APIs expose schemas and expressions, allowing Spark to optimize both the computation and its physical representation more extensively.

Spark SQL, DataFrames, and query optimization

Spark SQL provides relational processing over structured and semi-structured data. Its main programming abstraction is the DataFrame, a distributed collection with named columns and a schema. A DataFrame is not necessarily an already materialized table: it can represent a deferred query plan over files, external tables, or earlier transformations. SQL statements and DataFrame expressions commonly feed into the same optimizer and execution engine.

The Dataset interface adds typed objects in Scala and Java. In those languages, a DataFrame is represented as a Dataset of rows, while typed Datasets use encoders to translate between language-level values and Spark’s internal representation. Python and R expose DataFrame interfaces but not the same compile-time typed Dataset model. Thus, similarly named operations across languages do not imply identical type-checking or execution costs.

Catalyst, Spark SQL’s optimization framework, transforms a query through analysis, logical optimization, and physical planning. Analysis resolves column names, functions, and data types. Logical rules can simplify expressions, push filters toward data sources, eliminate unnecessary columns, and restructure parts of a query. Physical planning chooses executable operators, including different join strategies. Statistics about row counts, data sizes, and column distributions can influence these choices.

For example, joining a large fact table with a small reference table may use a broadcast hash join. Spark distributes the smaller relation to executors so that the larger input need not be shuffled for that join. If both inputs are large, a sort-merge join may be more appropriate. These alternatives have different requirements for network transfer, sorting, and memory. The logically identical SQL query can therefore produce substantially different resource usage depending on its physical plan.

The Tungsten initiative improved execution through compact data representations, reduced object overhead, and generated code. Whole-stage code generation can fuse compatible operators into generated Java code, reducing the cost of repeatedly dispatching through general-purpose operator interfaces. These optimizations are particularly effective for built-in expressions whose semantics the engine understands. An arbitrary user-defined function may prevent some optimizations or introduce extra conversion costs.

Adaptive query execution adjusts selected decisions using statistics gathered during execution. It can coalesce small shuffle partitions, change eligible join strategies, and mitigate certain skewed joins by splitting oversized partitions. It does not remove all consequences of poor data layout or uneven key frequencies.

Spark SQL implements important relational behaviors, including null handling, aggregate functions, and window functions. Type coercion, timestamp interpretation, case sensitivity, and error behavior can depend on configuration and release. Consequently, compatibility with another database management system involves more than accepting similar SQL syntax: expression semantics and supported operations must also match.

Structured Streaming and incremental computation

Structured Streaming expresses streaming computation through the structured query model. Conceptually, an incoming stream behaves like a table that gains rows over time, and a query incrementally updates a result derived from those rows. The same DataFrame-style expressions used for batch processing can therefore describe many streaming operations, although streaming imposes restrictions needed to make incremental execution meaningful.

Its principal execution model is micro-batching. Spark repeatedly identifies newly available input, builds a bounded batch, executes the corresponding computation, and records progress. Sources such as Apache Kafka expose offsets that identify positions in the input. A checkpoint directory records information required to resume a query, including progress and state-related metadata. Checkpoint compatibility places limits on how a running query can be changed and then restarted from its existing checkpoint.

Stateful operations retain information between batches. A running aggregation might store a count for each key; a stream-stream join may retain unmatched records from both sides. Without a policy for discarding old information, state can grow indefinitely. Event-time watermarks provide one mechanism for limiting this growth. A watermark relates observed event times to an allowed lateness threshold, enabling eligible operators to finalize results or remove state.

Event time is distinct from processing time. A sensor reading generated at 12:00 may not arrive until 12:07 because of a network interruption. A five-minute event-time window must assign that reading according to its timestamp rather than merely the time Spark received it. Watermarks accommodate some delayed arrival, but they do not promise that every arbitrarily late record will be incorporated. The precise behavior depends on the operator, its state, and the progress of the watermark.

Output modes describe how results are emitted. Append mode outputs newly finalized rows, update mode outputs changed rows, and complete mode outputs the full result table where supported. A streaming aggregation may require a watermark before its final rows can be appended. The appropriate mode therefore follows from the query’s semantics, not simply a preference about output volume.

“Exactly once” requires careful qualification. Spark’s replayable inputs, checkpointed progress, and compatible sinks can support exactly-once results for particular pipelines. They do not guarantee that every external side effect happens once. A foreachBatch callback, for example, needs an appropriate transactional or deduplication strategy if repeated execution would otherwise duplicate writes. Kafka output also requires attention to possible duplicates.

Structured Streaming also introduced a separate continuous processing mode with different latency and delivery characteristics and a narrower set of supported operations. Micro-batch processing remains architecturally distinct from systems that execute all operators continuously on individual events; trigger intervals, scheduling overhead, and checkpoint work influence its achievable latency.

Machine learning and graph processing

Spark’s machine-learning facilities are grouped under MLlib. They provide distributed implementations of common algorithms and utilities for data preparation, model fitting, and evaluation. The DataFrame-based API, under spark.ml, became the principal development interface, while the older RDD-based spark.mllib API entered maintenance mode. The shared name can obscure this distinction between the library as a whole and its earlier package.

A central abstraction is the pipeline. Transformers convert one DataFrame into another, while estimators learn fitted models from data. A fitted model is itself a transformer that can generate predictions or transformed features. For example, a text-classification pipeline can tokenize text, construct numerical features, and fit logistic regression. Pipeline composition helps preserve the relationship between preprocessing and prediction, rather than leaving those operations as unrelated application code.

Supported methods include linear and logistic regression, decision trees, random forests, gradient-boosted trees, and k-means clustering. Alternating least squares supports collaborative filtering for recommender systems. Feature-processing utilities include categorical indexing, vector assembly, scaling, and several text representations. Cross-validation and train-validation splitting support model selection, although repeated fitting can multiply cluster costs substantially.

Distributed learning is most useful when the data or intermediate computation benefits from partitioning. It is not automatically faster for small datasets. Many algorithms repeatedly scan partitions, compute local statistics or gradients, and combine them through communication. As a result, network exchanges, synchronization, serialization, and uneven partition sizes can limit scaling. Some models also require substantial driver-side metadata or impose constraints on feature representations.

Correct data preparation remains separate from execution scale. Fitting a vocabulary, category mapping, or scaling transformation on held-out data can introduce data leakage even when the subsequent model training is distributed correctly. Pipeline-based workflows can help keep fitted preprocessing within each training split, but the actual arrangement of fitting and evaluation determines whether leakage occurs.

Spark is also used around deep-learning systems, especially for preparing data and coordinating large preprocessing workloads. It is not itself a direct replacement for GPU-oriented frameworks such as PyTorch. Distributed execution of a DataFrame transformation and synchronized training of a neural network are different computational problems.

GraphX provides graph-oriented abstractions over Spark’s execution infrastructure. It represents vertex properties and edges using distributed collections and supports operations such as message aggregation and iterative graph computation. Algorithms such as PageRank and connected components illustrate its use. Repeated edge traversal and joins make partitioning and graph structure important performance factors, particularly for graphs with highly uneven vertex degrees.

Storage formats, catalogs, and table systems

Spark separates computation from durable storage. It can read and write local files, Hadoop-compatible distributed file systems, cloud object stores, and external databases through connectors. This flexibility allows one execution engine to process data held in several systems, but it also means that consistency, transaction support, and access control partly depend on those systems. Spark does not make an arbitrary directory of files behave like a fully transactional database.

Columnar formats are especially important for analytical workloads. Apache Parquet and ORC organize data so that queries can read selected columns rather than entire records. Their metadata can also help skip irrelevant data ranges. If a query needs three columns from a table containing hundreds, column pruning may reduce input substantially. Compression and encoded representations can further reduce storage and transfer costs, although the actual savings depend on the values and their organization.

Physical partitioning places records into different paths or table partitions according to selected columns, commonly dates or broad categories. A query restricted to a particular day can then avoid scanning unrelated partitions. Partitioning on a column with extremely many distinct values, however, can generate excessive directories and tiny files. Too many small files create planning, listing, and task-scheduling overhead; very large files can reduce available parallelism or make individual tasks costly.

Spark integrates with catalog services that resolve table names and supply schemas and metadata. Apache Hive compatibility, particularly through the Hive metastore, helped Spark fit into existing Hadoop deployments. A catalog entry identifies and describes a table, but does not by itself establish all transactional guarantees. Those guarantees depend on the table implementation, storage system, and commit protocol.

Table formats such as Apache Iceberg, Delta Lake, and Apache Hudi add metadata and commit mechanisms above data files. They can support features such as snapshots, schema evolution, and atomic table updates, with specific capabilities varying by format and integration. Their existence is important to Spark’s role in lakehouse architectures, which combine file-based analytical storage with some capabilities traditionally associated with data warehouses.

Writing distributed output creates its own failure problem. Many tasks may finish independently, while the overall job fails before all files are committed. Output committers and table transaction protocols determine which files become visible and how retries are handled. Renaming files may be cheap and atomic on one storage system but implemented very differently on an object store.

Database access through JDBC provides another integration route. Parallel reads can divide a source query into partitions, but excessive concurrency can overload the source database. Predicate pushdown and partition selection affect both Spark’s work and the load imposed on the remote service.

Programming interfaces and language boundaries

Spark’s APIs expose a common engine through several languages, but their execution paths differ. Spark itself is principally implemented in Scala and runs on the Java Virtual Machine. Java and Scala applications can interact directly with its JVM-based interfaces. Python applications use PySpark, while SparkR provides an R interface focused on structured data processing.

The language used to construct a query does not necessarily determine where each operation runs. A PySpark expression that filters a column, joins two DataFrames, or computes a built-in aggregate normally creates an expression executed by Spark’s engine. It does not imply a Python interpreter loop over every input row. Conversely, a Python user-defined function ordinarily requires data to cross a language boundary and execute in Python worker processes.

This distinction explains an important performance pattern: built-in column expressions often give the optimizer more information and avoid per-row language conversion. Python functions can still be necessary for specialized libraries or unsupported operations, but their execution introduces costs associated with serialization, worker management, and data representation. Their behavior can also be less transparent to query optimization.

Apache Arrow provides a columnar interchange representation used in several Spark–Python integration paths. Pandas user-defined functions and related interfaces can process batches rather than invoking a Python function separately for every scalar value. This reduces some overhead, but does not eliminate memory constraints. An entire group passed to a grouped operation may still be large, and converting distributed data into a local pandas DataFrame requires that the resulting data fit in the receiving environment.

Functions sent to executors also carry programming-language constraints. A closure can capture variables from its surrounding scope, potentially including large objects or resources that cannot be serialized meaningfully. A database connection created on the driver is not automatically a usable connection inside every executor process. Broadcast variables distribute read-only values efficiently, while accumulators provide a mechanism for combining supported updates from tasks. Neither is a general shared mutable object.

Spark Connect, introduced in version 3.4, changes the client-server boundary. A client sends a representation of a logical plan to a remote Spark server rather than requiring the traditional tightly coupled client arrangement. This can simplify remote applications and isolate client environments, but support depends on the API and Spark version; low-level interfaces tied to the driver process do not automatically fit the remote model.

Interactive notebooks and command-line shells make Spark accessible for exploration. They do not change its distributed semantics: an apparently simple expression may launch many tasks, retrieve a large result, or retain expensive cached state across later commands.

Performance, operations, and limitations

Spark performance depends on balancing parallelism against coordination and data movement. Too few partitions leave processors idle and can produce tasks too large for available memory. Too many partitions increase scheduling overhead, shuffle metadata, and the number of output files. A useful partition size therefore depends on record width, compression, operator behavior, hardware, and the workload rather than a universal fixed number.

Data skew is a recurring difficulty. If one key accounts for a large fraction of records, a partition receiving that key may run much longer than the others. The stage then waits for a small number of stragglers despite substantial idle capacity elsewhere. Adaptive skew handling can help eligible joins, while changes such as partial aggregation or carefully designed key salting can redistribute some workloads. These transformations must preserve the intended result; they are not interchangeable fixes for every uneven dataset.

Memory use has several components, including execution structures, cached data, JVM overhead, and memory used by Python or other native processes. Sorting and aggregation can spill intermediate data to disk. Spilling is part of normal execution, but sustained spill may indicate that individual tasks have too much working data. Increasing executor heap size alone can introduce longer garbage-collection pauses or fail to address memory used outside the JVM.

The Spark web interface and event logs expose jobs, stages, task durations, shuffle volumes, spills, and executor behavior. These measurements help distinguish computational bottlenecks from storage delays or skew. Query plans reveal scans, exchanges, joins, and code-generation boundaries. Interpreting a slow query generally requires connecting the logical operation to the physical plan and then to observed task behavior.

Operational reliability extends beyond retrying tasks. Driver recovery, checkpoint durability, dependency distribution, credential management, and external service availability all matter. Authentication, authorization, encrypted transport, and restricted access to administrative interfaces must be supplied through Spark’s configuration and its surrounding infrastructure. Executors run application code, so a shared cluster is also a code-execution environment, not merely a passive query endpoint.

Spark is well suited to large scans, transformations, joins, and repeatable analytical pipelines. Its scheduling and distributed coordination costs make it less suitable for tiny computations that fit comfortably in one process. It is also not a replacement for an online transaction-processing database, a low-latency key-value service, or every form of event-processing engine.

The same distinction applies to recovery guarantees. Spark can reconstruct lost computation, but this does not replace durable storage, backups, transactional output design, or validation of source data. The behavior of a complete application emerges from the combination of Spark’s execution model, user code, connectors, storage protocols, and operational configuration.