Remote memory access (RMA) is a method of accessing memory associated with another processor, process, or computing node. In parallel computing and distributed computing, the term commonly refers to operations that read, write, or atomically update a designated region of another process’s memory without requiring that process to execute a matching receive operation for each transfer. RMA can be provided by a programming library, a language runtime, or hardware. Its precise meaning depends on the system: a remote access within a shared-memory computer differs from an explicit transfer between machines connected by a network.
Scope and related concepts
“Remote” describes the relationship between an initiator and the memory being accessed; it does not necessarily imply a geographically distant computer. The memory may reside on another processor socket in the same machine or on another node in a computing cluster.
Three related concepts require distinction:
- Remote memory access is an access mechanism or programming abstraction. It specifies operations on memory associated with another execution context.
- Remote direct memory access (RDMA) is a communication capability in which network hardware can transfer data between designated memory regions with little or no involvement from the remote processor in the transfer itself. It can implement RMA operations, but RMA does not require RDMA hardware.
- Remote access in non-uniform memory access (NUMA) occurs when a processor accesses memory attached to another NUMA node. This normally uses ordinary load and store instructions within a shared address space rather than an explicit network communication interface.
These concepts overlap but are not interchangeable. An RMA library can implement transfers using software messages, while an RDMA transport can support communication interfaces other than an RMA programming model.
Programming model
One-sided communication
RMA is commonly described as one-sided communication. In conventional message passing, a sender and receiver coordinate through matching communication operations. In RMA, an initiating process specifies a remote memory location and requests an operation on it.
The initiating process is often called the origin, and the process whose memory is accessed is called the target. Typical operations include:
| Operation | Effect |
|---|---|
| Put | Copies data from the origin’s memory into a target memory region |
| Get | Copies data from a target memory region into the origin’s memory |
| Accumulate | Combines supplied values with values at the target using a specified operation |
| Atomic update | Performs a supported indivisible operation, such as compare-and-swap or fetch-and-add |
Not every interface provides all these operations, and the guarantees attached to them differ.
“One-sided” does not mean that the target has no responsibilities. The target generally participates in establishing accessible memory, exchanging addressing information, and following synchronization rules. Some implementations also depend on software progress at the target or on a separate progress thread.
Exposed memory and addressing
A process does not ordinarily make its entire address space accessible to peers. Instead, it exposes selected memory regions through an application programming interface. A remote location may be identified by a process rank, a region identifier, and an offset within that region.
A local pointer is generally not sufficient to identify remote data. Separate processes can have different virtual memory mappings, and the same numerical address can refer to unrelated objects.
Regions must remain valid for the required duration. Releasing or reusing memory while a remote operation is outstanding can cause incorrect access even if the initiating process has already returned from the call that issued it.
Synchronization and memory semantics
RMA separates data movement from coordination. Removing a matching receive operation does not remove the need for synchronization.
Initiation and completion
Several events may be distinct:
- Initiation: an operation has been submitted.
- Local completion: the origin can safely reuse an operation’s source buffer, or read its destination buffer, as appropriate.
- Remote completion: the operation has completed at the target according to the interface’s rules.
- Observation by target computation: target-side code can safely observe and use the result.
The exact distinctions depend on the operation and interface. For example, a put may become locally complete before its data has reached the target. Conversely, completion at a network adapter does not by itself establish every ordering or visibility guarantee needed by a concurrently executing target program.
Ordering and consistency
An RMA memory model defines how remote operations interact with each other and with ordinary processor loads and stores. It addresses questions such as:
- Whether two writes from one origin become visible in issue order.
- Whether operations to different targets have any ordering relationship.
- When a target can safely read remotely written data.
- How ordinary local stores interact with remote reads or updates.
These guarantees are not implied merely by reliable data delivery. Cache coherence, communication ordering, and application synchronization are related but distinct properties.
A common pattern writes a data block and then updates a flag indicating that the block is ready. This is correct only if the interface guarantees the necessary ordering, completion, and visibility. Without those guarantees, the target might observe the flag before it can safely consume the data.
Atomicity
An atomic operation prevents specified interference during a particular update. It does not automatically make a larger sequence of operations atomic.
For example, an atomic counter increment can support distributed work allocation, but updating a counter and a separate record does not thereby become a transaction. Programs must also respect restrictions on mixing ordinary loads and stores with remote atomic operations. Violating these rules can introduce a race condition even when individual operations are supported.
Major interfaces
MPI remote memory access
The Message Passing Interface (MPI) standardized one-sided communication in MPI-2 and substantially extended its RMA facilities in MPI-3.
MPI exposes memory through a window, an object describing memory that other processes in an associated group may access. Operations such as MPI_Put, MPI_Get, and MPI_Accumulate refer to locations within a target’s window.
MPI organizes access through epochs, intervals governed by particular synchronization rules:
- Active-target synchronization involves explicit participation by origin and target processes. Fence synchronization and post–start–complete–wait synchronization provide different coordination patterns.
- Passive-target synchronization uses window lock and unlock operations, allowing origins to initiate access without a matching target-side synchronization call for every access epoch.
Passive-target synchronization does not necessarily mean that the implementation makes progress independently of the target processor.
MPI also distinguishes unified and separate window memory models. The separate model distinguishes public window state accessed through RMA from private state accessed through ordinary local operations. The unified model provides stronger coherence between those views, but does not eliminate synchronization requirements for conflicting accesses.
OpenSHMEM and partitioned global address spaces
OpenSHMEM provides a one-sided communication interface organized around processing elements and symmetric data objects. Its operations include remote puts, gets, atomic updates, and synchronization facilities.
Partitioned global address space (PGAS) languages and libraries present memory as logically shared while retaining information about which process or locality owns each part. Remote access may appear as a language expression rather than an explicit communication call.
PGAS abstractions can simplify distributed data structures, but they do not make remote accesses equivalent in cost to local accesses. An apparently simple expression may initiate network communication.
Implementation
Hardware-assisted transfers
An RDMA implementation typically establishes registered memory regions and permissions that a network adapter can enforce. After setup, the adapter can perform transfers without requiring the remote application to copy each incoming message into its final destination.
Registration establishes mappings and protection information. Depending on the platform, it may involve pinning pages, on-demand paging support, or other integration with the operating system.
Avoiding intermediate copies and reducing CPU involvement can improve throughput and reduce per-transfer overhead. These benefits do not imply zero setup cost, zero processor involvement throughout the operation’s lifecycle, or automatic application-level synchronization.
Software implementations and progress
Without suitable hardware, a runtime can implement RMA by exchanging internal messages. A remote get, for example, can become a request message followed by a response containing the requested data.
Such an implementation preserves the programming interface’s one-sided character even if its internal transport uses two-sided communication.
Progress may be driven by network hardware, background threads, calls into the communication library, or combinations of these mechanisms. Progress behavior affects whether a computation can genuinely overlap communication and whether an operation completes while the target performs unrelated work.
Performance and applications
RMA is useful when direct access to distributed data avoids repeated request–reply exchanges or unnecessary target-side processing. Applications include distributed arrays, irregular graph processing, distributed work queues, scientific simulations, and memory-oriented services in a data center.
A simple transfer-cost approximation is
where is the transferred data size, is effective startup latency, and is effective transfer bandwidth. This model omits registration, synchronization, congestion, contention, and software overhead, but illustrates why many tiny operations may be more expensive than fewer aggregated transfers.
Important performance considerations include:
- Locality: repeated local accesses usually cost less than repeated remote accesses.
- Concurrency: multiple outstanding operations can hide latency, within hardware and runtime limits.
- Contention: many origins updating one target location can serialize execution.
- Synchronization frequency: flushing or synchronizing after every operation can dominate transfer costs.
- Access pattern: contiguous transfers often use available bandwidth more effectively than scattered small accesses.
- Resource limits: registered memory, communication queues, and completion-processing capacity are finite.
RMA is therefore not inherently faster than message passing. A well-structured bulk message exchange may outperform fine-grained remote accesses, especially when communication can be aggregated or when target-side computation is necessary anyway.
Protection and failure handling
Remote access requires explicit access control. Implementations may use memory-region permissions, remote access keys, process isolation, and hardware address-translation mechanisms to restrict which peers can access which memory.
A memory access key is not necessarily equivalent to authentication or encryption. Protection against unauthorized memory operations and protection of traffic in transit are separate concerns.
Failure handling also differs from ordinary local memory access. A target process can terminate, a connection can fail, or an operation can be interrupted after some effects have occurred. Applications must interpret completion and error guarantees carefully: retrying a non-idempotent operation, such as an increment, can duplicate its effect if the original outcome is uncertain.
Transport completion also does not establish application-level acceptance or durable storage. Systems that require recoverable updates need additional protocols for acknowledging, recording, or reconstructing those updates.