MVICH Design and Performance

MVICH implements four protocols to maximize performance over a range of message sizes:

• For short messages, MVICH uses an eager protocol, buffering data on the receiver.
• For long messages, MVICH uses one of three protocols, depending on whether the underlying hardware supports RDMA read, and whether user buffers can be pinned.
  1. The "r3" protocol is a standard rendezvous protocol in which data is pipelined through system buffers on sender and receiver.
  2. The "rput" protocol is a zero-copy protocol in which sender does an RDMA write from send buffer to receive buffer.
  3. The "rget" protocol is a zero-copy protocol in which receiver does an RDMA read from the send buffer.

To support the zero-copy protocols, MVICH provides dynamic memory registration (pinning) of user buffers, detecting quickly when memory has already been registered, so that for most applications the overhead of memory registration is negligible.

Process management in MVICH is cleanly separated from communication initialization, so that MVICH can work with any external process manager.

Although MVICH has not yet been tuned, it adds only 5 microseconds to one-way VIA latency and obtains full bandwidth available from VIA. (As usual, these numbers are computed from a pingpong test). For Giganet with 32-bit PCI on x86 platforms, MVICH has a latency of 13.5 microseconds and a bandwidth of 97 MB/s.