>> I'd like to use MPI for features like derived types, and moving
>> around non-flat data.
> You should be able to do that today.
The Axon PCIe-to-PCIe RDMA interface has a raw device and an ethernet
device in the Linux kernel, and I can indeed using the latter as a
workaround. It's more overhead but not a bottleneck for my
application. For those Roadrunner users where that cost cannot be
tolerated, the alternative is to use another less featureful non-MPI