We are using OpenFarics for our HPC. According to the FAQ, "With OpenFabrics (and therefore the openib BTL component), you need to set the available locked memory to a large number (or better yet, unlimited) -- the defaults with most Linux installations are usually too low for most HPC applications that utilize OpenFabrics"
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
For the solution, look here at http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
Or you can also look at my blog entry Encountering Segmentation Fault, Bus Error or No output