A GPU should be good for large Cholesky decompositions. If you can do a suitable break down of your problem into good chunks, then MPI will be amazing. I think MPI is more general in terms on when it speeds up your model while GPU is really specifc (but IF a GPU can speed up your case, then you will probably want it). For MPI: The more CPUs the better and what hasn’t been mentioned yet is fast RAM.
… but MPI will be a bit of a pain for the user to program up his model…