Latest stable DOLFIN on Cray system... HOWTO...

Compiling on Cray… again

I’m just doing a bit of an update on how to install DOLFIN on Cray systems, as the question seems to keep coming up. Maybe one day, we’ll do away with this by using containers everywhere [http://arxiv.org/abs/1608.07573] but I’m not holding my breath, well, at least not until 2017.

Let’s get started. First off, it’s probably best to switch to the gnu programming environment: module swap PrgEnv-intel PrgEnv-gnu (assuming the default is intel). With luck, your Cray will have a lot of nice modules already available, such as cmake, boost, swig, eigen3. If you are really lucky, they will also be up to date. If not, then you’ll either have to compile them yourself, or bang on to your sysadmin until they install them for you. Cray helpfully package PETSc, ParMETIS and SCOTCH as cray-petsc and cray-tpsl. Also, you will find cray-hdf5-parallel. Let’s load all these modules.

module load boost/1.59
module load python/2.7.9
module load eigen3/3.2.0
module load cmake/3.3.2
module load swig/3.0.5
module load git
module load cray-tpsl
module load cray-hdf5-parallel
module load cray-petsc/3.7.2.0

Maybe it’s enough? We also need to set a few environment variables, if they are not already set.

export SCOTCH_DIR=$CRAY_TPSL_PREFIX_DIR
export PARMETIS_DIR=$CRAY_TPSL_PREFIX_DIR
export EIGEN_DIR=$EIGEN3_DIR/include

Maybe I should also say, that if you want to have JIT Python working on the compute nodes, then all these packages need to work on the compute nodes too. Some systems only install packages like cmake or even the compilers, on the login nodes.

Now download all the FEniCS code. You can either get them as tar.gz files from bitbucket:

wget https://bitbucket.org/fenics-project/dolfin/downloads/dolfin-2016.1.0.tar.gz

or just use git to get the latest version:

git clone https://git@bitbucket.org/fenics-project/dolfin

Repeat for instant, fiat, ffc, dijitso and ufl.

The Python packages (instant, fiat, ffc, dititso and ufl) can all be simply installed with: python setup.py install --prefix=/your/install/dir. You may have to make some directories, and set the $PYTHONPATH in order to get them to install. Either way, you will need to set the $PYTHONPATH and adjust $PATH before compiling DOLFIN. A good test is to see if ffc is working:

chris@Cray.loginNode> ffc
*** FFC: Missing file.

OK. DOLFIN itself needs a working MPI C++ compiler, so it a bit harder to get right. It also uses cmake to configure. Go into the DOLFIN folder, make a “build” directory: mkdir build ; cd build. Classic cmake. For Cray HPC, you are likely to want a few options though. Here is my cmake line:

cmake -DCMAKE_INSTALL_PREFIX=/your/install/dir \
      -DDOLFIN_SKIP_BUILD_TESTS=ON \
      -DDOLFIN_AUTO_DETECT_MPI=false \
      -DDOLFIN_ENABLE_UNIT_TESTS=false \
      -DDOLFIN_ENABLE_BENCHMARKS=false \
      -DDOLFIN_ENABLE_VTK=false \
      -DDOLFIN_ENABLE_OPENMP=false \
      -DCMAKE_BUILD_TYPE=Developer \
      -DDOLFIN_DEPRECATION_ERROR=true \
      ..
make -j 4
make install

Because it links with MPI, it is unlikely that any DOLFIN demos will work on the login nodes, so you’ll have to compile them and submit to a queue. Good luck!

Written with StackEdit.