Compiling software: Mahuika

Where to build¶

You may compile (build) software on the Mahuika login nodes, login.mahuika.nesi.org.nz. Please be aware that these login nodes are limited and shared resources. Please limit the amount of processes on these nodes. For example, use make -j 5 instead of make -j. If you require many CPU cores or long run times for your build process, please request these resources through a batch job submitted to Slurm, where you can also ask for a larger amount of compute resources to build your code.

Compilers and Toolchains¶

GNU and Intel compilers are provided on Mahuika. They can be accessed by loading one of the toolchains:

The GNU and Intel compilers can be accessed by loading one of the toolchains:

module load gimkl/2022a - the default toolchain, providing GNU compilers (version 9.2.0), Intel MPI and Intel MKL
module load intel/2022a - Intel compilers (version 2020.0.166), Intel MPI and Intel MKL

There are also various smaller "sub-toolchains" which can be used in the same way. Toolchains should not be mixed, except where one is a subset of the other, so for example it is OK to use a library built with GCC/12.3.0 or gompi/2023a as a dependency of an application being built with foss/2023a.


Toolchain	Compilers	MPI	LAPACK	ScaLAPACK	FFTW
GCC/12.3.0	GCC
gompi/2023a	GCC	OpenMPI
foss/2023a	GCC	OpenMPI	FlexiBLAS	ScaLAPACK	FFTW
GCC/11.3.0	GCC
gimpi/2022a	GCC	Intel MPI	Intel MKL	Intel MKL	Intel MKL
gimkl/2022a	GCC	Intel MPI	Intel MKL	Intel MKL	Intel MKL
intel-compilers/2022.0.2	Intel
iimpi/2022a	Intel	Intel MPI
intel/2022a	Intel	Intel MPI	Intel MKL	Intel MKL	Intel MKL

Third party applications¶

Installation instructions vary from application to application, and we suggest that you carefully read the instructions provided by the developers of the software you plan to use. Nevertheless, the following should give you an impression which steps you usually need to consider:

Change into your desired source code directory. We suggest you use /nesi/project/<projectID>, or more typically one of its subdirectories. You may instead use /nesi/nobackup/<projectID> (or one of its subdirectories) if you don't mind the software not being backed up and prone to automatic deletion in certain circumstances.
Download the source code. This could be done via a repository checkout (git clone <source-url>) or via downloading a tarball (wget <source-url>).
Ensure the tarball is not a tarbomb, using tar tf <source.tar> | sort | less (tar tzf ... if the source code is a gzipped tarball, tar tjf ... if a bzip2 compressed tarball). If you find that the tarball is in fact a tarbomb, you will need to handle it using special techniques.
Unpack the tarball using tar xf <source.tar>. Change into the source directory.
Load the preferred toolchain (or compiler module) and modules for any additional required libraries (module load gimkl FFTW)
Run the configure script with appropriate options, e.g. ./configure --prefix=<desired install directory> --use-fftw=$EBROOTFFTW(options can usually be listed using ./configure --help)
In some applications you need to adjust the Makefile (generated by configure) to reflect your preferred compiler, and library options (see below)
Compile the code (make)
install the binaries and libraries into the specified directory (make install)

Compilers¶

Compilers are provided for Fortran, C, and C++. For MPI-parallelised code, MPI compiler wrappers typically need to be used.

Language	Intel	GNU
Fortran	ifort	gfortran
Fortran + MPI	mpiifort	mpif90
C	icc	gcc
C + MPI	mpiicc	mpicc
C++	icpc	g++
C++ + MPI	mpiicpc	mpicxx

In general you then compile your code using:

<compiler> <CompilerOptions> <source-file>

e.g.

gfortran -O3 hello.f90

Compiler options¶

Compilers are controlled using different options to control optimizations, output, source and library handling. There options vary between the different compiler vendors. That means you will need to change them if you decide to switch compilers. The following table provides a list of commonly used compiler options for the different compilers:

Group	Intel	GNU	Notes
Debugging	`-g` or `-debug [keyword]`	`-g` or `-g{0,1,2,3}`	Set level of debugging information, some levels may disable certain compiler optimisations
Light compiler optimisation	`-O2`	`-O2`
Aggressive compiler optimisation	`-O3 -ipo`	`-O3 -ffast-math -funroll-loops`	This may affect numerical accuracy
Architecture specific optimisation	`-xHost`	`-march=native` `-mtune=native`	Build and compute nodes have the same architecture (Broadwell)
Vectorisation reports	`-qopt-report`	`-fopt-info-vec` or `-fopt-info-missed`
OpenMP	`-qopenmp`	`-fopenmp``

Additional compiler options are documented in the compiler man pages, e.g. man mpicc, which are available after loading the related compiler module. Additional documentation can be also found at the vendor web pages:

Intel Parallel Studio XE Cluster Edition for Linux is installed on the Mahuika HPC Cluster, Mahuika Ancillary Nodes
Intel Developer Guides
GCC Manuals

For example, the following commands would be used to compile with the gfortran compiler, activate compiler warnings (-Wall), and requiring aggressive compiler optimisation (-O3):

module load gimkl/2022b

mpif90 -Wall -O3 -o simpleMpi simpleMpi.f90

Linking¶

Your application may depend on one or more external software packages, normally libraries, and if so it will need to link against them when you compile the program. In general, to link against an external package, one must specify:

The location of the header files, using the option -I/path/to/headers
The location of the compiled library or libraries, using -L/path/to/lib/
The name of each library, typically without prefixes and suffixes. For example, if the full library file name is libfoo.so.1.2.3 (with aliases libfoo.so.1 and libfoo.so), the expected entry on the link line is -lfoo.

Thus the linker expects to find the include headers in the /path/to/headers and the library at /path/to/lib/lib.so (we assume dynamic linking).

Note that you may need to list several libraries to link successfully, e.g., -lA -lB for linking against libraries "A" and "B". The order in which you list libraries matters, as the linker will go through the list in order of appearance. If library "A" depends on library "B", specifying -lA -lB will work. If library "B" depends on "A", use -lB -lA. If they depend on each other, use -lA -lB -lA (although such cases are quite rare).

External Libraries¶

There are already many libraries provided on the Mahuika platform. Most of them are provided in modules. You can search them using

module spider

and look in the module description using:

module help <module-name>

Most libraries are provided using the EasyBuild software management system, that we use to install modules. Easybuild automatically defines environment variables $EBROOT<library name in upper case> when a module is loaded, which help pointing the compiler and linker to include files and libraries as in the example above. Thus, you can keep your Makefile independent of library versions, by defining e.g. -L$EBROOT<library name in upper case>/lib. Therewith you can use another version by only swapping modules. If you are unsure which $EBROOT<...> variables are available, use

module show <module-name>

to find out.

Note that specifying search paths with -I and -L is not strictly necessary in case of the GNU and Intel compilers, which will use the contents of CPATH, LIRARY_PATH, and LD_LIBRARY_PATH provided by the environment module.

Important note: Make sure that you load the correct variant of a library, depending on your choice of compiler. Switching compiler environment will not switch environment modules automatically. Furthermore, loading an environment module may switch programming environment if it was built with a different compiler. In general, the used library should be build with the same compiler.

Common Linker Problems¶

Linking can easily go wrong. Most often, you will see linker errors about "missing symbols" when the linker could not find a function used in your program or in one of the libraries that you linked against. To resolve this problem, have a closer look at the function names that the linker reported:

Are you missing some object code files (these are compiled source files and have suffix .o) that should appear on the linker line? This can happen if the build system was not configured correctly or has a bug. Try running the linking step manually with all source files and debug the build system (which can be a lengthy and cumbersome process, unfortunately).
Do the missing functions have names that contain "mp" or "omp"? This could mean that some of your source files or external libraries were built with OpenMP support, which requires you to set an OpenMP flag (-fopenmp for GNU compilers, -qopenmp for Intel) in your linker command.
Do you see a very long list of complex-looking function names, and does your source code or external library dependency include C++ code? You may need to explicitly link against the C++ standard library (-lstdc++ for GNU compilers, -cxxlib for Intel compilers); this is a particularly common problem for statically linked code.
Do the function names end with an underscore ("_")? You might be missing some Fortran code, either from your own sources or from a library that was written in Fortran, or parts of your Fortran code were built with flags such as -assume nounderscore (Intel) or -fno-underscoring (GNU), while others were using different flags.
Do the function names end with double underscores ("__")? Fortran compilers offer an option to add double underscores to Fortran subroutine names for compatibility reasons (-h [no]second_underscore, -assume [no]2underscores, -f[no-]second-underscore) which you may have to add or remove.
Compiler not necessarily enable preprocessing, which could result in #ifndef VAR; Warning: Illegal preprocessor directive. For example, using preprocessor directives in .f files with gfortran requires the -cpp option.

Note that the linker requires that function names match exactly, so any variation in function name in your code will lead to a "missing symbols" error (with the exception of character case in Fortran source code).