CA 10.7 parallel Debian7 - make ptscotch

Quelle: Code-Aster 10.7 parallel Installation
Version
OS：Ubuntu 12.04
Code_Aster : ver.10.7


Preparation
Download files
Save location for files : ~/Install_Files
Install location : /opt and /opt/aster

Download and locate lower files to Install_Files

site file name
Code_Aster aster-full-src-10.7.0-1.noarch.tar.gz
ACML acml-5-1-0-gfortran-64bit.tgz
ScaLAPACK
scalapack_installer.tgz
ParMETIS ParMetis-3.2.0.tar.gz
PETSc petsc-2.3.3.tar.gz

Change owner of install location
Change owner of Install location to log-in user from root user.
$ sudo chown username /opt/

Install libraries for Code_Aster
Install libraries for Code_Aster.
$ sudo apt-get install gfortran g++ python-dev python-qt4 python-numpy liblapack-dev libblas-dev tcl tk zlib1g-dev bison flex checkinstall openmpi-bin openmpi-dev

Compile ACML
Compile ACML (Math LIbrary for Code_Aster).

$ cd ~/Install_Files/

$ mkdir ./acml

$ cp acml-5-1-0-gfortran-64bit.tgz ./acml

$ cd ./acml

$ tar xfvz acml-5-1-0-gfortran-64bit.tgz

$ ./install-acml-5-1-0-gfortran-64bit.sh -accept -installdir=/opt/acml5.1.0

$ echo /opt/acml5.1.0/gfortran64/lib | sudo tee -a /etc/ld.so.conf.d/amd_math.conf

$ sudo ldconfig

Compile Code_Aster (sequential)
Compile Code_Aster (sequential) with ACML.

$ cd ~/Install_Files

$ tar xfvz aster-full-src-10.7.0.noarch.tar.gz

$ cd aster-full-src-10.7.0/

$ sed -i "s:PREFER_COMPILER\ =\ 'GNU':PREFER_COMPILER\ =\'GNU_without_MATH'\nMATHLIB=\ '-L/opt/acml5.1.0/gfortran64/lib -lacml':g" setup.cfg

$ python setup.py install


Make host-file for parallel calculation. Type your PC'name in place of 'ubuntu' , and the number of processors for parallel calculation behind 'cpu='.
$ echo ubuntu cpu=2 >> /opt/aster/etc/codeaster/mpi_hostfile

Compile ScaLAPACK

$ cd ~/Install_Files

$ tar xfvz scalapack_installer.tgz

$ cd scalapack_installer_1.0.2

$ ./setup.py --lapacklib=/opt/acml5.1.0/gfortran64/lib/libacml.a --mpicc=mpicc --mpif90=mpif90 --mpiincdir=/usr/lib/openmpi/include --prefix=/opt/scalapack


An error message "BLACS: error running BLACS test routines xCbtest" will show up after compilation.

But you succeed , if there is an file "/opt/scalapack/lib/libscalapack.a".



Compile ParMETIS

$ cp ~/Install_Files/ParMetis-3.2.0.tar.gz /opt

$ cd /opt

$ tar xfvz ParMetis-3.2.0.tar.gz

$ cd ParMetis-3.2.0


Change 'Makefile.in' like ParMETIS Makefile.in

$ make

Compile PT-Scotch
Copy scotch-5.1.11_esmumps in Code_Aster's source files to '/opt.
Compile by mpi-compiler

$ cp ~/Install_Files/aster-full-src-10.7.0/SRC/scotch-5.1.11_esmumps-2.tar.gz /opt/

$ cd /opt

$ tar xfvz scotch-5.1.11_esmumps-2.tar.gz

$ mv scotch_5.1.11_esmumps scotch_5.1.11_esmumps_mpi

$ cd scotch_5.1.11_esmumps_mpi/src



Change 'Makefile.inc' like PT-Scotch Makefile.inc

$ make scotch

$ make ptscotch

Compile MUMPS
Copy mumps-4.9.2 in Code_Aster's source files to '/opt.
Compile by mpi-compiler
$ cp ~/Install_Files/aster-full-src-10.7.0/SRC/mumps-4.9.2.tar.gz /opt/

$ cd /opt

$ tar xfvz mumps-4.9.2.tar.gz

$ mv mumps-4.9.2 mumps-4.9.2_mpi

$ cd mumps-4.9.2_mpi/


Change 'Makefile.inc' like MUMPS Makefile.inc

$ make all


Compile PETSc

$ cp ~/Install_Files/petsc-2.3.3.tar.gz /opt

$ cd /opt

$ tar xfvz petsc-2.3.3.tar.gz

$ cd petsc-2.3.3-p16

$ ./config/configure.py --with-mpi-dir=/usr/lib/openmpi/lib --with-blas-lapack-lib=/opt/acml5.1.0/gfortran64/lib/libacml.a --with-debugging=0 COPTFLAGS=-O3 FOPTFLAGS=-O3 --with-shared=0 --configModules=PETSc.Configure --optionsModule=PETSc.compilerOptions --with-x=0

$ PETSC_ARCH=linux-gnu-c-opt; export PETSC_ARCH

$ PETSC_DIR=/opt/petsc-2.3.3-p16; export PETSC_DIR

$ make

Compile Code_Aster (parallel)
Change a part of 'mpi_get_procid_cmd' of '/opt/aster/etc/codeaster/asrun' to as following with text editor.
mpi_get_procid_cmd : echo $OMPI_COMM_WORLD_RANK

Change following numbers if the number of processors is over 32.
batch_mpi_nbpmax : 32
interactif_mpi_nbpmax : 32

Copy '/opt/aster/STA10.7’ and make 'PAR10.7'.

$ cd /opt/aster

$ cp -ax STA10.7 PAR10.7

$ cd /opt/aster/PAR10.7

Change config.txt and profile.sh like PAR config.txt PAR profile.sh, and re-compile Code_Aster.

$ ../bin/as_run --vers=PAR10.7 --make clean

$ ../bin/as_run --vers=PAR10.7 --make clean bibf90/mumps

$ ../bin/as_run --vers=PAR10.7 --make clean bibf90/petsc

$ ../bin/as_run --vers=PAR10.7 --make clean bibc/scotch

$ ../bin/as_run --vers=PAR10.7 --make

golbs@debian7-cae:/opt/scalapack_installer_1.0.2$ ./setup.py --lapacklib=/opt/acml5.1.0/gfortran64/lib/libacml.a --mpicc=mpicc --mpif90=mpif90 --mpiincdir=/usr/lib/openmpi/include --prefix=/opt/scalapack
========================================
Setting up the framework
ScaLAPACK installer version (1, 0, 2)
mpicc is mpicc
mpif90 is mpif90
MPI include dir is /usr/lib/openmpi/include
Creating directory /opt/scalapack
Install directory is... /opt/scalapack
Creating directory /opt/scalapack_installer_1.0.2/build
Build directory is... /opt/scalapack_installer_1.0.2/build
BLAS library is...
LAPACK library is... /opt/acml5.1.0/gfortran64/lib/libacml.a
Checking if mpicc works... yes
Checking if mpirun works... yes
Checking if mpif90 works... yes
Setting Fortran mangling... -DAdd_
Setting download command...
Checking availability of urllib... available
Testing urllib... working
Setting ranlib command... /usr/bin/ranlib
Detecting Fortran compiler... unknown
Detecting C compiler... unknown
C flags are... -O2
Fortran flags are... -O2
Selected loader flags (C main):
Selected loader flags (f90 main):
Selected NOOPT flags: -O0
AR flags are... rc
Checking loader... works

========================================
BLAS installation/verification
========================================
Checking if the provided LAPACK (/opt/acml5.1.0/gfortran64/lib/libacml.a) contains BLAS
BLAS library is set to /opt/acml5.1.0/gfortran64/lib/libacml.a
Checking if provided BLAS works... yes
Using the BLAS library contained in LAPACK library /opt/acml5.1.0/gfortran64/lib/libacml.a

========================================
Lapack installation/verification
========================================
LAPACK library is /opt/acml5.1.0/gfortran64/lib/libacml.a
Checking if provided LAPACK works... yes
Getting LAPACK version number... 3.3.0
Checking if provided LAPACK contains functions for test works... no

========================================
ScaLAPACK installer is starting now. Buckle up!
========================================
Downloading ScaLAPACK... Creating directory /opt/scalapack_installer_1.0.2/build/download
done
Installing scalapack-2.0.2 ...
Writing SLmake.inc... done.
Compiling BLACS, PBLAS and ScaLAPACK... done
Getting ScaLAPACK version number... 2.0.1
Installation of ScaLAPACK successful.
(log is in /opt/scalapack_installer_1.0.2/build/log/scalog )
Compiling test routines... done
Running BLACS test routines...

BLACS: error running BLACS test routines xCbtest


BLACS: Command -np 4 ./xCbtest
stderr:
****************************************
/bin/sh: 1: -np: not found

****************************************
golbs@debian7-cae:/opt/scalapack_installer_1.0.2$

golbs@debian7-cae:/opt/scotch_5.1.11_esmumps_mpi/src$ make ptscotch
(cd libscotch ; make VERSION=5 RELEASE=1 PATCHLEVEL=10 ptscotch && make ptinstall)
make[1]: Entering directory `/opt/scotch_5.1.11_esmumps_mpi/src/libscotch'
rm -f *~ *.o lib*.a parser_yy.c parser_ly.h parser_ll.c *scotch.h *scotchf.h y.output dummysizes
make CFLAGS="-O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_PTHREAD -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -DSCOTCH_RENAME_PARSER -DSCOTCH_PTHREAD -Drestrict=__restrict -DIDXSIZE64 -DSCOTCH_PTSCOTCH" CC="mpicc" \
scotch.h \
scotchf.h \
libptscotch.a \
libscotch.a \
libptscotcherr.a \
libptscotcherrexit.a
make[2]: Entering directory `/opt/scotch_5.1.11_esmumps_mpi/src/libscotch'
gcc -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_PTHREAD -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -DSCOTCH_RENAME_PARSER -DSCOTCH_PTHREAD -Drestrict=__restrict -DIDXSIZE64 -DSCOTCH_PTSCOTCH -DSCOTCH_VERSION=5 -DSCOTCH_RELEASE=1 -DSCOTCH_PATCHLEVEL=10 dummysizes.c -o dummysizes -lz -lm -lrt
In file included from dummysizes.c:80:0:
common.h:99:28: fatal error: mpi.h: Datei oder Verzeichnis nicht gefunden
compilation terminated.
make[2]: *** [dummysizes] Fehler 1
make[2]: Leaving directory `/opt/scotch_5.1.11_esmumps_mpi/src/libscotch'
make[1]: *** [ptscotch] Fehler 2
make[1]: Leaving directory `/opt/scotch_5.1.11_esmumps_mpi/src/libscotch'
make: *** [ptscotch] Fehler 2
golbs@debian7-cae:/opt/scotch_5.1.11_esmumps_mpi/src$

golbs@debian7-cae:/opt/mumps-4.9.2_mpi$ make all
make ARITH=s mumps_lib
make[1]: Entering directory `/opt/mumps-4.9.2_mpi'
(cd src ; make s)
make[2]: Entering directory `/opt/mumps-4.9.2_mpi/src'
make ARITH=s mumps_lib
make[3]: Entering directory `/opt/mumps-4.9.2_mpi/src'
cc -O -I. -I/usr/include -I../include -DAdd_ -I../PORD/include/ -Dpord -c mumps_orderings.c
cc -O -I. -I/usr/include -I../include -DAdd_ -I../PORD/include/ -Dpord -c mumps_size.c
cc -O -I. -I/usr/include -I../include -DAdd_ -I../PORD/include/ -Dpord -c mumps_io.c
cc -O -I. -I/usr/include -I../include -DAdd_ -I../PORD/include/ -Dpord -c mumps_io_basic.c
cc -O -I. -I/usr/include -I../include -DAdd_ -I../PORD/include/ -Dpord -c mumps_io_thread.c
cc -O -I. -I/usr/include -I../include -DAdd_ -I../PORD/include/ -Dpord -c mumps_io_err.c
f90 -O -I/usr/include -Dpord -I. -I../include -c mumps_static_mapping.F
make[3]: f90: Kommando nicht gefunden
make[3]: *** [mumps_static_mapping.o] Fehler 127
make[3]: Leaving directory `/opt/mumps-4.9.2_mpi/src'
make[2]: *** Fehler 2
make[2]: Leaving directory `/opt/mumps-4.9.2_mpi/src'
make[1]: *** [mumps_lib] Fehler 2
make[1]: Leaving directory `/opt/mumps-4.9.2_mpi'
make: *** Fehler 2
golbs@debian7-cae:/opt/mumps-4.9.2_mpi$ make all
make ARITH=s mumps_lib
make[1]: Entering directory `/opt/mumps-4.9.2_mpi'
(cd src ; make s)
make[2]: Entering directory `/opt/mumps-4.9.2_mpi/src'
make ARITH=s mumps_lib
make[3]: Entering directory `/opt/mumps-4.9.2_mpi/src'
gfortran -O -Dintel_ -DALLOW_NON_INIT -I/usr/local/mpich/include -Dpord -I. -I../include -c mumps_static_mapping.F
mumps_static_mapping.F:4407: Error: Can't open included file 'mpif.h'
make[3]: *** [mumps_static_mapping.o] Fehler 1
make[3]: Leaving directory `/opt/mumps-4.9.2_mpi/src'
make[2]: *** Fehler 2
make[2]: Leaving directory `/opt/mumps-4.9.2_mpi/src'
make[1]: *** [mumps_lib] Fehler 2
make[1]: Leaving directory `/opt/mumps-4.9.2_mpi'
make: *** Fehler 2
golbs@debian7-cae:/opt/mumps-4.9.2_mpi$

MGolbs wrote: Hallo Richard,
Wie sind mpif90 und mpicc zu bekommen, Paketverwaltung?
Gruß und Dank Markus

golbs@debian7-cae:/opt/scalapack_installer_1.0.2$ ./setup.py --lapacklib=/opt/acml5.1.0/gfortran64/lib/libacml.a --mpicc=mpicc --mpif90=mpif90 --mpiincdir=/usr/lib/openmpi/include --prefix=/opt/scalapack
========================================
Setting up the framework
ScaLAPACK installer version (1, 0, 2)
mpicc is mpicc
mpif90 is mpif90
MPI include dir is /usr/lib/openmpi/include
Creating directory /opt/scalapack
Install directory is... /opt/scalapack
Creating directory /opt/scalapack_installer_1.0.2/build
Build directory is... /opt/scalapack_installer_1.0.2/build
BLAS library is...
LAPACK library is... /opt/acml5.1.0/gfortran64/lib/libacml.a
Checking if mpicc works... yes
Checking if mpirun works... yes
Checking if mpif90 works... yes

--mpiincdir=/usr/lib/openmpi/include