ROMS 3.1 - OSX 10.5.2 Intel - ifort 10.1.006 - netcdf 3.6.2:

Message

amhsepa · #1 Unread post by **amhsepa** » Sun Mar 30, 2008 9:11 pm

Hi,

I'm looking for some advice on using the ifort (10.1.006) compiler to build ROMS 3.1. I've carried out the following steps:

1 - Installed ifort as per instructions and modified .bash_login to include all the necessary scripts.

2 - Successfully built and tested netcdf-3.6.2 using ifort. In the first case I used the Mac OS X default gcc and g++, with no Flags to the compilers. In the second case I used the Intel C/C++ compiler and followed the instructions on the Intel website: http://www.intel.com/support/performanc ... 027812.htm

3 - In both of the above cases identical things happen when I try to compile a basic serial upwelling test case. When using Debug on, ROMS compiles fine with no obvious errors. The resulting oceanG runs fine, with the usual slow speed. When I set debug to off, the code seems to compile ok, and I get lots of LOOP WAS VECTORIZED type comments. The resulting oceanS does not run and crashes, giving the following error:

Model Input Parameters: ROMS/TOMS version 3.0
Saturday - March 29, 2008 - 3:38:59 PM
-----------------------------------------------------------------------------
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
oceanS 00219873 Unknown Unknown Unknown
oceanS 00218D0F Unknown Unknown Unknown
oceanS 001DFFF8 Unknown Unknown Unknown
oceanS 001B0870 Unknown Unknown Unknown
oceanS 001B73A0 Unknown Unknown Unknown
libSystem.B.dylib 91EEA5EB Unknown Unknown Unknown
Unknown FFFFFFFF Unknown Unknown Unknown

4 - The compiler make file I'm using is Darwin-ifort.mk, which is a copy of the Linux-fort.mk. This worked ok for me in the past when using gfortran.

5 - I examined the Flags sent to ifort when debug is set to off. I've tried various combinations of these, based on the information in the ifort manual.

6 - I can only get the oceanS executable to work when I send the -O0 compiler option to ifort. This turns all code optimizations off. oceanS runs fine but just as slow as when debug was on.

7 - I did see in the forums that segmentation fault can be related to the stack size. I tried altering this but no success.

8 - I carried out the above steps using the new version of gfortran (binary) from the following excellent site:http://hpc.sourceforge.net/
netcdf and roms both compile great first time and run fast in serial. It may be worth mentioning I had problems compiling netcdf using the new versions of gcc and g++. These problems dissapear when using the default install, older, versions of gcc and G++.

I'd be really grateful if anyone has any tips on using ifort to compile ROMS. I've read that there are performance gains in using ifort to compile code on Intel chips. Any pointers would be much appreciated.

Thanks in advance.

Alan

hetland · #2 Unread post by **hetland** » Mon Mar 31, 2008 8:02 am

I had exactly the same problem. The solution appears to be to get the latest version of the compiler (10.1.012). Intel has had some pretty sever problems with Leopard, and I'm not sure whether to blame Apple or Intel. Either way, it has not been good. See:
http://softwarecommunity.intel.com/isn/ ... hread.aspx

In any case, gfortran (available with a clickable installer here: http://r.research.att.com/tools/) has worked very well for me through the transition. It is about 10% slower, but compiles quickly, and without segfaulting...

amhsepa · #3 Unread post by **amhsepa** » Mon Mar 31, 2008 10:02 am

Thanks very much for getting back so quickly, that's excellent info. It's a real shame there's been so many problems. I only purchased the Intel compiler last week, so I’m a bit puzzled as to why it let me download such an old version.

Thanks once again.

Alan

hetland · #4 Unread post by **hetland** » Mon Mar 31, 2008 3:26 pm

For the record, here are some build notes for netcdf and openmp on Leopard. I used ifort 10.1.012 and gfortran 4.3.0.

Nothing too surprising here, but I finally decided to write all this down after I kept on forgetting which options needed to be specified.

Notes for making netcdf3 and netcdf4:
------------------------------------

These libraries will be put into:

/opt/netcdf3/ifort
/opt/netcdf3/gfortran

/opt/netcdf4/ifort
/opt/netcdf4/gfortran

/opt/hdf5/ifort
/opt/hdf5/gfortran

NETCDF3 [version 3.6.2]
-------

To start with, this site has a long list of options for various compilers and systems:

http://www.unidata.ucar.edu/software/ne ... uilds.html

First, netcdf3 with ifort uses the environmental variables:

export FC=ifort
export FFLAGS='-O3'
./configure --prefix=/opt/netcdf3/ifort

Some options for FFLAGS might include: -O3 -xT -ip -no-prec-div -static (based on a build from the other builds site.) This is similar to the -fast option, which sets: -ipo, -O3, -no-prec-div, -static, and -xP. It seems that -ipo causes configure to fail, so it is left out of the options
-xT builds for 64bit duo processors, -xP for 32 bit. The default is fine, so nothing is set.
-static builds static libs (not needed, as make knows when to do this is seems)
-no-prec-div uses faster division algorithms at the expense of accuracy. Why do this?
So the only flag left standing is '-O3'. This could probably be set to '-O2' without much trouble.

To make netcdf3 gfortran, simply do

export FC=gfortran
export FFLAGS='-O3'
./configure --prefix=/opt/netcdf3/gfortran

Both can be made in parallel with make -j, and both pass make check.

HDF5 [version 1.8.0]
----

Environmental settings used are similar to netcdf3:

export F77=ifort
export F90=ifort
export FFLAGS='-O3'
./configure --prefix=/opt/hdf5/ifort --enable-hl --enable-fortran

The key is the --enable-hl flag, which allows the high-level language API needed by netcdf4. This cannot be built in full parallel, but make -j 2 seems to work fine. Passes make check, after a long, long time. For gfortran:

export F77=gfortran
export F90=gfortran
export FFLAGS='-O3'
./configure --prefix=/opt/hdf5/gfortran --enable-hl --enable-fortran

Also passes make check

NETCDF4 [version snapshot2008031713]
-------

Again, environmental settings used are similar to netcdf3:

export F77=ifort
export F90=ifort
export FFLAGS='-O3'
./configure --prefix=/opt/netcdf4/ifort -enable-netcdf-4 --with-hdf5=/opt/hdf5/ifort

The key is here is obviously the --enable-netcdf-4 command, otherwise netcdf4 is not really built. Here parallel make -j seems to work fine. Passes make check. For gfortran, no surprises:

export F77=gfortran
export F90=gfortran
export FFLAGS='-O3'
./configure --prefix=/opt/netcdf4/gfortran -enable-netcdf-4 --with-hdf5=/opt/hdf5/gfortran

Also passes make check.

Notes on making openmpi [version 1.2.5]
------------------------------

export F77=ifort
export FC=ifort
export FFLAGS='-O3'
export FCFLAGS='-O3'
./configure --prefix=/opt/openmpi/ifort --disable-mpi-profile

for gfortran

export F77=gfortran
export FC=gfortran
export FFLAGS='-O3'
export FCFLAGS='-O3'
./configure --prefix=/opt/openmpi/gfortran --disable-mpi-profile

Note that the --disable-mpi-profile is because of a poorly linked file in the profiling directory (the link refers to the parent directory instead of a file).

amhsepa · #5 Unread post by **amhsepa** » Mon Mar 31, 2008 4:46 pm

Cheers for more useful notes!

I wanted to ask about the ifort version numbers. I've looked at the available downloads and can see: (the characters in brackets are the various minor versions)

10.0.(CESR, 016, 017 and 020)

10.1.(CESR, 006,007,008 and 012)

The most recent one appears to be 10.1.012. Would you recommend this or 10.0.20?

Thanks

Alan

hetland · #6 Unread post by **hetland** » Mon Mar 31, 2008 5:48 pm

I mean 10.1.012, of course...

I have edited the above posts so that no one will ever know I was wrong. Sneaky, eh?

amhsepa · #7 Unread post by **amhsepa** » Mon Mar 31, 2008 9:26 pm

Ah...good...after peering at compiler output forever...the numbers were starting to blur! I'd better edit my posts to preserve continuity..

Thanks again for all the help and notes. Looking forward to testing it out.

enrique · #8 Unread post by **enrique** » Tue Apr 01, 2008 5:42 pm

Well, I just installed the new ifort and when trying to compile netCDF
following Rob's environment suggestions I get:

checking if Fortran "byte" is C "signed char"... configure: error: Could not link conftestf.o and conftest.o

Any ideas?

hetland · #9 Unread post by **hetland** » Tue Apr 01, 2008 6:39 pm

Versions? Which ifort, which netcdf?

powellb · #10 Unread post by **powellb** » Tue Apr 01, 2008 6:39 pm

As Rob mentioned, the problem is with ifort 10.1.006. This had a bug in the optimizer which would cause segmentation faults when unrolling loops that access arrays. This was fixed in the 008 release.

Basically, use the latest. You cannot use any release prior to 10.1 on Leopard.

Enrique, the following will setup netcdf 3.6.2 properly using ifort:

1) make distclean
2) ./configure FC=ifort F77=ifort --enable-f90 --prefix=/usr/local/netcdf-3.6.2-intel

If you still have a problem, let me know. I haven't tried the official HDF5 and latest Netcdf4 builds (but I didn't have problems with the alphas).

powellb · #11 Unread post by **powellb** » Tue Apr 01, 2008 6:49 pm

I forgot that due to some fancy arrangements that ifort 10.1 does to numerical calculations, it can wreak havoc on the symmetry between the tangent-linear and adjoint models.

I have added the following option to the Darwin-ifort.mk in my ROMS development branch (it should make it into the public branch in a while):

-assume protect_parens

This prevents the mathematical optimizer from trying to do some operations out of order. It should not have an impact on the nonlinear model.

UPDATE: This bug has been corrected in ifort, and there is no longer a need to specify this option.

enrique · #12 Unread post by **enrique** » Tue Apr 01, 2008 6:54 pm

I downloaded today ifort 10.1.012 and am trying netcdf 3.6.2

enrique · #13 Unread post by **enrique** » Tue Apr 01, 2008 7:53 pm

Brian wins the "Hacker-of-the-day" title.

I needed to source .... intel/fc/10.1.012/bin/ifortvars.csh

somehow I did not have to do this before

hetland · #14 Unread post by **hetland** » Wed Apr 02, 2008 6:30 am

Usually temporary environmental variable definitions go before the command (e.g., ./configure). Look at the script to make sure it is using the right fortran compiler. Also, try a which ifort command to make sure you are using the one you think (you will have to follow a few softlinks to get to the answer...)

enrique · #15 Unread post by **enrique** » Wed Apr 02, 2008 1:28 pm

The issue has to do with linked dynamic libraries. Ifort
has specific ones it wants ahead of others. This is
what fixes things:

if !($?LD_LIBRARY_PATH) then
setenv LD_LIBRARY_PATH /opt/intel/fc/10.1.012/lib
else
setenv LD_LIBRARY_PATH /opt/intel/fc/10.1.012/lib:${LD_LIBRARY_PATH}
endif

# DYLD_LIBRARY_PATH is used on MAC OS*
if !($?DYLD_LIBRARY_PATH) then
setenv DYLD_LIBRARY_PATH /opt/intel/fc/10.1.012/lib
else
setenv DYLD_LIBRARY_PATH /opt/intel/fc/10.1.012/lib:${DYLD_LIBRARY_PATH}
endif

[/b]

powellb · #16 Unread post by **powellb** » Wed Apr 02, 2008 11:46 pm

powellb wrote:I forgot that due to some fancy arrangements that ifort 10.1 does to numerical calculations, it can wreak havoc on the symmetry between the tangent-linear and adjoint models.

I have added the following option to the Darwin-ifort.mk in my ROMS development branch (it should make it into the public branch in a while):

-assume protect_parens

This prevents the mathematical optimizer from trying to do some operations out of order. It should not have an impact on the nonlinear model.

After installing and testing the latest ifort 10.1.012 on linux and mac, I found that they have resolved the numerical bug I listed above. You no longer need -assume protect_parens.

amhsepa · #17 Unread post by **amhsepa** » Thu Apr 10, 2008 4:32 pm

Got ROMS working with ifort 10.1.012, thanks to all for the useful info.

Intel have now released 10.1.014, although I've not tried it yet.

amhsepa · #18 Unread post by **amhsepa** » Mon Apr 14, 2008 8:46 pm

Hi. Some issues arise when you have both ifort and gfortran installed on your system (at least they do on mine!) Despite setting all the correct Environment Variables, the configure files of netcdf and openmpi both want to use gfortran, even when you specifiy ifort as your compiler. I guess this could be related to the PATH variable, but I found the following workaround successful:

- Alter the configure file and put ifort before gfortran in some of the script lines.

For netcdf you only need edit lines 4318, 4319, 4322 and 4323. For openmpi I edited every line where ifort was referenced.

I also had an issue with HDF5 where it wouldn't recognise environment variables and threw an error when trying to configure the fortran compiler.

I found the following information useful at this site:

http://www.tuinder.net/~tuinder/howto/h ... ifort.html

Specifically the command:

env F9X=ifort ./configure --enable-fortran (You can set other configuration options, if required).

This solved the compiler problem, but you need to set the environment variable right before the configure (not as a separate command). I didn't use any of the other options specified.

Ocean Modeling Discussion

ROMS 3.1 - OSX 10.5.2 Intel - ifort 10.1.006 - netcdf 3.6.2:

ROMS 3.1 - OSX 10.5.2 Intel - ifort 10.1.006 - netcdf 3.6.2:

ifort 10.1.006

Forgot