big jump occured when restart

[xiaozhu557]

I have a trouble with RESTART even I defined PERFECT_RESTART.

For the first run, it ended in 930240 step, the screen output like this

Code: Select all

 930238  3229 23:50:00  2.343559E-03  2.323374E+04  2.323374E+04  4.688070E+17
         (0389,498,18)  6.291085E-03  3.352070E-03  1.438732E-01  2.145083E+00
 930239  3229 23:55:00  2.343540E-03  2.323374E+04  2.323374E+04  4.688070E+17
         (0389,498,18)  6.287961E-03  3.355134E-03  1.438942E-01  2.145129E+00
 930240  3230 00:00:00  2.343522E-03  2.323374E+04  2.323374E+04  4.688070E+17
         (0389,498,18)  6.284754E-03  3.358239E-03  1.439150E-01  2.145175E+00
      WRT_HIS   - wrote history  fields (Index=1,1) into time record = 0000070
      WRT_AVG   - wrote averaged fields into time record =             0000035
Terminated

Then the model outputed a restart file in step 930240, I restart the model from that step with the restart file, but got the screen output from the restart run, as follow,

Code: Select all

   STEP   Day HH:MM:SS  KINETIC_ENRG   POTEN_ENRG    TOTAL_ENRG    NET_VOLUME
          C => (i,j,k)       Cu            Cv            Cw         Max Speed

 930240  3230 00:00:00  2.343522E-03  2.323367E+04  2.323368E+04  4.688070E+17
         (0200,050,02)  3.130760E-02  3.470927E-02  0.000000E+00  2.145175E+00
      DEF_HIS   - inquiring history file, Grid 01: np8_his_0009.nc
      DEF_AVG   - inquiring average file, Grid 01: np8_avg_0009.nc
 930241  3230 00:05:00  2.345268E-03  2.323367E+04  2.323368E+04  4.688070E+17
         (0389,498,18)  6.270005E-03  3.394958E-03  1.439145E-01  2.144243E+00
 930242  3230 00:10:00  2.345296E-03  2.323367E+04  2.323368E+04  4.688070E+17
         (0406,495,30)  3.082030E-02  4.629990E-03  4.852346E-01  2.143983E+00
 930243  3230 00:15:00  2.345144E-03  2.323367E+04  2.323368E+04  4.688070E+17
         (0401,494,30)  2.442397E-02  8.826006E-03  4.320690E-01  2.143462E+00

I have got so much differences for the variables Cu, Cv, Cw between the first and restart run in step 930240. I also upload two figures about the variations of the total kinetic Energy (KE) with model time. It included a too big jump after the restart step.



What is the reason for that? How can I fix it?

[kate]

Did you check to make sure that PERFECT_RESTART was defined in your model output? It was not correct in your post above. Misspellings will cause ROMS to ignore your #define directives.

[xiaozhu557]

Did you check to make sure that PERFECT_RESTART was defined in your model output? It was not correct in your post above. Misspellings will cause ROMS to ignore your #define directives.

Yes, I have checked that it was defined. I just typed incorrectly in my last post. Sorry about that.

Here is some of my model's screen output,

Code: Select all

Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
./oceanM

 Model Input Parameters:  ROMS/TOMS version 3.7  
                          Sunday - February 9, 2014 -  2:25:57 PM
 -----------------------------------------------------------------------------

 North Pacific Model

 Operating system : Linux
 CPU/hardware     : x86_64
 Compiler system  : ifort
 Compiler command : /apps/mvapich2/1.8-r5609-intel/bin/mpif90
 Compiler flags   : -heap-arrays -fp-model precise -ip -O3 -free -free -free

 Input Script  : 

 SVN Root URL  : https://www.myroms.org/svn/src/trunk
 SVN Revision  : exported

 Local Root    : /mnt/lfs2/projects/lecpbm/zhuxm/trunk
 Header Dir    : /mnt/lfs2/projects/lecpbm/zhuxm/np8_new
 Header file   : np8.h
 Analytical Dir: /mnt/lfs2/projects/lecpbm/zhuxm/np8_new

 Resolution, Grid 01: 1543x0631x030,  Parallel Nodes: 432,  Tiling: 016x027


 Physical Parameters, Grid: 01
 =============================

   10368000  ntimes          Number of timesteps for 3-D equations.
    300.000  dt              Timestep size (s) for 3-D equations.
         10  ndtfast         Number of timesteps for 2-D equations between
                               each 3D timestep.
          1  ERstr           Starting ensemble/perturbation run number.
          1  ERend           Ending ensemble/perturbation run number.
         -1  nrrec           Number of restart records to read from disk.
          T  LcycleRST       Switch to recycle time-records in restart file.
       2880  nRST            Number of timesteps between the writing of data
                               into restart fields.
          1  ninfo           Number of timesteps between print of information
                               to standard output.
          T  ldefout         Switch to create a new output NetCDF file(s).
       1440  nHIS            Number of timesteps between the writing fields
                               into history file.
     103680  ndefHIS         Number of timesteps between creation of new
                               history files.
          1  ntsAVG          Starting timestep for the accumulation of output
                               time-averaged data.
       2880  nAVG            Number of timesteps between the writing of
                               time-averaged data into averages file.
     103680  ndefAVG         Number of timesteps between creation of new
                               time-averaged file.
 5.0000E+01  nl_tnu4(01)     NLM Horizontal, biharmonic mixing coefficient
                               (m4/s) for tracer 01: temp
 5.0000E+01  nl_tnu4(02)     NLM Horizontal, biharmonic mixing coefficient
                               (m4/s) for tracer 02: salt
 5.0000E+01  nl_visc4        NLM Horizontal, biharmonic mixing coefficient
                               (m4/s) for momentum.
 1.0000E-05  Akt_bak(01)     Background vertical mixing coefficient (m2/s)
                               for tracer 01: temp
 1.0000E-05  Akt_bak(02)     Background vertical mixing coefficient (m2/s)
                               for tracer 02: salt
 1.0000E-05  Akv_bak         Background vertical mixing coefficient (m2/s)
                               for momentum.
 3.0000E-03  rdrg            Linear bottom drag coefficient (m/s).
 3.0000E-03  rdrg2           Quadratic bottom drag coefficient.
 1.0000E-03  Zob             Bottom roughness (m).
          1  lmd_Jwt         Jerlov water type.
          2  Vtransform      S-coordinate transformation equation.
          4  Vstretching     S-coordinate stretching function.
 5.0000E+00  theta_s         S-coordinate surface control parameter.
 3.0000E-01  theta_b         S-coordinate bottom  control parameter.
     30.000  Tcline          S-coordinate surface/bottom layer width (m) used
                               in vertical coordinate stretching.
   1000.000  rho0            Mean density (kg/m3) for Boussinesq approximation.
      0.000  dstart          Time-stamp assigned to model initialization (days).
19900101.00  time_ref        Reference time for units attribute (yyyymmdd.dd)
 3.6000E+02  Tnudg(01)       Nudging/relaxation time scale (days)
                               for tracer 01: temp
 3.6000E+02  Tnudg(02)       Nudging/relaxation time scale (days)
                               for tracer 02: salt
 3.6000E+02  Znudg           Nudging/relaxation time scale (days)
                               for free-surface.
 3.6000E+02  M2nudg          Nudging/relaxation time scale (days)
                               for 2D momentum.
 3.6000E+02  M3nudg          Nudging/relaxation time scale (days)
                               for 3D momentum.
 1.2000E+02  obcfac          Factor between passive and active
                               open boundary conditions.
          T  VolCons(1)      NLM western  edge boundary volume conservation.
          T  VolCons(2)      NLM southern edge boundary volume conservation.
          F  VolCons(3)      NLM eastern  edge boundary volume conservation.
          T  VolCons(4)      NLM northern edge boundary volume conservation.
     10.000  T0              Background potential temperature (C) constant.
     35.000  S0              Background salinity (PSU) constant.
      1.000  gamma2          Slipperiness variable: free-slip (1.0) or 
                                                    no-slip (-1.0).
          T  LuvSrc          Turning ON  momentum point Sources/Sinks.
          T  LwSrc           Turning ON  volume influx point Sources/Sinks.
          T  LtracerSrc(01)  Turning ON  point Sources/Sinks on tracer 01: temp
          T  LtracerSrc(02)  Turning ON  point Sources/Sinks on tracer 02: salt
          T  Hout(idFsur)    Write out free-surface.
          T  Hout(idUbar)    Write out 2D U-momentum component.
          T  Hout(idVbar)    Write out 2D V-momentum component.
          T  Hout(idUvel)    Write out 3D U-momentum component.
          T  Hout(idVvel)    Write out 3D V-momentum component.
          T  Hout(idWvel)    Write out W-momentum component.
          T  Hout(idTvar)    Write out tracer 01: temp
          T  Hout(idTvar)    Write out tracer 02: salt
          T  Hout(idDano)    Write out density anomaly.
          T  Hout(idVvis)    Write out vertical viscosity: AKv.
          T  Hout(idTdif)    Write out vertical diffusion: AKt(itemp).
          T  Hout(idSdif)    Write out vertical diffusion: AKt(isalt).
          T  Hout(idHsbl)    Write out depth of surface boundary layer.

          T  Aout(idFsur)    Write out averaged free-surface.
          T  Aout(idUbar)    Write out averaged 2D U-momentum component.
          T  Aout(idVbar)    Write out averaged 2D V-momentum component.
          T  Aout(idu2dE)    Write out averaged 2D U-eastward  at RHO-points.
          T  Aout(idv2dN)    Write out averaged 2D V-northward at RHO-points.
          T  Aout(idUvel)    Write out averaged 3D U-momentum component.
          T  Aout(idVvel)    Write out averaged 3D V-momentum component.
          T  Aout(idWvel)    Write out averaged W-momentum component.
          T  Aout(idTvar)    Write out averaged tracer 01: temp
          T  Aout(idTvar)    Write out averaged tracer 02: salt
          T  Aout(idDano)    Write out averaged density anomaly.
          T  Aout(idVvis)    Write out averaged vertical viscosity: AKv.
          T  Aout(idTdif)    Write out averaged vertical diffusion: AKt(itemp).
          T  Aout(idSdif)    Write out averaged vertical diffusion: AKt(isalt).
          T  Aout(idHsbl)    Write out averaged depth of surface boundary layer.

 Output/Input Files:

             Output Restart File:  np8_rst.nc
        Prefix for History Files:  np8_his
       Prefix for Averages Files:  np8_avg
                 Input Grid File:  np8_grd.nc
    Input Nonlinear Initial File:  np8_rst.nc
        Input Sources/Sinks File:  np8_river_clm.nc
           Input Forcing File 01:  np8_frc.nc
             Input Boundary File:  np8_bry.nc

 Tile partition information for Grid 01:  1543x0631x0030  tiling: 016x027

     tile     Istr     Iend     Jstr     Jend     Npts


 Tile minimum and maximum fractional coordinates for Grid 01:
   (interior points only)

     tile     Xmin     Xmax     Ymin     Ymax     grid

 RHO-points
   U-points
   V-points

 Maximum halo size in XI and ETA directions:

               HaloSizeI(1) =     333
               HaloSizeJ(1) =     114
                TileSide(1) =     105
                TileSize(1) =    3360


 Lateral Boundary Conditions: NLM
 ============================

 Variable               Grid  West Edge    South Edge   East Edge    North Edge
 ---------              ----  ----------   ----------   ----------   ----------

 zeta                     1   Chapman Imp  Chapman Imp  Closed       Chapman Imp

 ubar                     1   Flather      Flather      Closed       Flather

 vbar                     1   Flather      Flather      Closed       Flather

 u                        1   Rad + Nud    Rad + Nud    Closed       Rad + Nud

 v                        1   Rad + Nud    Rad + Nud    Closed       Rad + Nud

 temp                     1   Rad + Nud    Rad + Nud    Closed       Rad + Nud

 salt                     1   Rad + Nud    Rad + Nud    Closed       Rad + Nud

 Activated C-preprocessing Options:

 NP                  North Pacific Model
 ANA_BSFLUX          Analytical kinematic bottom salinity flux.
 ANA_BTFLUX          Analytical kinematic bottom temperature flux.
 ASSUMED_SHAPE       Using assumed-shape arrays.
 AVERAGES            Writing out time-averaged nonlinear model fields.
 CURVGRID            Orthogonal curvilinear grid.
 DIFF_3DCOEF         Horizontal, time-dependent 3D diffusion coefficient.
 DIFF_GRID           Horizontal diffusion coefficient scaled by grid size.
 DIURNAL_SRFLUX      Modulate shortwave radiation by the local diurnal cycle.
 DJ_GRADPS           Parabolic Splines density Jacobian (Shchepetkin, 2002).
 DOUBLE_PRECISION    Double precision arithmetic.
 INLINE_2DIO         Processing 3D IO level by level to reduce memory needs.
 LMD_CONVEC          LMD convective mixing due to shear instability.
 LMD_MIXING          Large/McWilliams/Doney interior mixing.
 LMD_NONLOCAL        LMD convective nonlocal transport.
 LMD_RIMIX           LMD diffusivity due to shear instability.
 LMD_SKPP            KPP surface boundary layer mixing.
 M2CLM_NUDGING       Nudging toward 2D momentum climatology.
 M3CLM_NUDGING       Nudging toward 3D momentum climatology.
 MASKING             Land/Sea masking.
 MIX_GEO_TS          Mixing of tracers along geopotential surfaces.
 MIX_S_UV            Mixing of momentum along constant S-surfaces.
 MPI                 MPI distributed-memory configuration.
 NONLINEAR           Nonlinear Model.
 NONLIN_EOS          Nonlinear Equation of State for seawater.
 NO_WRITE_GRID       Not Writing grid arrays into NetCDF ouput files.
 PERFECT_RESTART     Processing perfect restart variables.
 POWER_LAW           Power-law shape time-averaging barotropic filter.
 PROFILE             Time profiling activated .
 !RST_SINGLE         Double precision fields in restart NetCDF file.
 SALINITY            Using salinity.
 SCORRECTION         Surface salinity flux correction.
 SOLAR_SOURCE        Solar Radiation Source Term.
 SOLVE3D             Solving 3D Primitive Equations.
 SPLINES             Conservative parabolic spline reconstruction.
 SPONGE              Enhanced horizontal mixing in the sponge areas.
 TCLM_NUDGING        Nudging toward tracer climatology.
 TS_U3HADVECTION     Third-order upstream horizontal advection of tracers.
 TS_C4VADVECTION     Fourth-order centered vertical advection of tracers.
 TS_DIF4             Biharmonic mixing of tracers.
 TS_SMAGORINSKY      Smagorinksy-like time-dependent diffusion coefficients.
 UV_ADV              Advection of momentum.
 UV_COR              Coriolis term.
 UV_U3HADVECTION     Third-order upstream horizontal advection of 3D momentum.
 UV_C4VADVECTION     Fourth-order centered vertical advection of momentum.
 UV_QDRAG            Quadratic bottom stress.
 UV_VIS4             Biharmonic mixing of momentum.
 UV_SMAGORINSKY      Smagorinksy-like time-dependent viscosity coefficients.
 VAR_RHO_2D          Variable density barotropic mode.
 VISC_GRID           Horizontal viscosity coefficient scaled by grid size.
 VISC_3DCOEF         Horizontal, time-dependent 3D viscosity coefficient.

 CHECKADJ - use caution when activating: TS_SMAGORINSKY
            REASON: stability problems, WARNING.

 CHECKADJ - use caution when activating: UV_SMAGORINSKY
            REASON: stability problems, WARNING.

 Process Information:


 INITIAL: Configuring and initializing forward nonlinear model ...
 *******

[xiaozhu557]

Hi Kate or anybody,

Can you give me any clue for this problem? I really need your help.

Thanks.

[kate]

I would check to make sure that PERFECT_RESTART really is giving a perfect restart for the options you have chosen. I see you are getting a warning about the little-used SMAGORINSKY options. Perhaps PERFECT_RESTART doesn't work with them.

In other words:
* Check to see if PERFECT_RESTART behaves as advertised by running some short tests.
* If not, fix it.
* Check your energetics again.

I should add that step two can be accomplished by running two debugger sessions, one in a restart case, one in a case run from the beginning. Or perhaps print statements of the fields in question if you get lucky in guessing what they are, perhaps after seeing what differs in step one.

[xiaozhu557]

I would check to make sure that PERFECT_RESTART really is giving a perfect restart for the options you have chosen. I see you are getting a warning about the little-used SMAGORINSKY options. Perhaps PERFECT_RESTART doesn't work with them.

In other words:
* Check to see if PERFECT_RESTART behaves as advertised by running some short tests.
* If not, fix it.
* Check your energetics again.

Thank you for your reply. I have used the same defined options to restart the model many times, I have run my model for 10 years, 1 year each time, but it works fine in sometime, sometime not. Do you mean that PERFECT_RESTART may not work if I define the SMAGORINSKY options with it together? Are you sure this?

By the way, I have used the same forcing files for each running.

[kate]

I can be sure of nothing without trying it. PERFECT_RESTART can be checked by starting from a known state and running for a day, restarting and running for a second day. Then, start from the same known state and run for two days. Compare results. They should match perfectly.

[osua72]

Hello,

I am running into a similar problem, I'm trying to run a model for a year, outputting both hourly data and daily averages. Because of the time-limit on the super computer I need to restart the model every three days which I do using NRREC == -1. I'm running with PERFECT_RESTART defined and the results in my history file seem fine, however in the averages files I get a spike when the model is restarted and was wondering whether anybody could shed a light on why this might be:

Activated C-preprocessing Options:

IS Irish Sea
ADD_FSOBC Adding tidal elevation to processed OBC data.
ADD_M2OBC Adding tidal currents to processed OBC data.
ANA_BSFLUX Analytical kinematic bottom salinity flux.
ANA_BTFLUX Analytical kinematic bottom temperature flux.
ANA_FSOBC Analytical free-surface boundary conditions.
ANA_M2OBC Analytical 2D momentum boundary conditions.
ASSUMED_SHAPE Using assumed-shape arrays.
AVERAGES Writing out time-averaged nonlinear model fields.
BULK_FLUXES Surface bulk fluxes parameterization.
DJ_GRADPS Parabolic Splines density Jacobian (Shchepetkin, 2002).
DOUBLE_PRECISION Double precision arithmetic.
EMINUSP Compute Salt Flux using E-P.
GLS_MIXING Generic Length-Scale turbulence closure.
LONGWAVE Compute net longwave radiation internally.
MASKING Land/Sea masking.
MIX_ISO_TS Mixing of tracers along isopycnal surfaces.
MIX_S_TS Mixing of tracers along constant S-surfaces.
MIX_S_UV Mixing of momentum along constant S-surfaces.
MPI MPI distributed-memory configuration.
NONLINEAR Nonlinear Model.
!NONLIN_EOS Linear Equation of State for seawater.
PERFECT_RESTART Processing perfect restart variables.
POWER_LAW Power-law shape time-averaging barotropic filter.
PROFILE Time profiling activated .
K_GSCHEME Third-order upstream advection of TKE fields.
RADIATION_2D Use tangential phase speed in radiation conditions.
RAMP_TIDES Ramping tidal forcing for one day.
!RST_SINGLE Double precision fields in restart NetCDF file.
SALINITY Using salinity.
SOLVE3D Solving 3D Primitive Equations.
SPHERICAL Spherical grid configuration.
SSH_TIDES Add tidal elevation to SSH climatology.
STATIONS Writing out station data.
TS_U3HADVECTION Third-order upstream horizontal advection of tracers.
TS_A4VADVECTION Fourth-order Akima vertical advection of tracers.
TS_DIF2 Harmonic mixing of tracers.
UV_ADV Advection of momentum.
UV_COR Coriolis term.
UV_U3HADVECTION Third-order upstream horizontal advection of 3D momentum.
UV_C4VADVECTION Fourth-order centered vertical advection of momentum.
UV_QDRAG Quadratic bottom stress.
UV_TIDES Add tidal currents to 2D momentum climatologies.
UV_VIS2 Harmonic mixing of momentum.
VAR_RHO_2D Variable density barotropic mode.