Blow up in Second step

[sun_92]

Dear All
I'm stuck at one point where I'm trying to run a standalone simulation for one month using ROMS_3.7. I have tried two different grids and it is showing diag speed trouble for one grid and a blow up in the first step for the other grid. My DT is 30 seconds. I'm trying to initialise the model using INCOIS GODAS data for the month of May 2020. I could successfully prepare the initial and boundary files using matlab tool box. However for two different types of grid, I'm getting a blow up and diag speed error. I guess there is some fundamental error. I'm attaching the log files of runs using both grids. Kindly help me with this. Previously I have done simulations with the same model, have not encountered these kind of errors.

[wilkin]

I see a number of things in your logfiles that are a bit unconventional and you might consider adjusting.

Both runs have both these defined:

Code: Select all

UV_VIS2                  Harmonic mixing of momentum
UV_VIS4                  Biharmonic mixing of momentum

I recommend you start with only #undef UV_VIS4 until to show you need it. Especially since you have #define UV_DRAG_GRID and #define ANA_DRAG_GRID. Do you really have an analytical specification of spatially varying linear drag? If that's on purpose, check that those coefficients are sensible and you don't have a parallel bug in the analytical code crossing tile boundaries.

You still have #define UPWELLING which activates certain options in the ana_*.h functionals and made be overshadowing your own customizations (like the bottom drag). It is customary to set one's own ROMS_APPLICATION name and use that in build_roms.sh (export ROMS_APPLICATION=UPWELLING) in place of the examples. Look at Build/analytical.f90 to check if UPWELLING is giving you code you did not intend.

You have Flather and Chapman OBC on ubar and zeta, and analytical boundary ubar,vbar (in roms_test2.txt) but clamped u,v. Those can't possibly be consistent (i.e vertical average of u,v won't be consistent with ubar,vbar. I recommend radiation on u,v until you get this running. Or, vertically average your u,v in ROMS discrete coordinates and make sure that is the ubar,vbar in the boundary file.

You have clamped T,S boundary conditions, yet the range of values on e.g. the east boundary is

Code: Select all

  GET_NGFLD_NF90   - temperature eastern boundary condition,              2020-05-01 00:00:00.00
                      (Grid= 01, Rec=1, Index=2, File: roms_bry_amphan.nc)
                      (Tmin=       7426.0000 Tmax=       7456.0000)   t =       7426.0000
                      (Min =  0.00000000E+00 Max =  3.20222588E+01)

That zero might be in the land mask, but be careful that you aren't clamping T to zero at one water point and a value closer to 32 immediately next to it. That will introduce very strong horizontal density gradients along the boundary.

You mix advection schemes on the active tracers.

Code: Select all

Tracer Advection Scheme: NLM
 ========================
 Variable               Grid  Horizontal         Vertical
 ---------              ----  ------------       ------------
 temp                     1   Upstream3          Centered4
 salt                     1   HSIMT              HSIMT

That's not fundamentally wrong, but we have seen this to give unstable behavior. Try making these the same until you get this running.

In test 1 you have both of

Code: Select all

MIX_GEO_TS               Mixing of tracers along geopotential surfaces
 MIX_S_TS                 Mixing of tracers along constant S-surfaces

Just use #define MIX_GEO_TS. You don't want both (ROMS might not give you both, but don't chance it).

Is there a good reason to use this?

Code: Select all

VISC_3DCOEF              Horizontal, time-dependent 3D viscosity coefficient

Start simple and #undef that.

Be careful for #define SRELAXATION. There are some posts of mine from last year with a user having trouble with this option. Again, start without so many options activated until you get a starting stable run.

The fact that you have a NaN in the kinetic energy on the first time step ...

Code: Select all

TIME-STEP YYYY-MM-DD hh:mm:ss.ss  KINETIC_ENRG   POTEN_ENRG    TOTAL_ENRG    NET_VOLUME
                     C => (i,j,k)       Cu            Cv            Cw         Max Speed

         0 2020-05-01 00:00:00.00           NaN  1.630508E+04           NaN  9.985486E+15
                     (063,163,40)  1.679527E-03  2.074747E-03  0.000000E+00  8.896352E-01

says to me you initialized a NaN in velocity, possibly in the land mask but that is still not allowed. If you have a _FillValue in your initial conditions velocity that may unwittingly introduce a NaN in velocity.

[sun_92]

Thank you very much for your very detailed reply. I will try these and will post the outcome of my run in the same section. Good day!

[sun_92]

Dear Sir

You have Flather and Chapman OBC on ubar and zeta, and analytical boundary ubar,vbar (in roms_test2.txt) but clamped u,v. Those can't possibly be consistent (i.e vertical average of u,v won't be consistent with ubar,vbar. I recommend radiation on u,v until you get this running. Or, vertically average your u,v in ROMS discrete coordinates and make sure that is the ubar,vbar in the boundary file.

You have clamped T,S boundary conditions, yet the range of values on e.g. the east boundary is

I couldn't understand what this means. However, I have changed the conditions to RadNud for T,S, U and V. I have made changes to the run mentioned in test2 (as per your suggestions) and this is the log file. As per your suggestion, I have renamed my application file to BOB to avoid mismatch in the options. The point (035,17,32) in which the speed is beyond 20 m/s looks like water point only. Am i missing some fundamental point?

[wilkin]

Now you have deactivated all horizontal mixing. You have neither #define TS_DIF2 not #define TS_DIF4, which is why no horizontal mixing coefficient values nl_tnu2 are reported to the log. #define MIX_GEO_TS

You do have some horizontal viscosity (#define MIX_GEO_UV) but the viscosity coefficient is zero in the log:

Code: Select all

 0.0000E+00  nl_visc2          NLM Horizontal, harmonic mixing coefficient
                                 (m2/s) for momentum.

I recommend #define MIX_S_UV (not GEO). You had that before but changed it.

You choice of momentum advection scheme is notoriously noisy

Code: Select all

 UV_C2ADVECTION          Second-order centered differences advection of momentum

and with zero viscosity to control the noisiness I would expect this to blow-up, which it did. Just use the default advection scheme for momentum.

In a change from the log for test2, this latest is reporting old cpp defs that have been removed

Code: Select all

 TS_C4HADVECTION         Fourth-order centered horizontal advection of tracers
 TS_C4VADVECTION         Fourth-order centered vertical advection of tracers

which explains why the log no longer reports the advection scheme. Looks like you compiled a different version of the code for this experiment.

In test2 you had STATIONS and now you have no stations but have FLOATS. Wait until you get your run stable before you mess with all that. I'm using diff to compare your log files and it's tricky to make sense of it all with all these changes.

Yes, you should look at what's going on at point (35,17,32). You'll want to plot at every time step because vertical velocity is going nuts right away.

TIME-STEP YYYY-MM-DD hh:mm:ss.ss KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
C => (i,j,k) Cu Cv Cw Max Speed

0 2020-05-13 00:00:00.00 9.447909E-04 1.493786E+04 1.493786E+04 6.898608E+15
(192,006,32) 1.724965E-03 2.002071E-03 0.000000E+00 9.554885E-01
1 2020-05-13 00:00:30.00 1.171937E-03 1.511131E+04 1.511131E+04 6.819455E+15
(166,047,32) 7.491161E-03 8.240924E-03 7.429613E-06 3.397028E+00
2 2020-05-13 00:01:00.00 1.749324E-03 1.511126E+04 1.511126E+04 6.819455E+15
(035,017,32) 1.676149E-04 5.337860E-04 1.095465E+03 6.325016E+00
3 2020-05-13 00:01:30.00 2.615535E-03 1.511118E+04 1.511118E+04 6.819454E+15

You might to adjust the bathymetry or land mask there. It's impossible to tell from the logs.

But add some horizontal viscosity first. That should help.

[sun_92]

I tried with all of these. now the MODEL runs for 200 time steps and then stops. Could someone tell what could be the possible reasons for the increase in maximum speed. My initial and boundary files seems to be fine and without NaN's. I'm doing the simulation for BoB region with south, west open boundary and closed north and east boundaries. The forcing are also fine and are provided every six hour. Have tried running with dt =30 and dt=10 I would like to know what else should be taken care while setting up the simulations. I'm attaching the log file.

[wilkin]

196 2020-05-01 00:32:40.00 3.869347E-01 1.555930E+04 1.555969E+04 8.325429E+15
(225,061,23) 3.047532E-04 2.696655E-04 5.290505E-01 1.944507E+01
197 2020-05-01 00:32:50.00 3.923971E-01 1.555928E+04 1.555967E+04 8.325429E+15
(225,061,23) 3.276896E-04 2.693862E-04 5.415630E-01 1.996086E+01
198 2020-05-01 00:33:00.00 3.979175E-01 1.555925E+04 1.555965E+04 8.325429E+15
(225,061,23) 3.519085E-04 2.693302E-04 5.541660E-01 2.049019E+01
Found Error: 01 Line: 298 Source: ROMS/Nonlinear/main3d.F
Found Error: 01 Line: 298 Source: ROMS/Drivers/nl_ocean.h

Blowing-up: Saving latest model state into RESTART file

WRT_RST - wrote re-start fields (Index=1,1) in record = 0000003

The log gives you some guidance that the model is blowing up near point i=225, j=61, k=23.

ROMS writes the model state at blow-up to record 3 of your RESTART file.

So, plot those data and see what is going on. Usually these issues are solved by modifying the bathymetry or land/sea mask near points that are causing extreme velocity. It does not look like you have strong horizontal velocities, so it's probably vertical. Some smoothing of steep bathymetry might be in order.

[sun_92]

Dear All
I'm attaching the screen shot of my grids made using 25 km resolution for whole NIO with east and north closed and for Bay of Bengal with 9 km resolution keeping east and north closed. I have used the modified ETOPO which most of literature suggests. I have used theta_s and theta_b as 7 and 0.1 with N=40 and TCline as 250. I have even tried with TCline = 150. For both these grids keeping the options similar, I have used dt = 10 ad dt = 30. While the run progresses for abput 300 time steps with dt = 10 , it stops after 100 time steps with dt = 30. I have been trying different grid making softwares such as roms_tools and gridbuilder for exactly solving the issue. I suspect that it could be the problem of the grid while the boundary and initial files looks ok. The forcing values are also reasonable. The only issue could be grid or the bathymetry. As suggested , I have used modified ETOPO instead of standard ETOPO which has adapted by most of the studies in these regions. Kindly help and I would be forever grateful if someone could help me solve this issue.

[sun_92]

Dear Dr Wilkin,
I have solved this error. The issue was with the making of initial file and I have rectified it. Now the model is running fine. Me and my team would like to thank you for your valuable suggestions which has helped us in improving our set up.

[jz6791]

Dear Dr Wilkin,
I have solved this error. The issue was with the making of initial file and I have rectified it. Now the model is running fine. Me and my team would like to thank you for your valuable suggestions which has helped us in improving our set up.

Dear Sun_92,
I'm a new user of ROMS and got a very similar issue with you when I plan to try different horizontal resolutions. My model setup works fine for 1 and 2km horizontal resolution, but blows up in the second time step in coarser resolution (5 and 10km). My DT is 50s. So could u plz have look about the log file of 5km version?

Thank you so much .
Best regards
JZ

[sun_92]

Hello
The options used and the values of input files are not seen in the log. Without that it is not possible for me to figure out what exactly goes wrong and why the blow up. Usually if the speed is above 20 m/s, there will be a blow up which can be adjusted by smoothing the bathymetry and by checking for bad values in your input files. Schemes used may also give you spurious results, therefore try to change the options as mentioned in the post by Dr Wilkin and make sure all your input files are good and without bad values. Also please plot and see the point which gives you the blow up to see if something is going wrong.

[jz6791]

Sorry Im a new user of ROMS, maybe my .in file covers what you want?
Yes, Im following Dr Wilkin's suggestion and trying to find where is the bad value. By a quick check, max u and v are not exceed 7m/s.

Thanks

[jz6791]

Also, what I want to is to test model sensitivity to different resolution. So I keep everything the same, run the model with 1km, 2km, 5km and 10km resolution. It works fine for 1 and 2km but blows up for 5 and 10km in ~58S. I changed the TNU2 and VISC2 but didnt help.