Submitting a ROMS JOB on CHPC-IQudu

Hi All

Finally, I have ROMS running again on CHPC's iQudu cluster (e1350 IBM Linux cluster)

IQudu used to be set up with IBM's load-leveler but recently CHPC made the change to Moab. This meant starting from scratch!

I have made a testcase directory in my work directory on IQudu that a new user can copy into their work directories and use as a starting point.

/CHPC/work/nburls/ROMS_iQudu_testcase

The files of interest in it are

1) jobcomp_iqudu_intel

Note! As mentioned at the top of this script, you have to source the Intel compiler in your .bashrc script

Also note that as it is currently configured this script makes use of source code in my home directory.

set SOURCE=/CHPC/home/nburls/Roms_Agrif_v2.1_20_07_2010/Roms_Agrif

and a mvapich library (/CHPC/home/nburls/mvapich_intel) and netcdf library (/CHPC/home/nburls/netcdf_intel) in my home directory.

2) run_roms_iqudu.moab

Note! Make sure you change the following lines to point to your directory.
#MSUB -o /CHPC/work/nburls/ROMS_iQudu_testcase/roms_log.out
#MSUB -e /CHPC/work/nburls/ROMS_iQudu_testcase/roms_log.err
#MSUB -d /CHPC/work/nburls/ROMS_iQudu_testcase/

Let me know if any issues pop up. Hopefully not.

Natalie

Experiments in HYCOM

The Hybrid Coordinate Ocean Model (HYCOM) combines the optimal features of isopycnic-coordinate and fixed-grid ocean circulation models in one framework [1]. The name “hybrid” is derived from the models ability to dynamically change its vertical layer distribution between isopycnic (ρ), z-level and bathymetry following σ-coordinates, regularly adjusting to an optimal vertical structure most suitable for a specific region of the ocean. The adaptive vertical grid conveniently resolves regions of vertical density gradients, such as the thermocline and surface fronts.

Over the past few years I have been developing a nested regional 1/10° HYCOM of the greater Agulhas region (for more info, see http://www.nersc.no/~bjornb/PhdThesisBjornBackeberg.pdf). Model validation is an important aspect in developing realistic ocean simulations.

Marjolaine Rouault has been working on deriving ocean surface current velocities from the Advanced Synthetic Aperture Radar (ASAR; [2]), which is mounted on the Envisat satellite. The ASAR velocity data product has a resolution ranging from 4 – 8 km. It is a microwave sensor and its ability to “see” through clouds makes it a very powerful data set to use to map the Agulhas Current at high resolution, and hence also to validation models.

 

The figure above on the left shows the 2 year mean surface radial velocity component derived from ASAR. It is the radial component of the velocities because Envisat's ground track is -15° from North, and one can only derive current velocities from ASAR, when the current is perpendicular to the satellite ground track, which is luckily the case for the Agulhas. The black contour lines represent the 200, 500, 1000, 2000, and 4000 m isobaths.

One can see that in the southern part of the Agulhas, the current is strongly steered by the bathymetry. At the eastern edge of the Agulhas Bank, the core of the current closely follows the 1000 m isobath.

The above figure to the right shows the 2 year mean radial velocity component derived from HYCOM version 2.1. Two noticeable differences are evident:

1. The current velocities of the southern Agulhas Current simulated in HYCOM are markedly weaker (almost a factor of 3 difference) than those derived from ASAR.
2. At the eastern edge of the Agulhas Bank, the current follows the 2000 m isobath instead of the 1000 m isobath, as suggested by the ASAR observations.

In HYCOM, each vertical layer is assigned a reference density, which is the density it reverts to once when changing from fixed vertical coordinates to isopycnic coordinates. The reference densities in version 2.1 of HYCOM were chosen to mimic the reference densities of the parent model supplying the lateral boundary conditions, which range from 21.0 to 28.3 kg/m³. Plotting the vertical distribution of the layers in HYCOM 2.1 (not shown) it is evident that too many layers are found within the mixed layer and the upper 200 m, and only 7 layers remain to simulate the remainder of the water column, which suggests that the reference densities selected are inadequate for the Agulhas region.

Version 2.1 of HYCOM applies a relatively crude vertical interpolation scheme to the mixed layer, which causes enhanced, and artificial, diapycnal mixing within the mixed layer that may diffuse the core of the current.

Recently, we upgraded HYCOM to version 2.2, and some new and improved features include:

• A new GISS mixed layer scheme
• WENO-like PPM interpolation of the mixed layer
  (WENO = Weighted Essentially Non-Oscilatory; PPM = Piecewise Parabolic Method)
• Bottom layer KPP
• Slow evolution of the barotropic mode, which allows us to double the barotropic time-step.

Then, to test the importance of the vertical layer distribution, I ran two experiments using version 2.2 of HYCOM:

expt01.0: Using the same reference densities as in HYCOM 2.1

expt01.1: Adjust the reference densities to a range more suitable for the Agulhas region. Based on potential density observations from the WOCE transect I6, a new density range from 23.6 to 27.6 kg/m³ was chosen. The vertical resolution resolution was increased between 23.5 and 26.8 kg/m³ to capture the salinity maximum in the Mozambique Channel at approximately 150 – 300 m, and between 27.1 and 27.7 kg/m³ to capture the salinity minimum in the South Atlantic at approximately 600 – 1200 m.

The below figures show the 2 year mean surface radial velocity component derived from expt01.0 and expt01.1 (left and right respectively). The model was run for 2 years only, so is not spun up, these are preliminary tests only.

However, these preliminary results indicate that implementing the new version of HYCOM increases the velocities of the simulated current. There is also some improvement in HYCOM's ability to simulate an Agulhas Current following the 1000 m isobath along the eastern edge of the Agulhas Bank, despite using the same reference densities as in version 2.1.

Qualitatively comparing HYCOM 2.2 expt01.0 and expt01.1, indicates that adjusting the reference densities has an effect on the position on the mean flow of the Agulhas Current. Expt01.1 seems to follow the 1000 m isobath more closely, although in general the current remains too wide compared to the ASAR derived observations. The below results highlight the importance of the vertical discretisation of the model grid when simulation the ocean.

 

These experiments with HYCOM 2.2 have been carried out using a 2^nd order momentum advection scheme, and it has been shown, using HYCOM 2.1, that a 4^th order momentum advection scheme has a significant impact on the solution [3]. 

 

The 4^th order momentum advection scheme uses a super-slip condition at the coast. Presently I am testing a simulation using the free-slip condition at the coast, to see whether this has an impact on the simulation of the Agulhas.

References

1. Bleck, R.: An oceanic general circulation model framed in hybrid isopycnic-Cartesian coordinates, Ocean Modell. 37, 55–88, 2002
2. Rouault, M. J., A. Mouche, F. Collard, J. A. Johannessen and B. Chapron: Mapping the Agulhas Current from space: an assessment of ASAR surface current velocities, J. Geophys. Res., 115, C10026, doi:10.1029/2009JC006050
3. Backeberg, B. C., Bertino, L., and Johannessen, J. A.: Evaluating two numerical advection schemes in HYCOM for eddy-resolving modelling of the Agulhas Current, Ocean Sci. 5(2), 173–190, 2009

ROMS bulk forcing update

In case anyone is interested I now have a version of ROMS_TOOLS that will accommodate the CORE 2 forcing data set. The data-set needs to be pre-prepared top put all variables on a consistent time-step and separate it into monthly blocks. Roms_blk files now contain no wind stress variables (redundant space) and specific rather than relative humidity. Only downward long-wave radiation is included - black body radiation is corrected for in the bulk routine. Shortwave radiation is corrected for with a flat albedo for sea-water (0.065). The ROMS algorithms have been adapted to cope with specific humidity.

I have also engineered a way to allow ROMS to accept an anomaly on the fly (labelled 'anom' in roms_anm.nc), this way anomalies can be applied to data sets without rebuilding the forcing fields. Let me know if this is of use - the anomaly I have made is currently just applicable to wind stress.

Lastly, we now have the entire CORE2 data-set, normal year and inter-annual series, both in corrected forms. I am currently working on adapting the boundary routines to take data from a few ORCA configurations, which we now have the data for in house (contact me if you want more details on this: configs are ORCA05 and ORCA2 from the Kiel Climate Model).

If anyone spots that I have done something a bit stupid above, please let me know!

Cheers

Ben

Getting email notifications for this blog

To get email notifications of new posts / comments in this blog you need to join the Google group:

http://groups.google.com/group/uct-ocean-modelling

Members of this group will receive email notifications when the UCT ocean modelling blog is updated.

Unfortunately this will only send you an update when a new POST is created, not when people comment on posts.

It is possible subscribe to the blog using the link at the bottom of the blog "Subscribe to: Posts (Atom)".