Submitting a ROMS JOB on the Oceanography Department's High Performance Computing Facilities - hosted by the Climate Systems Analysis Group

Hi

I have run a ROMS test case on the Oceanography Department's new High Performance Computing Facilities - core.csag.uct.ac.za

For useful information and guidelines on using the cluster I suggest you read the “Guidelines for the use and administration of the Oceanography Department's High Performance Computing Facilities - hosted by the Climate Systems Analysis Group” which should be given to you once an account has been created for you on the cluster. Information on the Queue management system (Sun Grid Engine) and shared libraries can be found in this document.

I have made a testcase directory in my scratch directory that a new user can copy into their scratch directory and use as a starting point - /scratch/ocean/nburls/ROMS_CORE_testcase

The files of interest in it are:

1) jobcomp_mpich2_intel64

Note! As mentioned at the top of this script, you have to source the Intel compiler in your .bashrc script or in the jobcomp script.

# source /share/apps/intel/Compiler/11.1/073/bin/ifortvars.sh intel64

# source /share/apps/intel/Compiler/11.1/073/bin/iccvars.sh intel64

Also note that as it is currently configured this script makes use of source code in my home directory,

set SOURCE=/home/nburls/Roms_tools/Roms_Agrif

and the mvapich library,

mva_base=/share/apps/mpich2-1.3.2p1-intel64

and netcdf library,

set NETCDFLIB=-L/share/apps/netcdf-3.6.3-intel_intel64/lib

set NETCDFINC=-I/share/apps/netcdf-3.6.3-intel_intel64/include

2) run_roms_CORE_joinfiles.csh OR run_roms_CORE_nojoin.csh

Currently set to run on 8 processors.

Using

set NCJOIN=/home/nburls/bin/ncjoin

set PARTIT=/home/nburls/bin/partit

As per Nicolette's previous post the run_roms_CORE_nojoin.csh run script does not rejoin the average and diagnostics output files, while run_roms_CORE_joinfiles.csh does.

See Nicolette's SUN Runscripts - http://uctoceanmodelling.blogspot.com/2011/02/update-of-sun-cluster-running-scripts.html

Let me know if any issues pop up. Hopefully not.

The next step will be to get the matlab licence server up and running on the CORE cluster so that one can do ROMS pre and post processing in batch mode on the cluster.

Natalie

Update of SUN cluster running scripts

I have updated/written a few scripts for running on the SUN cluster at the CHPC. They are available in my work directory under "runscripts".

run_roms_l.moab

• The previous run script waited for the tiles to be joined before timestepping of the next month could proceed. This holds up all the processors as this is a serial (1 CPU) job.
• Changes: the tiles are renamed with the dates for joining later in the next script
• Restart and history files are still joined as these files are small and quick to join and so you can monitor your model run.

join_files.moab

• This takes the renamed tiles created above and joins them with ncjoin.
• This is done in parallel so can be submitted to the SUN, else it can run in serial on your desktop by changing the number of parallel processes.
• Careful of diagnostic files. I created a monster 28GB average file!!!!!

rename_files.sh or rename_files.moab

• When the model crashes due to MPI error or something, the timestepping and the file naming become out of sync. This renames all the files with the correct dates after the run.
• This is a serial procedure. The .moab is so you can submit it to the cluster.

I've tested writing, joining and renaming monthly average files. The scripts produce log and error files. I haven't tested for nesting. Read the description in the file headers.

Let me know if I missed something.
Nicolette

Ncview: A quick and easy visual browser for netcdf files.

Hi All,

Im sure this is probably boring old news to most of you...but at the risk of being redundant I thought I would share just incase.

Ncview is a quick and easy way of viewing netcdf formatted files. No code required just a click of a button. Ncview typically needs to be run in a linux based environment (mac os x works nicely too). It can open up several data files at once and will run a little movie of all your data if you so fancy. It also has a range of different colour bars you can choose from.

For more information and the download:

http://meteora.ucsd.edu/~pierce/ncview_home_page.html

Sarah