Sunday, 20 December 2015

A Quick Start to Using SMAUG

Guidelines for Using Sheffield Magnetohydrodynamics Algorithm Using GPUs (SMAUG)

Introduction

Parallel magnetohydrodynamic (MHD) algorithms are important for numerical modelling of highly inhomogeneous solar and astrophysical plasmas. SMAUG is the Sheffield Magnetohydrodynamics Algorithm Using GPUs. SMAUG is a 1-3D MHD code capable of modelling magnetised and gravitationally stratified magnetised plasma. The methods employed have been justied by performance benchmarks and validation results demonstrating that the code successfully simulates the physics for a range of test scenarios including a full 3D realistic model of wave propagation in the magnetised and stratified solar atmosphere. For details about smaug see the preprint at: smaugpreprint
SMAUG is based on the Sheffield Advanced Code (SAC), which is a novel fully non-linear MHD code, designed for simulations of linear and non-linear wave propagation in gravitationally strongly stratified magnetised plasma. See the reference at the Astronomy Abstracts Service. sacpaper
The smaug code has been developed by the Solar Wave theory group SWAT at The University of Sheffield. Researchers and users downloading the code are requested to acknowledge and give credit to the Solar Physics and Space Plasma Research Centre (SP2RC) at The University of Sheffield.

Requirements

CUDA-Enabled Tesla GPU Computing Product with at least compute capability 1.3.
See
http://developer.nvidia.com/cuda-gpus
CUDA toolkit
https://developer.nvidia.com/cuda-downloads
The SMAUG has been developed and tested on a range of different 64 bit (and 32 bit ) linux platforms.
Guidelines for correct installation of the CUDA toolkit and drivers can be found at cudagettingstarted

Installation

SMAUG can be downloaded in 3 ways
  • Download and extract the latest tarball from the google code site
  • Checkout the distribution from the user release version from the google repository. This is recommended if you require regular updates and bugfixes.
  • Checkout the distribution from the developer repository. This is recommended if you wish to participate in the development of future code versions.
Method 1:
Download the distribution Link to revision 257 source code
Copy the distribution to a suitable directory in your working area and extract the distribution tar -zxvf smaug_v1_rev255.tgz
Method 2:
Create a directory and from that directory,
using a subversion client checkout the latest distribution using the command:
svn checkout https://ccpforge.cse.rl.ac.uk/gf/project/sac/scmsvn/?action=browse&path=%2Frelease%2Fsmaug%2F
when prompted, Password for 'anonymous' just press return.

Building and running a Model

From the distribution base directory change directory to the src folder.
Building a smaug based simulation requires the use of the make utility. The make may be tuned for a particular platform by editing the include line near the top of the file.
The default input file is make_inputs. If you are building an MPI distributionthen edit this line and use make_inputs_mpi. Any particular options can be edited within the make_inputs file. It is most likely that the library path for the cuda libraries is set correctly. This can easily be changed by editing the CUDA.
The make_inputs file includes a number of compiler switches the smaug specific switches used for both the host compiler and cuda compiler are as follows.
USE_SAC            //Set this to build and compile 2D models
USE_SAC_3D         //Set this to build and compile 3D models
USE_MULTIGPU       //Set this to build for multi GPU models (e.g. if you are using MPI)
USE_MPI            //Set this if you are using MPI (need to set the host                           compiler to a suitabel MPI compiler)
USE_USERSOURCE     //Set this if you are using user provide source terms
The cuda specific switches are as follows
--ptxas-options=-v Provide verbose output when generating CUDA code -arch sm_20 Set the correct CUDA architecture (sm_20 allows printf debugging in CUDA kernels) -maxregcount=32 Set the numbe of register variables for the CUDA compiler
To make the Brio-Wu test (use the following commands)
make clean make bw make sac
Change back to the distribution base directory
Run the model
./iosmaug a
As each step is run the program outputs the current simulation time step, iteration and the time taken to compute that step. Generated configuration files are written to the out directory. These may be visualised with IDL using the procedures in the Idl directory.
For the Brio-Wu test use
visex22D_BW_test1.pro
For the Orszag-Tang test use
visex22d_cu_OT_test1.pro
The test models available are as follows. The code used with make to make the model is shown in the second column
1d Brio-Wu bw
2d Orszag-Tang ot 2d Kelvin-Helmholtz test kh
These tests have pre-prepared configuration files and should run by default. Configuration files may be generated using either SAMUG or the vacini routine with VAC or SAC. SMAUG generates binary output files but currently takes as input ascii configuration files. IDL procedures in the Idl folder are available to translate configuration data as required.

Modifying The Run Parameters

To change the input parameters edit the file iosmaugparams.h in the include folder. At any time you can revert to the default parameters for the OT test (or a particular model) by changing to the src folder and issuing the command
make ot
As soon as the parameter files have been updated, move to the src folder and recompile the model using
make smaug
Move back to the base directory and run the model.
The following parameters can be altered in the iosmaugparams.h file
ni,nj,nk           //size of the domain (note this will be shifted by 2X the number of ghost cells)
xmax, ymax, zmax   //the physical domain size for each direction
xmin, ymin, zmin

cfgfile             //The name and path of the input configuration file
cfgout              //The name and path of the output configuration file              (each file generated will
                     //be appended with an integer denoting the step index)

dt                  //The time step (if using fixed)
nt                  //Number of iterations to perform

p->gamma            //The adiabatic constant
p->courant          //The courant parameter used to determine the time step parameter
p->moddton          //Set to 0.0 for fixed time steps setto 1  to enable
p->hyperdifmom      //Set to 1.0 to switch hyperdiffusion stabilisation on 0.0 disables
p->readini          //Set to 1.0 to read initial configuration from an input file. 
                    //The 0.0 will generate a configutation using the default if provided
                    //or written by the user.
The hypediffusion parameters may be altered slightly but it is recommended to leave them at their default tuned settings.
p->chyp[rho]=0.02;
p->chyp[energy]=0.02;
p->chyp[b1]=0.02;
p->chyp[b2]=0.02;
p->chyp[mom1]=0.4;
p->chyp[mom2]=0.4;
p->chyp[rho]=0.02;
Boundary types are also set here according to field variable, direction and top or bottom boundary. The boundary conditions may also be user configured as briefly outlined in the following section.

Guidelines for Users Developing Customised Models

Users wishing to develop customised models require a basic knowledge of C programming. An indepth knowledge of CUDA programming is not required.
The following source files may be modified by the user.
iosmaugparams.h init_user.cu boundary.cu usersource.cu initialisation_user.h
The above files can be appended with a .mymodelname and stored in the models folder. The Makefile should be updated by including the following lines.
mymodelname:
cp ../models/iosmaugparams.h.mymodelname ../include/iosmaugparams.h
cp ../models/init_user.cu.mymodelname ../src/init_user.cu
cp ../models/boundary.cu.mymodelname ../src/boundary.cu
cp ../models/usersource.cu.mymodelname ../src/usersource.cu
cp ../models/initialisation_user.h.default ../include/initialisation_user.h
The custome model can then be set using
make mymodelname
Further details about building and compiling use defined models will be provided on line.
iosmaugparams.h Contains parameter settings init_user.cu Contains code which can be used to initialise a configuration on the GPU boundary.cu Allows the user to define which boundary conditions called and how they will be called usersource.cu Allows the user to include additional source terms (for example velocity driver terms) initialisation_user.h Allows the user to provide custom code generating a configuration on the host machine.
This is useful when a user needs to generate configurations scattered across multiple GPU's

Help Support

The developers may be contacted directly

SMAUG is the Sheffield Magnetohydrodynamics Algorithm Using GPUs

Parallel magnetohydrodynamics (MHD) algorithms are important for numerical modelling of highly inhomogeneous solar and astrophysical plasmas. SMAUG is the Sheffield Magnetohydrodynamics Algorithm Using GPUs. SMAUG is a 1-3D MHD code capable of modelling magnetised and gravitationally stratified magnetised plasma.

The methods employed have been justified by performance benchmarks and validation results demonstrating that the code successfully simulates the physics for a range of test scenarios including a full 3D realistic model of wave propagation in the magnetised and stratifi ed solar atmosphere.
For details about smaug see the preprint at:smaug-preprint
 
SMAUG is based on the Sheffield Advanced Code (SAC), which is a novel fully non-linear MHD code, designed for simulations of linear and non-linear wave propagation in gravitationally strongly stratified magnetised plasma.

See the reference at the Journal of Astrophysics and Astronomy. SMAUG paper springer DOI: 10.1007/s12036-015-9328-y

See the reference at the Astronomy Abstracts Service. SMAUG paper at ADS
 
See the reference at the Astronomy Abstracts Service. SAC paper at ADS
 
A quick start guide to using smaug is available: Quickstart
 
The SMAUG code has been developed by the Solar Wave theory group SWAT at The University of Sheffield. Researchers and users downloading the code are requested to acknowledge and give credit to the Solar Physics and Space Plasma Research Centre (SP2RC) at The University of Sheffield.
For example
"Courtesy: SP2RC, University of Sheffield"

Monday, 14 December 2015

Space Weather Forecasting Using Photospheric data

This semester we have had two excellent seminars on space weather forecasting using sunspot and x-ray observations of the sun. The reported investigations have  focused on  methods for predicting solar flares using studies of X-ray emission data from satellites such as GOES and RHESSI. By investigating the  variation of the sunspot distributions using datasets such as the Debrecen Data (SDD) it is possible to forecast solar flares significantly in advance of current methods.

Solar flares and CME's are both explosions that occur on the sun the events are quite different. Solar flares are classified as A, B, C, M or X according to the peak flux (in watts per square metre, W/m2) of 100 to 800 picometre X-rays near Earth, as measured on the GOES spacecraft.

Classification Peak Flux Range at 100-800 picometre

(Watts/square metre)
A < 10−7
B 10−7 – 10−6
C 10−6 – 10−5
M 10−5 – 10−4
X > 10−4

The following SDO video provides an interesting visual overview of the different kind of events.


An example HMI magnetogram of an active region is shown below shortly after the occurence of a C2.7 class flare which occurred at 2036. We observe two groups of opposite polarity in close proximity.
The active region NOAA AR12443 gave rise to a number of flare events and a CME of class M1.0 shown in the video below



A 171, LASCO C2 (2015-10-30 05:48:10 - 2015-11-01 05:29:10 U

This movie was produced by Helioviewer.org. See the original at http://helioviewer.org/?movieId=296d5 or download a high-quality version from http://helioviewer.org/api/?action=downloadMovie&id=296d5&format=mp4&hq=true

Understanding flaring phenomena requires a detailed study of the magnetic field configurations and the study of reconnection phenomena. Although coronal loops are remarkably stable for long periods it is a study of instability which can reveal the mechanisms resulting in the generation of solar flares. Understanding the flaring events requires a detailed study of the stability of the magnetic flux ropes participating in reconnection events, also important is the interaction of the flux rope photospheric footpoints.


The animated gif below, illustrates the change in the configuration of the magnetic field occuring during a reconnection. Such processes may be driven by huge mechanical motions of more dense plasma within the photosphere.



The discussion of the Debrecen sunspot data collection presented the new insights into pre-flare and Coronal Mass Ejection behavior and the evolution of the Active Regions (ARs). This was achieved by analysing the SOHO/MDI data and the  Debrecen Data  sunspot catalogues. This is a statistical study of the spatio-temporal distribution of precursor flares before major solar flares, how is this achieved and how does it link with the physical phenomena generating the events.

In earlier work measurements of the umbral area of sunspots and the mean magnetic field from the Debrecen data over a 10 year period produced a logarithmic distribution, as follows;


 
The curve fit is satisfied when |K1| = 265 gauss and |K2| = 1067 gauss. This function relates a Bmean magnetic field to an A umbral area.  

The proxy measure used to represent the magnetic field gradient between two spots of opposite polarities having areas A1 and A2 and at a distance d, is defined as:



The study of the sunspot magnetic fields and the associated gradient GM revealed a number of interesting features. There is a rise in GM of around 2 days prior to an event. The maximum in GM is around 3MWb/m, after the maximum and before the flare event this decreases, the fluctuations during this phase are indicative of the pre flare dynamics. Of 57 events studied, for half of them the flare occured within 10 hours after the maximum of GM.

 In the second paper the method was updated and employed the weighted horizontal gradient of the magnetic field (WG_M) defined between opposite polarity spot-groups at the polarity inversion line of active regions (ARs). This parameter provides important diagnostic information about the accurate prediction of onset time, on the flare intensity and towards CME risk assessment from C class to the X class flare.


The revised method computes the flux gradient for sunspot groups in addition to confirming the earlier results, the new method improves the information which can be obtained about flare events. If the maximum flare energy is less than 42% of the energy stored in the group further flares are more likely. The method improves the estimation of the onset time for flare events.

http://www.solarmonitor.org/index.php?date=19991127&region=08771


NOAA AR 8771


http://www.solarmonitor.org/index.php?date=20010326&region=09393


NOAA AR 9393

NOAA AR 8771, for 1999 November 23–26. Right column: continuum white-light image (top), reconstruction from SDD (middle), and magnetogram (bottom). Left column: variation of (top), distance between the area-weighted centers of the spots of opposite polarities (middle), and unsigned flux of all spots in the encircled area (bottom).
  but of NOAA AR 9393 with a single (i.e., X1.7) and multiple (i.e., X1.4 and X20) flares, for 2001 March 26 April 3.

For these studies a lot of use was made of potential field theory and non-linear force field theory to predict the fields at different levels from magnetograms  with the proxies these were then used to predict flare occurrences 10-12 hours in advance.

Further studies of the variations of solar non-axisymmetric activity have been used as predictors of geomagnetic activity. The dynamic properties of the active longitudes may be of predictive importance. The most flare-productive active regions tend to be located in or close to the active longitudinal belt. This may allow to predict the geoeffective position of the domain of enhanced flaring probability for a couple of years ahead. The magnetic flux emergence exhibits a fluctuation of 1.3 years within the active belt, this fluctuation is absent out of this belt.  The observed behaviour may allow an estimation of  the time intervals of higher geomagnetic activity for a couple of months ahead.

 Solar Flare Statistics (key presentation for mac)

Flare Prediction by Sunspot Dynamics
 http://solarwavetheory.blogspot.co.uk/2014/10/flare-prediction-by-sunspot-dynamics.html

 SOHO/MDI - Debrecen Data (SDD)

 On Flare Predictability Based on Sunspot Group Evolution  arxiv
 Dynamic Precursors of Flares in Active Region NOAA 10486 arxiv

Friday, 11 December 2015

Experiences with the TRISTAN-MP

Tristan-mp (and on TRAC ) is a massively parallel, fully relativistic, Particle-In-Cell code for plasma physics applications. Inspired by a tutorial by Rony Keppens on the induction equation and the Petschek model of magnetic reconnection  ( sample models for VAC )

This code is barely documented and this posting provides a few clues on how to get going!

Recipe for Building Tristan-MP

We built the hdf5 libraries and compiled your code in the home directory. It seems there is no error with a missing library problem mentioned by a colleague.
 
Just a brief instruction of building hdf5 libraries and compiling Tristan-MPI is here:
1- inside hdf/hdf5-1.8.9 folder
 
- make clean
- module add mpi/gcc/openmpi/1.4.4
- ./configure --prefix=/home/usr/hdf/hdf5-1.8.9 --enable-fortran --enable-parallel --enable-unsupported --enable-debug=all
- make
- make check
- make install
- make check-install
 
2- compiling Tristan-MPI
 
- inside Tristan-MPI
- module add mpi/gcc/openmpi/1.4.4
- go to source
- make clean
- make
 
* Note: We editted Makefile which is inside Tristan-MPI/source :
e.g. :
 
FC= /home/usr_dir/hdf/hdf5-1.8.9/bin/h5pfc
LD= /home/usr_dir/hdf/hdf5-1.8.9/bin/h5pfc
INCPATH= -I/usr/local/mpi/gcc/openmpi/1.4.4/include
 
and also we added LIBPATH to the Makefile
 
LIBPATH = -L/usr/local/mpi/gcc/openmpi/1.4.4/lib -L/home/usr_dir/hdf/hdf5-1.8.9/lib -lmpi -lmpi_f90  -lmpi_f77 -lhdf5_fortran -lhdf5
You can do the same for your own makefile.
 
Guidelines on building HDF5 are on the HDF5 site.
 
A link to  the Makefile we use is here

 Running the Code and Analysing the Ouput

The compiled model is run using an MPI with the following command embedded in a script file
mpirun tristan-mp2d
 
Loading the output was easy starting Matlab on our HPC cluster
h5i=h5disp('flds.tot.001','/')
densi=h5read('flds.tot.001','/densi');

The code generates a mountain of data
 
For each timestep (where xxx is the timestep)
flds.tot.xxx (bx,by,bz,dens,densi, ex,ey,ez,jx,jy,jz)
momentum.xxx (dens,dens0,dpx,dpy,dpz,p(x,y,z)bin, p(x,y,z)lim, p(x,y,z)elogsp,xshock,xsl
param.xxx
prtl.xxx  (particle data)
spect.xxx


All we need now is to complete this post with some output from one of the initial test runs!




Saturday, 3 October 2015

Solar Observing Space Missions (Past, Present and Future)

Our studies of solar global oscillations require validation against observational data from both ground and space based observations. One such observatory is the Solar dynamics observatory  which continues to produce huge quantities of data inspiring much public interest and used to discern the mysteries of our star.  Before we consider future solar observing missions we shall consider the development over the last 60 years.  Shrouded in secrecy some of the first space based observations of the sun were made in the late 40's using rockets based on designs of the V2 and carrying payloads upto altitudes achieved by the space shuttle.  The earlier "sounding rockets" carried instrumentation for around 10 minutes before re-entering the atmosphere. Using the techniques developed for high resolution imaging of the solar corona the skylab mission in 1974 undertook some of the first ground breaking observations of solar phenomena. By making observations situated above the earths atmosphere it is possible to extend the wavelength range over which observations can be made. Such observations reduce the atmospheric scattering/distortion which occurs. They also make continuous observations possible. Continuous observations allow for the possibility of studying long duration events and oscillatory phenomena. Space weather monitoring is important for the protection of satellites, airborne aircraft safety, power systems and for the safety of astronauts. Our increasing dependence on global communications and monitoring systems requires an array of space borne monitoring and detection systems to provide reliable space weather forecasts. The diversity of measurements required for space weather forecasting can be seen on sites such as;
Examples of solar wind prediction from NOAA are shown above. It should be noted that modern space weather studies is dependent on an armoury of computational modelling tools a key topic of this blog.
 
There have been many other missions including
  1. Solwind launched in 1979
  2. Orbiting Solar Observatories (1962-1975)
  3. GOES  Geo-stationary operational environmental satellites (1975-2016)
  4. The solar maximum mission  launched in 1980
  5. Yohkoh launched in 1991
  6. Hinode launched in 2006
  7. Ace launched in 1997
  8. Cluster II mission, launched 2000
  9. RHESSI (Reuven Ramaty High Energy Solar Spectroscopic Imager), launched 2002
  10. Solar and Heliospheric Observatory, launched 1995
  11. Transition Region and Coronal Explorer (TRACE), launched 1998
  12. Stereo (Solar Terrestrial Relations Observatory) launched 2006
  13. Solar Dynamics Observatory  launched 2010
  14. Proba2 launched 2009
We have included in this list satellites monitoring the near earth environment, the magnetosphere and the solar wind. The list of instruments used is huge including spectrometers, coronagraphs, charge analysers, mass spectrometers, telescopes and magnetometers.

The solar dynamics observatory (SDO) features three main sets of instruments
  • Helioseismic and magnetic imager (HMI)
  • Extreme ultraviolet variability experiment (EVE)
  • Atmospheric Imaging Assembly (AMI)
SDOs purpose is to study solar variability and the space weather which results from that. Whilst the HMI are the main instruments for monitoring magnetic variability the AIA provide views of the solar corona extending to 1.3 solar diameters. The AIA covers 10 wavelengths and provides a resolution of 1 arcsecond with a cadence of approximately 10seconds. This level of data provides excellent opportunities to improve our understanding of the activities and dynamics of the solar atmosphere. The range of wavelengths covered by the AIA provides opportunities to study the propagation of phenomena through the layers of the solar atmosphere.

AIA wavelength channel Source Region of solar atmosphere Characteristic
temperature
White light continuum Photosphere 5000 K
170 nm continuum Temperature minimum, photosphere 5000 K
30.4 nm He II Chromosphere & transition region 50,000 K
160 nm C IV + continuum Transition region & upper photosphere 105 & 5000 K
17.1 nm Fe IX Quiet corona, upper transition region 6.3×105 K
19.3 nm Fe XII, XXIV Corona & hot flare plasma 1.2×106 & 2x107 K
21.1 nm Fe XIV Active region corona 2×106 K
33.5 nm Fe XVI Active region corona 2.5×106 K
 9.4 nm Fe XVIII Flaring regions 6.3×106 K
13.1 nm Fe VIII, XX, XXIII Flaring regions 4×105, 107 & 1.6×107 K


Live Imagery from SDO AIA
As well as the fourth generation GOES-R missions planned for 2016, future missions include
  The solar orbiter  is an ESA mission partnering with NASA one of the objectives is to perform close observations of the polar regions of the sun. Observations would be made from distances as close as 45 solar radii (0.22AU).  Solar orbiter has a range of instrumentation for solar remote sensing, including EUV imagers covering different depths through the solar atmosphere, high resolution and full disk imagers.  The METIS coronagraph will provide imaging of the corona at 121.6nm. An important feature is that observations willl be coordinated with those of the solar probe plus mission.  The solar  probe plus mission will use gravity assist from Venus to acheive increasingly close distances to the sun to approximately 8.5 solar radii. The range of instruments it includes is as follows SWEAP for analysing the composition of the solar wind. A suite of magnetometers and antennae for modelling the characteristics of the electromagnetic fields. The Thor mission is part of the ESA Cosmic Vision 2015-2025 the purpose of Thor is to investigate plasma heating and energy dissipation.

Turbulence formation at shock, Vlasiator
The figure above shows the formation of turbulence at the bow shock as seen by the global hybrid-Vlasov simulation code VlasiatorThe figure shows a zoomed in cut from the full simulation, where zooming has been done on the quasi-parallel part of the shock where the strongest turbulent fluctuations are observed.  The full simulation box extends from -10 Re to +50 Re in X, and +-50 Re in Y. The ordinary space resolution in this run was set to 227 km (of the order of the ion skin depth), while the velocity space resolution is 30 km/s. Color-scaling depicts density, solar wind flows from the right-hand-side-of the simulation box, while the interplanetary magnetic field forms a 30 degree angle with respect of the Sun-Earth line. The waves in the lower part are the foreshock waves due to solar wind ion reflection at the bow shock surface.


Thursday, 23 July 2015

Characterisation of Modes for solar global oscillations

This post is a summary of our work to classify energy propagation into the solar atmosphere resulting from different modes of solar global oscillations.

The background to the work is described at
Dynamics of the Solar Atmosphere Generated by the Eigen Modes of Solar Global Oscillations

Theoretical background to this work is described in
Our Wobbling Star!

and

The Solar Global Eigenmodes of Oscillation

Source code used for the runs including the input files may be obtained from github https://github.com/mikeg64/smaug_pmode/

We have performed two series of simulations delivering the same quantity of energy into the solar atmosphere with the extended driver across the lower boundary of the model.

The series are;

  1. Normal modes for a given value for the speed of sound in the solar atmosphere
  2. Mode series for drivers corresponding to the 5 minute and 3 minute oscillation

The series for the normal modes are calculated from the following expressions for a given of the speed of sound c and a value for the length of the simulation box of 4Mm. The frequency is computed using


The series of normal modes are computed such that the modes satisfy the following condition
The speed of sound for the VALIIc solar atmosphere is shown in the plot below
Variation of Sound Speed with height above the Photosphere Calculated from the VALIIC and McWhirter Model of the Solar Atmosphere

The simulations are tabulated below. The tables indicate the mode, the period for the driver, the driver amplitude and the label used for each of the data sets (for the input files and the configuration outputs). Remember that for both series the amplitudes have been set so that the same amount of energy is delivered by the driver. For the normal modes we have

For sound speed 20km/s

Mode Driver Period (s) Amplitude (m/s) Label
(0,0) 282.84 500 spic_2p82_0_0_3d
(0,1) 200.0 200 spic_2p00_0_1_3d
(0,2) 133.33 100 spic_1p33_0_2_3d
(0,3) 100.0 58.8 spic_1p00_0_3_3d

 For sound speed 13km/s

Mode Driver Period (s) Amplitude (m/s) Label
(0,0) 435.1 500 spic_4p35_0_0_3d
(0,1) 307.7 200 spic_3p07_0_1_3d
(0,2) 205.1 100 spic_2p05_0_2_3d
(0,3) 153.8 58.8 spic_1p53_0_3_3d

 For sound speed 8.4km/s

Mode Driver Period (s) Amplitude (m/s) Label
(0,0) 673.4 500 spic_6p7_0_0_3d
(0,1) 425.9 200 spic_4p3_0_1_3d
(0,2) 301.2 100 spic_3p0_0_2_3d
(0,3) 147.1 58.8 spic_2p3_0_3_3d


 For sound speed 31.43km/s

Mode Driver Period (s) Amplitude (m/s) Label
(0,0) 179.98 500 spic_1p79_0_0_3d
(0,1) 127.27 200 spic_1p27_0_1_3d
(0,2) 84.84 100 spic_0p84_0_2_3d
(0,3) 63.63 58.8 spic_0p63_0_3_3d


The series for the five and three minute modes are as follows


Mode Driver Period (s) Amplitude (m/s) Label
(0,0) 300.0 1250 spic5b0_3d_rep_3d
(0,1) 300.0 500 spic5b0_1_3d
(0,2) 300.0 250 spic5b0_2_3d
(0,3) 300.0 147.06 spic5b0_3_3d


Mode Driver Period (s) Amplitude (m/s) Label
(0,0) 180.0 1250 spic6b0_3d_rep_3d
(0,1) 180.0 500 spic6b0_1_3d
(0,2) 180.0 250 spic6b0_2_3d
(0,3) 180.0 147.06 spic6b0_3_3d

 We present below distance-time plots, plots of the energy flux and the variation of the energy flux for the different modes.

The plots below compare the distance time plots for the vertical component of the plasma velocity with the (0,0),(0,1),(0,2) and (0,3) modes.
Distance Time plot for 180s Driver the d-t plot has been taken through a horizontal section through the model in the Chromosphere at 1.06Mm.

Distance Time plot for 180s Driver the d-t plot has been taken through a horizontal section through the model in the Transition Layer at 2Mm.

Distance Time plot for 180s Driver the d-t plot has been taken through a horizontal section through the model in the Solar Corona at 4.3Mm.

Distance Time plot for 180s Driver the d-t plot has been taken through a vertical section through the model in the x-direction (section is taken at the horizontal location corresponding to the maxima)

Distance Time plot for 180s Driver the d-t plot has been taken through a vertical section through the model in the y-direction (section is taken at the horizontal location corresponding to the maxima)

Distance Time plot for 300s Driver the d-t plot has been taken through a horizontal section through the model in the Chromosphere at 1.06Mm.

Distance Time plot for 300s Driver the d-t plot has been taken through a horizontal section through the model in the Transition Layer at 2Mm.

Distance Time plot for 300s Driver the d-t plot has been taken through a horizontal section through the model in the Solar Corona at 4.3Mm.

Distance Time plot for 180s Driver the d-t plot has been taken through a vertical section through the model in the x-direction (section is taken at the horizontal location corresponding to the maxima)

Distance Time plot for 180s Driver the d-t plot has been taken through a vertical section through the model in the y-direction (section is taken at the horizontal location corresponding to the maxima)

The plots below show the integrated energy across a section of the computational model for different heights through the solar atmosphere the variation with time is shown.

Integrated Energy Variation for 180s Driver

Integrated Energy Variation for 300s Driver


The plots below compare the integrated time averaged flux for the solar atmosphere and the transition layer. For each mode the plots illustrate the variation of the integrated flux with the driver period.
Time Averaged Flux Variation for the Solar Corona


Time Averaged Flux Variation for the Transition Layer
Although the contribution of individual modes is a relatively small contribution we observe that as the driver period increases

The full set of results (images) is available from the following link
http://bit.ly/1TUMfOf

Thursday, 7 May 2015

SMAUG Updates and GPU Training

In this post I describe codes used at Sheffield for Magnetohydrodynamics in particular I mention GPU codes and describe some of the training and resources that are available for GPU computing. 

Parallel magnetohydrodynamic (MHD) algorithms are important for numerical modelling of highly inhomogeneous solar and astrophysical plasmas. SMAUG is the Sheffi eld Magnetohydrodynamics Algorithm Using GPUs. SMAUG is a 1-3D MHD code capable of modelling magnetised and gravitationally strati fied magnetised plasma

SMAUG is based on the Sheffield Advanced Code (SAC), which is a novel fully non-linear MHD code, designed for simulations of linear and non-linear wave propagation in gravitationally strongly stratified magnetised plasma. See the  reference at the Astronomy Abstracts Service. A full description of SMAUG is available in the paper entitled "A Fast MHD Code for Gravitationally Stratified Media using Graphical Processing Units: SMAUG".
 
The development version of SMAUG is available at CCP forge. The devlopment code can be obtained using SVN, as follows

svn checkout http://ccpforge.cse.rl.ac.uk/svn/sac/dev/smaug

SMAUG is published  on google code and the release version is available using the following SVN command (see checkout the SMAUG release version)

# Non-members may check out a read-only working copy anonymously over HTTP.
svn checkout http://smaug.googlecode.com/svn/trunk/ smaug-read-only

The key priority for the development SMAUG code is the correct implementation of a scalable multi-GPU version. Significant effort has been made to develop this using the Wilkes, Emerald and the Iceberg cluster at The University of Sheffield.

In summary, the main tasks are as follows
One of the best updated guides is the Programming guide at the NVIDIA developer zone there is a good collection of CUDA links on Delicious. There is support for GPU computing through the NVIDIA GPU research centre at The University of Sheffield. In addition to the provision of GPUs on the iceberg HPC cluster the GPU research centre provides training in GPU computing.

The recent introduction to CUDA course organised at Sheffield is based on a course provided by the EPCC. As well as an introduction to basic CUDA programming the hands on exercise focuses on the techniques that can be used to drive the performance of a highly threaded code such as that running on a GPU. As a GPU is a co-processor used for code acceleration it is necessary to consider the movement of date between the GPU and the host processor. For example we look at the need to consider the following inhibitors.
  1. Data transfer to and from the GPU
  2. Device under utilisation - a GPU has many cores we need to ensure that our computation provides many threads of execution fully utilising all of the cores.
  3. GPU memory bandwidth -  as with any processing unit data must be moved into memory (e.g. registers and caches) used for floating point operations. One of the reasons for the excellent performance of a GPU is its data bandwidth and the ability to move data to the processing cores. On a GPU it is best to move data in a few large chunks rather than in many small chunks. The checks of memory should also continuous portions of memory rather than scattered fragments. In GPU speak this is referred to as memory coalescing.
  4. Code branching operations such as if statements are dependent on the number of threads executed within a processing core on a GPU. It is generally best to group these calls together.
The course exercises provide excellent practice in understanding these points.The problem is based on edge detection algorithm for image processing. A Gauss-Seidel iterative algorithm is used to reconstruct an image from edge data. The exercise identifies the data transfer bottleneck for example rather than copying data between device and host at every iteration a memory address swapping technique is used and data copying at every step avoided. The example considers occupancy by comparing performance when each thread is used to process first a single row of the image matrix, a single column of the image matrix and finally each thread processes an individual element of the  matrix, thereby improving the occupancy and arithmetic intensity. Arithmetic intensity is the ratio of computation to memory accesses. The third example considers memory coalescing and compares the performance of the algorithm when threads are used to compute each row and then each column of the matrix. The row case is clearly faster because the matrix elements used by the thread are in adjacent memory locations i.e. coalesced. The column case is slower!  GPUs group threads into collections called thread blocks the GPU programmer can control the number of thread blocks and the number of threads which execute per thread block. The final example investigates how the performance is influenced by the sizes of the thread block sizes.

Once armed with an understanding of these ideas a programmer can begin to develop GPU codes the course problem is a great help in understanding these concepts.



Wednesday, 6 May 2015

Remote Visualisation: A Guide and Example

The need to explore and visualise large data sets is an important capability for many areas of research including computational solar physics. Since many of the data sets are stored on remote servers and not local workstations  it is necessary to use a remote visualisation technique delivering visual output from the data set and delivering a rendered imagery to the users desktop. Remote Visualisation is undertaken using thin clients accessing remote high quality visualisation hardware. Remote visualisation removes the need to transfer data and allows researchers to visualise data sets on remote visualisation servers attached to the high performance computer and its storage facility.


Just as OpenGL is the standard used for 3D graphical visualisation, VirtualGL is the standard used fro remote visualisation. VirtualGL is an open source package which gives any UNIX or Linux remote display software the ability to run 3D applications with full hardware accelerations. VirtualGL can also be used in conjunction with remote display software such as VNC to provide 3D hardware accelerated rendering for OpenGL applications. VirtualGL is very useful in providing remote display to thin clients which lack the 3D hardware acceleration.
Protocols used to communicate graphical information over a network.


A VirtualGL Client Runs on the User Workstation Which is Served with Graphics from a High Quality High Performance Rendering Device Such as an NVIDIA Graphics Card e.g. the Fermi M2070Q 
To use the NVIDIA Fermi M2070Q Graphical Processing Unit on the iceberg High Perfromance Computer at The University of Sheffield a number of options are available for starting a virtualGL client. With Iceberg we make use of, TigerVNC and the  Sheffield application portal, Sun Global Desktop (SGD). To use this capability it is necessary for users to join the gpu visualisation group (gpu-vis ) by emailing hpchub@sheffield.ac.uk.

To initiate a remote visualisation session the following steps should be followed:
Star a browser and goto


–login to Sun Global Desktop (as shown below)

•Under Iceberg Applications start the Remote visualisation session

•This opens a shell providing a port number (XXXX) and instructions to either open a web

browser or to start the tigerVNC client.


           –Open a browser and enter the address

http://iceberg-gateway.shef.ac.uk:XXXX



Start Tiger VNCViewer on your desktop

Use the address iceberg-gateway.shef.ac.uk:XXXX

XXXX is a port address provided on the iceberg terminal

When requested use your usual iceberg user credentials

From a Microsoft windows work station, the SSH client Putty and Tiger VNC can be used as follows.

Login in to iceberg using putty

•At the prompt type qsh-vis

•This opens a shell with instructions to either

Open a browser and enter the address


Start Tiger VNCViewer on your desktop

  • Use the address iceberg.shef.ac.uk:XXXX
  • XXXX is a port address provided on the iceberg terminal
  • When requested use your usual iceberg user credentials
Putty Session Starting the VirtualGL Server (note the instructions provided)

Starting Tiger VNC
Typical VirtualGL Remote Visualisation Session
Instead of using the qsh-vis command an alternative is to start the vncjob with the following SGE command e.g. if there is a different memory requirement
qrsh -l gfx=1 -l mem=16G -P gpu-vis -q gpu-vis.q /usr/local/bin/startvncjob