Difference between revisions of "User talk:Ferrandm"

Revision as of 10:37, 29 June 2012 (view source) Ferrandm (Talk | contribs) (→‎Vtk_bin_write.c) ← Older edit		Revision as of 10:39, 29 June 2012 (view source) Ferrandm (Talk | contribs) (→‎Job) Newer edit →
Line 214:		Line 214:
	The extension of this file is .ll because it is a LoadLeveler file.		The extension of this file is .ll because it is a LoadLeveler file.
−	A job has to be created like this :	+	A job for Antares has to be created like this :
	*Working name : locating more easily the calculations.		*Working name : locating more easily the calculations.
	@ job_name = $NAME-$dir		@ job_name = $NAME-$dir

---

Revision as of 10:39, 29 June 2012

Hypotheses

The hypotheses to simplify the Navier-Stokes equations are :

• The total energy balance according to Poinsot-Veynante is descreibed below, if ${\displaystyle E=e_{s}+{\dfrac {1}{2}}u_{i}^{2}}$ :

${\displaystyle {\rho }{\dfrac {DE}{Dt}}={\dfrac {\partial {{\rho }E}}{\partial {t}}}+{\dfrac {\partial {{\rho }u_{i}E}}{\partial {x_{i}}}}={\dot {\omega _{T}}}+{\dfrac {\partial }{\partial {x_{i}}}}({\lambda }{\dfrac {\partial {T}}{\partial {x_{i}}}})-{\dfrac {\partial }{\partial {x_{i}}}}({\rho }{\sum }_{k=1}^{N}h_{s,k}Y_{k}V_{k,i})+{\dfrac {\partial {\sigma _{ij}}u_{i}}{\partial {x_{j}}}}+{\dot {Q}}+{\rho }{\sum }_{k-1}^{N}Y_{k}f_{k,i}(u_{i}+V_{k,i})}$

The term ${\displaystyle {\dfrac {\partial }{\partial {x_{i}}}}({\rho }{\sum }_{k=1}^{N}h_{s,k}Y_{k}V_{k,i})}$ can be ignored with regard to ${\displaystyle {\dot {\omega _{T}}}}$, which is the heat release due to combustion.

This release can be defined by this equations : ${\displaystyle {\dot {\omega _{T}}}=-\sum _{k=1}^{N}\Delta h_{f,k}^{0}{\dot {\omega _{k}}}=\nu _{F}M_{F}{\dot {\omega }}Q_{m}}$, where ${\displaystyle Q_{m}}$ is the specific heat of reaction, ${\displaystyle \Delta h_{f,k}^{0}}$ is the standard enthalpy of formation and ${\displaystyle {\dot {\omega _{k}}}}$ is the reaction rate.

${\displaystyle {\rho }{\sum }_{k=1}^{N}h_{s,k}Y_{k}V_{k,i}}$ is the power produced by volume forces ${\displaystyle f_{k}}$ on species k.

${\displaystyle {\dot {Q}}}$ is the heat source term which is null here since there is no electric spark, laser or radiative flux.

${\displaystyle Y_{k}}$ is the mass fraction of the species k.

The tensor ${\displaystyle \sigma _{ij}}$ is composed of the tensor of constraints and the tensor of pressure : ${\displaystyle \sigma _{ij}=\tau _{ij}-P\delta _{ij}}$

Thanks to these hypotheses, the total energy balance becomes :

${\displaystyle {\dfrac {\partial \rho E}{\partial t}}+{\dfrac {\partial (P+\rho E)U_{j}}{\partial x_{j}}}={\dfrac {\partial q_{j}}{\partial x_{j}}}+{\dfrac {\partial \tau _{ij}U_{i}}{\partial x_{j}}}+S_{5}}$.

With ${\displaystyle S_{5}}$ which corresponds to the heat release due to combustion, ${\displaystyle {\dot {\omega _{T}}}}$.

• The term ${\displaystyle S_{i}^{*}}$ is not used. Thanks to it we can define other forces like electromagnetical force or weight.

• ${\displaystyle S_{k}}$ is either a fuel or a combustive source term.

${\displaystyle S_{6}=\nu _{F}M_{F}{\dot {\omega }}}$ if it is a fuel source term or ${\displaystyle S_{7}=\nu _{O}M_{O}{\dot {\omega }}}$ if it is an oxidant source term.

${\displaystyle M_{F}}$ and ${\displaystyle M_{O}}$ are respectively the molar mass of fuel and oxidant.

• ${\displaystyle {\dot {\omega }}}$ can be determined by the Arrhenius law :

${\displaystyle {\dot {\omega }}=B_{1}(\phi )\rho Y_{F}Y_{O}e^{\dfrac {\beta }{\alpha }}e^{\dfrac {-T_{a}}{T}},B_{1}(\phi )=cst}$

• The heat flows due to mass fraction gradients and the species molecular distribution due to temperature gradients are neglicted (cf : Soret and Dufour).

• The termal conductivity, ${\displaystyle \lambda }$ is given by :

${\displaystyle \lambda ={\dfrac {\mu C_{p}}{P_{r}}}}$

Where ${\displaystyle P_{r}}$ is the Prandtl number :

${\displaystyle P_{r}={\dfrac {\nu }{D_{th}}}}$

This number compares the momemtum distribution with the heat distribution.

• The Lewis number for the species k, ${\displaystyle Le_{k}}$, is :

${\displaystyle Le_{k}={\dfrac {\lambda }{\rho C_{P}D_{k}}}}$ ${\displaystyle ={\dfrac {D_{th}}{D_{k}}}}$

Where ${\displaystyle D_{k}}$ is the species diffusion.

We will consider the Lewis number egals to 1 in order to simplify the physics of the flame of pre-mixture.

• The Schmidt number, ${\displaystyle S_{ck}}$, is :

${\displaystyle S_{ck}=P_{r}Le_{k}}$

• From the relation of these last three numbers, we can deduct that :

${\displaystyle D_{th}=D_{k}}$ and ${\displaystyle D_{k}={\dfrac {D}{\rho }}}$

${\displaystyle P_{r}={\dfrac {\mu }{D}}}$

Therefore, the diffusion coefficient can be defined as :

${\displaystyle D={\dfrac {\mu }{S_{ck}}}}$

• We use the Law of Fick which is a simplified diffusion law :

${\displaystyle {\vec {\phi }}=-D.{\vec {\nabla }}C}$

${\displaystyle {\vec {\phi }}}$ is the flux of particles density.

C is the particles density.

Thus the diffusion coefficients of the various species are characterized by the number of Lewis.

• We consider that the gas is a perfect, viscous, reagent and diatomic gas. Consequently as the gas is viscous, the compressible effects are not dominating. What involves that the bulk viscosity is useless.

• The gas is reagent thus the mixture of various species is not isothermal, they must be individually followed. It implies that the calorific capacities depend on the temperature and on the composition.

• For a diatomic gas, the calorific capacities can be defined as follows :

${\displaystyle C_{P}={\dfrac {7}{2}}r}$

${\displaystyle C_{V}={\dfrac {5}{2}}r}$

• The gas respects the law of perfect gases :

${\displaystyle P=\rho rT}$

• The combustion is an irreversible transformation whose the creation of entropy is compensated with an entropy given up by the system to the outside, because of a thermal transfer, thus we can consider that the transformation is isentropic. Consequently, the law of Laplace for the thermodynamics is applicable :

${\displaystyle {\dfrac {C_{P}}{C_{V}}}=\gamma =1.4}$

• According to the power law, the dynamic viscosity depends only on the temperature :

${\displaystyle \mu =\mu _{0}\left({\dfrac {T}{T_{0}}}\right)^{\alpha _{s}}}$

${\displaystyle \alpha _{s}=0.76}$

• The tensor of the constraints by respecting the hypothesis of a Newtonian fluid is :

${\displaystyle \tau _{ij}=\mu \left({\dfrac {\partial U_{i}}{\partial x_{j}}}+{\dfrac {\partial U_{j}}{\partial x_{i}}}\right)-{\dfrac {2}{3}}\mu {\dfrac {\partial U_{k}}{\partial x_{k}}}\delta _{ij}}$

The Kronecker symbol, ${\displaystyle \delta _{ij}}$, egals 1 if i=j, 0 else.

• The acoustic Reynolds number is defined such as :

${\displaystyle R_{c}={\dfrac {cL}{\nu }}}$

c is the speed of sound : 340 m.${\displaystyle s^{-1}}$, L is the characteristic length and ${\displaystyle \nu }$ is the kinematic viscosity of air : 1.45e-5 m².${\displaystyle s^{-1}}$.

Closure equations

• ${\displaystyle \rho E=\rho C_{V}T+{\dfrac {1}{2}}\rho U_{i}^{2}}$

• ${\displaystyle P=\rho rT}$

• ${\displaystyle \mu =\mu _{0}\left({\dfrac {T}{T_{0}}}\right)^{\alpha _{s}}}$

     ${\displaystyle \alpha _{s}=0.76}$

• ${\displaystyle q_{j}=\lambda {\dfrac {\partial T}{\partial x_{j}}}}$

• ${\displaystyle q_{j}^{k}=D{\dfrac {\partial Y_{k}}{\partial x_{j}}}}$

• ${\displaystyle \tau _{ij}=\mu \left({\dfrac {\partial U_{i}}{\partial x_{j}}}+{\dfrac {\partial U_{j}}{\partial x_{i}}}\right)-{\dfrac {2}{3}}\mu {\dfrac {\partial U_{k}}{\partial x_{k}}}\delta _{ij}}$

Parameters

Makefile

The makefile allows to compile routines and subroutines.

In order to compile, the command "make" will be typped in the terminal.

The default target of this file is "all : :".

Since various computers can be used, the compilation has to be conditional. That is to say that several configurations of compilation are available in function of the cluster considered (Antares, Jade, ...) :

• FC indicates what is the used compiler. For instance : if the language is fortran90 and MPI, it will be written FC = mpif90.
• A part of the code is written in C, it is about the file Vtk_bin_write.c. So the compiler for the C language has to be defined thanks to CFLAGS.
• FFLAGS defines the optimization options for the Fortran compiler. For instance :

FFLAGS = -O3 -ipo -p.

"O3" is an optimization option of level 3. From this level, semantics of the routine can be modified. The optimization will be more aggressive concerning the memory access and the buckles management. The compilation time will be more important than for a lower level and the code size will be also bigger.

"ipo" is an interprocedural optimization option. It authorizes the inlining of present subroutines in seperated source files.

"p" is a profiling option.

The common files of the executable for calculation and the common files of the executable for postprocessing have to be defined.

For example :

all : : Mezo3D df6_pospro
COMMON_OBJS += communs.o
COMMON_OBJS += lectus.o

Then Allegro files are compiled and then an executable is created :

Mezo3D : $(ALLEGRO_OBJS)
$(QUIET_LINK)$(FC) $(FFLAGS) -o $@ $(filter %.o,$ ˆ)

The same thing is done for the postprocessing files :

df6_pospro : $(POSTPRO_OBJS)
$(QUIET_LINK)$(FC) $(FFLAGS) -o $@ $(filter %.o,$ˆ)

An option named clean is available. This one allows to erase files with extensions .o, .mod, .d. The executable files, the solution files and the gmon.out files are also deleted.

It is possible to change this option if necessary .

Executable

Fort.10.sh

Fort.10.sh is a file written in shell language in purpose to ease the creation of initial directories for the submission of new jobs.

As it has been first thought it works with the file launch-run.sh. In launch-run.sh the variables NX, NY, NZ (that define the mesh size per processors) and the variables NPX, NPY, NPZ (that define the number of processors in each direction) are defined to be exported in fort.10.sh. In this way, several calculations can be launched by changing only a few parameters.

Initialization example :

sh launch-run -g

The option -g calls the subroutine generate(). This subroutine creates a directory named by the variable "dir". In this directory, two other directories will be created. A directory "IN" will contain the input files while a directory "OUT" will contain the output files. Then the template of the executable "NAME" is redirected towards a file named fort.10. The file fort.10 will be located in the directory "IN". This step is realized like this :

sh $Name.template > $dir/IN/fort.10.

When the file fort.10 is created, the subroutine copies the executable calculating the case, the executable dealing the results as well as the file fort.11 in the directory "IN". Then the template of the job is redirected towards the file named "job".

Vtk_bin_write.c

This file concerns the distribution of processors and the choice of the cluster.

It is possible to launch calculations either on ATON, or on VARGAS. In that case, it is necessary to put the value 1 for the computer which is going to be used. For instance, if the used computer is ATON, then the value entered will be 1 and so, it will be 0 for the computer VARGAS :

#define ATON 1
#define VARGAS 0

Otherwise, if the used computer is VARGAS, then the value entered will be 1 and it will be 0 for the computer ATON :

#define ATON 0
#define VARGAS 1

Be careful, some options will be possibly modified in function of the cluster.

The distribution of processors can be defined in function of

• Coordinates.
• Vectors : in that case it will depend on the dimension of the problem, NDIM.
• Scalar : this function will define the id number of the processor.

Job

The job file is used if the calculation of the problem is made on a computer which is not local.

The extension of this file is .ll because it is a LoadLeveler file.

A job for Antares has to be created like this :

• Working name : locating more easily the calculations.

@ job_name = $NAME-$dir 

• Standard exit file :

@ output = \$(job_name).o\$(jobid)

• Error exit file :

@ error = \$(job_name).e\$(jobid)

• Working type :

@ job_type = MPICH

• Total number of wanted processus :

@ total_tasks = $NP

• Maximal elapsed time for the job :

@ wall_clock_limit = 24 :00 :00

• Maximal memory by MPI process : if a process exceeds the specified limit, then the calculation is killed. Conversely, if this parameter is overestimated, the waiting time will be longer. It is preferable to execute calculations with a data limit subordinate or equal in 3gb. The really necessary memory by process has to get closer as much as possible to the wanted memory.

@ data_limit = 550mb

• Initial directory to send : this directory contains all the data for the calculation.

@ cri_initialdir = $PWD/$dir/IN

• Final directory for the results : the computer has to send the results to this directory.

@ cri_finaldir = $PWD/$dir/OUT

• User mail : an e-mail will be send in case of overtaking of elapsed time.

@ notify_user = name@adress.fr

• Buffering working directory :

cd $LOCAL_WORK_DIR

The arborescence of the file must be defined below this line. Generally, it corresponds to the arborescence of the final directory.

The result files are moved in a directory named Spool before being copied in the final directory.

Launch-run

Test case

Scaling

Profiling

Profiling is a way to optimize the code skill. It consists of the determination of the parts which use most resources.

Thanks to the command "gprof", it is possible to create a report of execution. The executable which calculates the case must be compilated with the option -pg, then it must be executed in order to obtain the report, gmon.out.

The file gmon.out is divided into 2 parts :

• Call-graph profile : list of the routines in decreasing order of the CPU consummate time as well as the routines which are called from them.
• Flat profile : list of the routines in the decreasing order of the CPU consummate time and the number of calls.

Analysis of the flat profile

Description

• % time : percentage of total CPU time consumed by the routine.
• Cumalative seconds : partial sum of CPU time.
• Self seconds : CPU time of a routine.
• Calls : number of calls of a routine.
• Self ms/call : average of CPU time consumed by call to the routine, exclusive time.
• Total ms/call : total time (exclusive + inclusive) consumed by call to the routine.

Analysis

Having to test a calculation on various combinations of processors, it seems clearly that the most consumer routines are :

• der_sub_MOD_interpolate1_on_slice
• der_sub_MOD_derivative2_on_slice
• all_solver_MOD_momentum
• der_sub_MOD_interpolate2_on_slice

For instance for a calculation on 4x1x1 processors, the flat profile will be the following :

figure

It indicates that the subroutine derivative2 on slice represents 20 % of total CPU time consumed. Thanks to the cumulative seconds, we can deduce that the total CPU time consumed egals 0.25 second. The number of seconds represented by the subroutine derivative2 on slice is about 0.05 second. This subroutine is invoked 545 times. We can say that the average number of milliseconds spent in this subroutine and its children per call is 0.09 ms.

Remark : If the calculation is run on 2x2x1 instead of 4x1x1, the number of calls of the subroutines derivative and interpolate periodic are divided by 2, while the number of calls of the subroutines derivative and interpolate on slice increases.

Analysis of the call-graph profile

This part is very interesting because it allows to know which routine calls another one and how long does it spend.

The routines are indexed in decreasing order by comsumption of CPU time.

This report details the percentage of the total time that was spent in a function and its children. The total of this percentages is not equal to 100 %. For instance the % run time for the main egals 100 % because it is a routine which takes a lot of time, but the routine solving the time stepping has a run time of 97.3%. This report gives also the amount of time that was propagated from the function into the parent (self), the amount of time that was propagated from the children into the function (children), the number of calls from a routine to another.

Figure

For instance, if we consider the subroutine der_sub_MOD_echange [5], we can see that the parent functions are der_MOD_interpolate2 [16], der_MOD_derivative1 [15], der_MOD_derivative2 [11], der_MOD_interpolate1 [8].

This subroutine is called 1014 times by the routines der_MOD_interpolate2 (93 calls), der_MOD_derivative1 (174 calls), der_MOD_derivative2 (327 calls) and der_MOD_interpolate1 (420 calls). The time spent in this function is about 0.01 second.

The children functions of this subroutine are der_sub_MOD_interpolate1_on_slice [7], der_sub_MOD_derivative2_on_slice [9], der_sub_MOD_derivative1_on_slice [17], der_sub_MOD_interpolate2_on_slice [18], der_sub_MOD_interpolate1_periodic_on_right [32] and so on.

The total amount of time propagated into this subroutine by its children is 0.13 second.

We can see that the subroutine echange calls 545 times the subroutine derivative2 on slice. This last one is called only by the subroutine echange since it is indicated 545/545.

The subroutine echange spends 56 % of the total time in run time.

Optimization of communication time

The run time consists of the calculation time and the communication time. Profiling aims also to minimize this communication time which can be decomposed in the time of latency, of additional cost and of transfer.

The additional cost is the time of preparation of a message.

How to optimize the communication time ?

The first step is to cover the communications by the calculations.

Then, it is necessary to avoid copying out messages in buffering memory spaces.

It is also necessary to minimize additional costs led by the repetitive calls to the subroutines of communication.

Finally, it is better to send an only big message rather than several small messages.

How to know the execution time of a routine ?

This information is given by the subroutine of MPI communication : MPI_WTIME().

This subroutine informs the user of the run time as it is called. Therefore, if this subroutine is called at the beginning and at the end of a routine, the run time can be deducted.

What way is it preferable to use for the sending and the reception of messages ?

Standard sending : If the MPI makes a buffering copy of the message then the sending ends when this one is ended. Else, the sending ends when the reception of the message is ended.

Blocking synchronous sending : This sending way uses the subroutines MPI_SSEND() and MPI_RECV().

The blocking synchronous sending allows to avoid buffering copies of messages, in this way it also avoids additional costs.

Thanks to it, a factor 2 is gained with regard to the standard mode.

Non-blocking synchronous sending  : This way uses the subroutines MPI_ISSEND() and MPI_IRECV().

The non-blocking way allows to cover the communications by the calculations. The sending of a message ends when the reception was posted and when the reading of the message is ended.

The sending is coupled with the reception thus a factor 3 is gained with regard to the standard sending.

So that this mode can work correctly, it is necessary to add MPI_WAIT() after MPI_ISSEND() and MPI_IRECV(). It allows to synchronize the process till the end of the request.

Model of calculation

Time discretization

A third order Runge-Kutta method is used. This one allows to improve the precision of time integration and the stability.

The equations used in H-Allegro are defined below :

${\displaystyle {\dfrac {Q_{1}^{1}-Q_{1}}{\Delta t/3}}+{\dfrac {8}{5}}RHS(Q_{1})=0}$

${\displaystyle Q^{2}=Q^{1}-{\dfrac {\Delta t}{4}}RHS(Q_{1})}$

${\displaystyle {\dfrac {Q_{1}^{2}-Q^{2}}{\Delta t/2}}+{\dfrac {5}{6}}RHS(Q_{1}^{1})}$

${\displaystyle {\dfrac {Q^{3}-Q^{2}}{\Delta t}}+{\dfrac {3}{4}}RHS(Q_{1}^{2})}$

${\displaystyle Q^{i}}$ is the matrix of gross conservative data at the ith step and at the time t.

${\displaystyle Q_{1}^{i}}$ is the same matrix as before at the time ${\displaystyle t+\Delta t}$.

${\displaystyle Q^{i}}$ and ${\displaystyle Q_{1}^{i}}$ are Np x Nvar matrix, with Np, number of points and Nvar, number of variables.

Spatial discretization