CFD Online Discussion Forums

CFD Online Discussion Forums (https://www.cfd-online.com/Forums/)
-   Fluent UDF and Scheme Programming (https://www.cfd-online.com/Forums/fluent-udf/)
-   -   Calculating source term total cell-zone volume parallel computation (https://www.cfd-online.com/Forums/fluent-udf/242993-calculating-source-term-total-cell-zone-volume-parallel-computation.html)

RobV May 24, 2022 09:58
Calculating total cell-zone volume (source-term zone) parallel computation
 
Dear forum user(s),

For one of my ANSYS Fluent case studies, I need to define a source term (for a user-defined scalar) which is dependent on the total volume of the cell-zone in which the source term is applied.

During my first tests, when the UDF was not yet prepared for parallel computing, I noticed that the total cell-zone volume, computed within the DEFINE_SOURCE function (using macro C_VOLUME(c,t), was not properly calculated; the more nodes assigned to the task; the smaller the calculated cell-zone volume (in the serial solver, the cell-zone volume was calculated correctly). At this moment I realized that I needed to prepare the UDF for parallel computing.

The UDF that I wrote can be found below. The UDF can be compiled without any issues (no warnings or errors whatsoever), but when the simulation is started, it is not iterating. All the assigned cores are under 100% load. The simulation does not crash, but needs to be forcibly terminated since it seems to stay within the first iteration for an infinite time. A short description of the UDF functions can be found below (and in the UDF comments):

1. The first function [DEFINE_SOURCE(src_SCA_0, c, t, dS, eqn)] is used to define the source term. This UDF is assigned to a cell-zone (in the current tests only one cell-zone; when the UDF is functionalized it should be assigned to a multitude of cell zones). In this function, the thread id of the cell-zone is obtained which is used as input ‘fed’ to another function (outside of DEFINE_SOURCE) in which the volume of the cell-zone is calculated. Several calculations related to the cell-zone volume will be conducted within the DEFINE_SOURCE function. At this stage, these computations are excluded from the UDF (too limit complexity; for now the main goal is to get the cell-zone volume computation right).

2. The second function [double ZONE_VOLUME(zone_identification)] is used to calculate the total cell-zone volume by summing up the cell-zone volume for the various mesh partitions as returned from all the computing nodes. The input for this calculation is the thread id of the cell-zone in which the source term is assigned (obtained from the DEFINE_SOURCE function). The total cell-zone volume should be returned to the DEFINE_SOURCE function.

3. A third DEFINE_ON_DEMAND function to check whether the cell-zone volume computation, defined in the second function, is functioning correctly. When executed on demand, results of the calculation are printed in the Fluent console (see console output below). In this case I assigned the source term in a cell-zone of 200 cells / 1.8 m3 (therefore the calculation seems to be correct).

Code:
/* %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%          Scalar source term          %%%%%%%%%%
%%%%%%        UDF for 1 UD-scalar(s)        %%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% */

#include "udf.h"
#include "surf.h"
#include "metric.h"
#include "mem.h"
#include "para.h"
#include "prf.h"

/* %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%    Define Static Variables    %%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% */

/* Scalar 0 */
#define INDEX_SCA_0            0                  /* Index for user defined scalar */


/* ~~~~~~~~~~~ FUNCTION 1 ~~~~~~~~~~~ FUNCTION 1 ~~~~~~~~~~~ FUNCTION 1 ~~~~~~~~~~~ FUNCTION 1 ~~~~~~~~~~~ FUNCTION 1 ~~~~~~~~~~~ */
 
/* UDF attached to cell-zone as source term */
DEFINE_SOURCE(src_SCA_0, c, t, dS, eqn) /* function name, thread and variable number */
{
        Domain *d = Get_Domain(1);
        real x[ND_ND];
        real ion_source;
        real con;
        /* Obtaining the cell-zone id */
        int zone_ID = THREAD_ID(t);
        t = Lookup_Thread(d, zone_ID);
        int ncount = 0;
        double volume_calc = 0;
       
        /* Send cell-zone identification 'zone_ID' to function ZONE_VOLUME to calculate cell-zone volume */       
        volume_calc = ZONE_VOLUME(zone_ID);
       
        /* Loading cell-zone concentration variable(s) */
        con = C_UDSI(c, t, INDEX_SCA_0);
       
        /* Random calculation of source term value using calculated cell-zone volume (function ZONE_VOLUME) */
        /* Note: just a simple example for testing purposes, actual UDF will involve more complex computations */
        C_CENTROID(x,c,t);
        ion_source = volume_calc*0.1*con;
        dS[eqn] = 0;
       
        return ion_source;       
       
        /* Export value to see if volume value is parsed from function ZONE_VOLUME properly */
        FILE * fp1;
        fp1 = fopen("udf_test_volume_calc_parsed.txt", "w");
        fprintf(fp1, "Parsed volume cell-zone = %f", volume_calc);
        fclose(fp1);
}

/* ~~~~~~~~~~~ FUNCTION 2 ~~~~~~~~~~~ FUNCTION 2 ~~~~~~~~~~~ FUNCTION 2 ~~~~~~~~~~~ FUNCTION 2 ~~~~~~~~~~~ FUNCTION 2 ~~~~~~~~~~~ */

double ZONE_VOLUME(zone_identification) /* Cell-zone ID parsed from cell-zone in which source term 'DEFINE_SOURCE' is assigned */
{
        Domain *d = Get_Domain(1);
        Thread *t;
        cell_t c;
        int ncount = 0;
        double volume = 0;
        t = Lookup_Thread(d, zone_identification);

        host_to_node_int_1(zone_identification); /* Does nothing in SERIAL */
               
        #if !RP_HOST
                begin_c_loop_int(c,t)
                {
                        volume += C_VOLUME(c,t);
                        ncount += 1;
                }
                end_c_loop_int(c,t)
        #endif /* RP_HOST */

        PRF_GSYNC(); /* Synchronize compute nodes*/

        #if RP_NODE
                ncount = PRF_GISUM1(ncount); /* Summing up node values */
                volume = PRF_GRSUM1(volume); /* Summing up node values */
        #endif /* RP_NODE */

        node_to_host_int_1(ncount); /* Does nothing in SERIAL */
        node_to_host_real_1(volume); /* Does nothing in SERIAL */
        return volume; /* return value of 'volume' to DEFINE_SOURCE function */
       
        #if !RP_NODE
                /* Setup text-file(s) to export function solution data (for testing purposes) */
                FILE * fp2;
                fp2 = fopen("udf_test_volume_loop.txt", "w");
                fprintf(fp2,"Total volume of %d cells = %f m3 \n", ncount, volume);
                fclose(fp2);
        #endif /* RP_NODE */
}

/* ~~~~~~~~~~~ FUNCTION 3 ~~~~~~~~~~~ FUNCTION 3 ~~~~~~~~~~~ FUNCTION 3 ~~~~~~~~~~~ FUNCTION 3 ~~~~~~~~~~~ FUNCTION 3 ~~~~~~~~~~~ */

DEFINE_ON_DEMAND(test_ZONE_VOLUME)
{
        Domain *d = Get_Domain(1);
        Thread *t;
        cell_t c;
        int zone_identification = 21; /* This is the id of the cell-zone in which the source term is defined */
        int ncount = 0;
        double volume = 0;
        t = Lookup_Thread(d, zone_identification);

        host_to_node_int_1(zone_identification); /* Does nothing in SERIAL */
                       
        #if !RP_HOST
                begin_c_loop_int(c,t)
                {
                        volume += C_VOLUME(c,t);
                        ncount += 1;
                }
                end_c_loop_int(c,t)
        #endif /* RP_HOST */
       
        #if RP_NODE
                Message("Node %d is calculating on thread # %d: Total volume of %d cells = %f m3\n",myid,zone_identification, ncount, volume);
        #endif /* RP_NODE */

        PRF_GSYNC();

        #if RP_NODE
                ncount = PRF_GISUM1(ncount);
                volume = PRF_GRSUM1(volume);
        #endif

        node_to_host_int_1(ncount);  /* Does nothing in SERIAL */
        node_to_host_real_1(volume);  /* Does nothing in SERIAL */
       
        #if !RP_NODE
                Message("I am in the node process with ID %d\n",myid);
                Message("Total number of cells in cell-zone for all mesh partitions = %d\n",ncount);
                Message("Total volume of cells in cell-zone for all mesh partitions = %f m3\n",volume);
       
                /* Setup text-file(s) to export function solution data */
                FILE * fp2;
                fp2 = fopen("udf_test_volume_loop-DOD.txt", "w");
                fprintf(fp2,"Total volume of %d cells = %f m3 \n", ncount, volume);
                fclose(fp2);
        #endif /* RP_NODE */
}
Code:
Fluent console output DEFINE_ON_DEMAND function:

I am in the node process with ID 999999
Total number of cells in cell-zone for all mesh partitions = 200
Total volume of cells in cell-zone for all mesh partitions = 1.800000 m3
I am in the host process with ID 0
Node 0 is calculating on thread # 21: Total volume of 76 cells in mesh partition = 0.697373 m3
I am in the host process with ID 1
Node 1 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 2
Node 2 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 3
Node 3 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 4
Node 4 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 5
Node 5 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 6
Node 6 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 7
Node 7 is calculating on thread # 21: Total volume of 123 cells in mesh partition = 1.095815 m3
I am in the host process with ID 8
Node 8 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 9
Node 9 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 10
Node 10 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 11
Node 11 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 12
Node 12 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 13
Node 13 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 14
Node 14 is calculating on thread # 21: Total volume of 1 cells in mesh partition = 0.006812 m3
I am in the host process with ID 15
Node 15 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 16
Node 16 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 17
Node 17 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 18
Node 18 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 19
Node 19 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 20
Node 20 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 21
Node 21 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 22
Node 22 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
I am in the host process with ID 23
Node 23 is calculating on thread # 21: Total volume of 0 cells in mesh partition = 0.000000 m3
Since the third function, gives me the exact (correct!) result that I expected, I am quite surprised that I’m experiencing issues (seems like infinite computation within first iteration) when performing the simulations.

I seek your assistance in resolving this matter. Your help would be very much appreciated, thank you in advance,

Kind regards,
Rob

AlexanderZ May 27, 2022 00:45
next time put questions explicitly

remove line68
Code:
        host_to_node_int_1(zone_identification); /* Does nothing in SERIAL */
everything below keyword return is not executed (for instance line 90-96)

as you are using C_UDSI it must be defined and memory must be allocated in Fluent GUI

writing to file should be done on HOST, the way you did in function3

RobV May 30, 2022 03:46
Dear AlexanderZ,

thank you for your reply.

I implemented your suggestions and tried to rerun the simulation/code.
Unfortunately, it doesn't run. The following error pops up in the Fluent console (when starting the simulation).

Error at host: floating point exception

The User-Defined-Scalars are indeed defined in Fluent. User-Defined-Memory was declared for other purposes, but I am not using the UDM in this function. I disabled all the printf commands to make sure these woudn't cause any issues.

When you mentioned the User-Defined-Memory, I started to think about a 'workaround' to solve this issue. Currently I am working on a DEFINE_INIT function to perform the volume calculation which is then stored in the UDM (which subsequently can be accessed by other functions). I defined the function in a similar way as the DEFINE_ON_DEMAND function mentioned in my first post (function 3), and it seems to work.

But still I can't understand why it won't properly work using function 1 and function 2...Ideally the simulation should run using that approach.

If you have any suggestions to get it work, that would be helpful.
Otherwise I will focus my efforts on the 'workaround' method.


All times are GMT -4. The time now is 20:19.