User Fortran with CFX 14.5.7 on winnt_amd64
 

Hello,
when looking for a means for doing time integration of a expression, I read a little into user fortran. As the fortran environment is complex and expensive I did some research (to be honest: more trial-and-error) with free Mingw64.
It is a little tricky as the systems are not compatible concerning function names. At least I succeded in implementing some subroutines with simple arithmetic, not using any of the CFX internal calls. It is enough for storing a previous value, doing time integration or calculting a derivative of an expression.
The essential thing is to provide upper case aliases for the lowercase function names of Mingw, as CFX expects them in uppercase despite they appear lowercae in CFX-Pre. And prohibit adding underscores to function names.
Works in my environment ANSYS CFX 14.5.7 both single and double precision, no idea whether it works distributed, partitioned etc.
You need a installation of TDM64-GCC 4.8.1-3, include gfortran.
In a DOS-Box (best with <TDM-GCC-Inst-Dir>\bin in the PATH variable) 'cd' to the directory where you unpack the zip to. Customize the paths in compile.bat to your paths and run it.
Please give iti a try.

Time Integration itself can be done more consitently in the solver itself:
http://www.cfd-online.com/Forums/cfx...tml#post465756

HLo,

Thank you so much for posting this. I have been looking around a lot recently for a way to use the MinGW compiler instead of the Intel one.

I have found a way to get the CFX Utility Routines working for use with MMS based on your solution.

The problem boils down to the fact the GNU c++ preprocessor is case in-sensitive and converts the output symbols in lower case, but the object file library (in my case solver-hpmpi.lib) has the (non-mangled) symbol names in uppercase.

When I compiled naively I got errors like:

undefined reference to 'user_assemble_info'

This led me to the above conclusion. Since we can't recompile the libraries (don't have the source) we probably need to rename or create aliases for the symbols at compile-time.

The solution I found was to add the command line option you are already using to the linker for each symbol (subroutine that needs to be converted) e.g. :

In addition, I created an additional .def file (named solver-hpmpi.def) containing:

So don't forget to append this new .def file into the compile/link command.

Using this technique I was able to compile the dll for the AdVarSource.F example located in %CFXROOT%/examples/UserFortran/ (using TGM-MinGW64)

Disclaimer: I don't really get why this works I thought I would just need the .def or the linker but it only works if both are included.

Disclaimer: I haven't tested the .dll file yet. Will edit this post when I have confirmed that it works.

Again, thanks HLo, I wouldn't have been able to get this far without your help. :)

Future work:
Maybe somebody could come up with a modified cfx5mkext.ccl file (or batch file) that automatically converts the symbols for all the CFX utility routines that are being used. That way everyone on windows could easily compile user fortan without depending on the expensive intel compiler.

Symbol Table
 

Hello njcyck,

I tried with the defsym switch myself, but no success then.
With your proposal I'll give it a new try now.
In an other experiment I hexeditet the object file from mingw and uppercased the function name (here SET_A_0), linked it, but CFX solver crashed. Please let me know about your experiments and vice versa.

BTW: did the example run in solver?

Thanks
HLo

HLo,

I forgot to mention a couple of things:

I am including the .def file that is modelled after yours (I called it AdVarSource.def):

Code:

---

LIBRARY "ADVARSOURCE.DLL"
EXPORTS
USER_SOURCE2 = user_source2

---

and I used the linker option:

Code:

---

-Wl,--defsym,set_a_0=set_a_0

---

This is in addition to the other --defsym commands I used to convert the subroutine names.

If I exclude either the --defsym set_a_0=set_a_0 option or the AdVarSource.def file (or both) it fails to compile/link.

I am still confused about something:
the --defsym set_a_0=set_a_0 command seems to be renaming a symbol to itself? I'm glad that it works, but it really doesn't make sense.

I also removed the -Wl,--defsym,set_a_0=set_a_0 option from your batch file and ran it: it still makes the Utility.dll file without any issues. You don't need it unless there's a subroutine called SET_A_0.

Just to be clear, I am using cfx5mkext.exe; here is the important chunk from the cfx5mkext.ccl file I am using (obviously I haven't bothered with the double precision chunk just yet):

defsym
 

Hello,
yes the set_a_0 stuff is not needed, if it is not used.

First: Does the example run in CFX? And what is your system when you use solver-hpmpi.lib?

I did some further research, as my code crashes with segmentation fault, when using one of the CFX routines like SET_A_0, MESAGE etc.
One thing that gave me a hint was that one has to change the definition of argument CRESLT form CHARACTER*(*) to fixed length. The reason for it -I assume- is the different handling of character arguments. The compiler argument -iface:qvf of Intel Fortran influences the order how the hidden length values are passed. In gfortran there is no such switch, I read somewhere, that the hidden strings lengths are appended to the argument list. Whereas the -iface:qvf gives them immedeately following the string arg.
As a consequence the different arg order or hidden additional args mess up the processor stack, which does not matter as long as these are not used in your program.
You may verify if you include an additional arg NLNG after CRESLT and declare as INTEGER,VALUE :: NLNG. if you watch it it gives 4.
The key to solve the entire problem is the correct number and order of the passed arguments, their bytelength I think is always 4 (pointer or integer)

For double precision in your ccl fflags should be the same plus -fdefault-real-8

HLo

Hello, do you have any experience on User Fortran Compiling on Win X64 OS? Ansys 14.0 is used on my PC. I installed Mircosoft Visual 2012 and Intel Visual Fortran Composer XE2013. But when I try to compile the user fortran in the tutorial file, a message appears like "The compiler command ifort can't be found. Please ensure that the compiler is on your path and try again." Can you help me? Thx.

ifort not found
 

Hello Raijin,
First: the directory, where the ifort executable resides, must be in your system's path:
Open a dos box and enter 'set path' (without quotes), the printout should contain "C:\Program Files (x86)\Intel\Composer XE 2013 SP1\bin\intel64" or similar, otherwise Composer is not installed correctly.
I myself worked with an eval version for 32-bit, which compiled after some tweaks, but did not work, as CFX is 64-bit.
The tweaks go into the mess with appended underscores, which ifort creates, but cannot be linked, maybe something messed up in the examples or I didn't get far enough
Sorry that's all I can offer, I focus on Mingw
HLo

HLo,

I have been doing some testing and it has not been successful. :(

By system do you mean my machine specs?
I'm on windows 7 64-bit with an Intel i7 cpu. Also I'm using ANSYS 13.0

Whenever I try and run something that uses the CFX utility routines I get the error:

+--------------------------------------------------------------------+
| ERROR #001100279 has occurred in subroutine ErrAction. |
| Message: |
| Signal caught: Segmentation violation |
| |
| |
| |
| |
| |
+--------------------------------------------------------------------+

+--------------------------------------------------------------------+
| ERROR #001100279 has occurred in subroutine ErrAction. |
| Message: |
| Stopped in routine FPX: SIG_HANDLER |
| |
| |
| |
| |
| |
+--------------------------------------------------------------------+

I assume this is the same segmentation fault that you have run into.

Your theory about the string lengths is intriguing. I also changed the CRESULT argument length to fixed to satisfy the compiler.

I am trying to understand the consequences, so please confirm my thinking or correct me if I'm wrong:

Whenever the intel compiler encounters a variable-length character string in an argument list for a subroutine, it writes the machine instructions so that the length of the string is passed as an additional (hidden) variable which immediately follows the string itself. If an intel compiler also compiles the calling code for the routine it will write instructions to calculate the length of the string that is being passed (at run time), and pass that length argument immediately following the string itself.

Whenever the gfortran compiler encounters a variable length character string in an argument list, it does something similar except it appends the hidden string length list to the end of the argument list. When the calling code is compiled it also writes the instructions to append the string lengths (calculated at runtime) to the end of the list. Everything works well if the subroutine and the calling code are compiled with gfortran.

So we have a problem because the subroutines we are calling expect that string lengths immediately follow strings, and there are no gfortran compiler options to tell gfortran to pass the hidden string lengths in a different way.

Ok so if that is true then you are saying we just need to (whenever calling a CFX utility routine) replace all of the variable length string arguments with fixed width arguments, and insert those string lengths (integers) as makeshift hidden arguments after every argument that is expected to be a variable string.

Is this correct? I am just trying to get a clear picture of what's happening, so I haven't tried anything yet. Let me know what you think.

hidden string args
 

Hello,
"my" error ist the same, but ONLY when I call CFX subroutines like SET_A_0, ERRMSG, MESAGE etc. so if you can avoid them, like in my example it should work. I didn't succeed in calling these CFX subs without error.
To overcome the problem of hidden string lengths at end of arg list try to wrap character variables into a new type and explicitely add the string lengths in your functions:
SUBROUTINE BLABLA(NLOC,NRET,NARG,RET,ARGS,CRESLT,NLNG,
& CZ,NLNG2,DZ,IZ,LZ,RZ)
and define:
TYPE CWRAP4
CHARACTER(LEN=4) str
END TYPE
INTEGER,VALUE :: NLNG, NLNG2
the latter for the CZ stack
...
TYPE(CWRAP4) :: CRESLT
and later
CRESLT%str = 'GOOD'
So in theory the CALL ERRMSG(string) in gfortan should be replaced with CALL ERRMSG(wrapped_str, %VAL(length_of_wrapped_str)) (the hidden length is always "by value" not like fortran default "by ref")

To the linking with solver-....lib:
I think it is better to replace the lowercase symbols of gfortran to uppercase ones with 3rd party tools e.g objconv, then the "def" stuff is obsolete:
1) compile file.F to file.o:
gfortran -mthreads -I "C:\Program Files\ANSYS Inc\v145\CFX\include" -shared -fno-underscoring TStat_Control.F -c
2) then uppercase your symbols (output in TStat_Control.obj):
objconv -nr:errmsg:ERRMSG -nr:blabla:BLABLA TStat_Control.o
3) then link
gfortran -mthreads -I "C:\Program Files\ANSYS Inc\v145\CFX\include" -shared -L"C:\Program Files\ANSYS Inc\v145\CFX\lib\winnt-amd64" -o winnt-amd64\example.dll -Bstatic -static-libgfortran -static-libgcc TStat_Control.obj -lsolver-XXmpi
(XX for your environment)
I'd recommend the "-static-libgfortran -static-libgcc" options because otherwise these libraries are called at runtime and they point to the libs in the TDM installation directory, so on another machine where they are missing .....

Solved
 

Hello,
finally got the solution:
1) compile source file with options:
gfortran -fno-underscoring -c -I "C:\....CFX\include" file.F
in source do CHARACTER to TYPE mangling, to enable string length arg directly after the string pointer, and always CFX's char function with string pointer and length pair
2) do "objconv.exe" to uppercase all references to CFX subroutines e.g:
objconv -nr:set_a_0:SET_A_0 -nr:mesage:MESAGE file.o file.obj (will create file.obj)
3) create a def file with contents:
LIBRARAY "MyDLL.dll"
EXPORTS
MYSUB = mysub @1
MYOTHERSUB = myothersub @2
4) link using -static-libgfortran and -static-libgcc against a rebuilt library of solver-pcmpi.lib (my case) in mingw format:
reimp.exe -d "C:\Pro....\CFX\lib\winnt-amd64\solver-pcmpi.lib"
dlltool.exe -d solver-pcmpi.def -k -l libsolver-pcmpi.a

Example for double prec.:
gfortran -fno-underscoring -fno-range-check -cpp -fdefault-real-8 -c -I "C:\Program Files\ANSYS Inc\v145\CFX\include" "TStat_Control.F"
objconv -ew2015 -nr:wrttxt:WRTTXT -nr:out_section_header:OUT_SECTION_HEADER -nr:set_a_0:SET_A_0 "TStat_Control.o" "TStat_Control.obj"
gfortran -O2 -mthreads -static-libgfortran -static-libgcc -shared -L. "TStat_Control.obj" "TStat_Control.def" -ldoublesolver-pcmpi-imp -o "winnt-amd64\double\TStat_Control.dll"
Import library "libdoublesolver-pcmpi-imp.a" built with:
reimp -d "C:\Program Files\ANSYS Inc\v145\CFX\lib\winnt-amd64\double\solver-pcmpi.lib"
dlltool -d "solver-pcmpi.def" -k -l libdoublesolver-pcmpi-imp.a
Definiton file "TStat_Control.def" contains:
LIBRARY "TStat_Control.dll"
EXPORTS
TESTSUB = testsub @1

Does someone give it a try?
HLo

HLo,

Thanks so much for posting your solution. I will try it in the next couple of days and let you know how it goes.

Happy New Year!

HLo,

It works!! Here are the steps I followed:

1. I downloaded objconv from http://www.agner.org/optimize/. I used the .exe file given in the folder.

2. I downloaded the reimp source from here, and built it using TDM-GCC with the following commands:

3. I created the following batch file and ran it:

MomentumSource1.def:
MomentumSource1.F:
At this point the .dll file was created as expected (109 KB)

I ran the test program in serial. It prints "TEST" (without quotations) a bunch of times at every timestep.

successful test
 

Hello njdyck,
congrats to your successful test.
I wasn't sure whether it worked in another environment. In my case I had to use solver-pcmpi.lib instead of solver-hpmpi.lib. I also was afraid that the successful run depended on a certain Microsoft VC++ runtime.
To be sure it works reliably we should run all the tutorials' fortran examples later.
Remark: For double precision use flag '-fdefault-real-8' when creating the object file and put dll in winnt-amd64\double directory
Of course in other experiments the wrapped strings maybe longer than 4
I'll post other test results when applicable.

HLo

Ok so it's nice to be able to use the MinGW compiler instead of Intel's for compiling fortran code, however it would be much nicer if we didn't have to mangle the character strings and write messy subroutine calls like:

It might be possible to wrap all of the cfx utility routines into a new library whose source code looks something like:

Using objconv or objcopy we could easily redifine all of the symbols in solver-XXmpi.lib to have "_CFX" appended to avoid name clashes. Also, we could compile it so that the symbol (wrapping subroutine) names are mangled in the gfortran fashion (uppercase); then the user subroutines don't need to redefine symbol names that correspond to calls to cfx utility routines.

This will of course be less efficient than mangling the user source code, but it will enable the same (unmangled) code to be compiled using intel and mingw compilers. Intel compilers will use the original lib and gfortran compilers use the wrapped lib.

I also don't know how much inlining/optimization the gfortran compiler will do so it might just increase the final .dll size without to much of a performance hit.

What do you think? Would this be useful? or is there an easier way to strip out the CHARACTER -> TYPE mangling from the user source code?

#define macros
 

Hello njdyck,

should work. an alternative may be using macros (#define) for that
and later
Remark: The wrapping isn't mandatory for CRESLT, simplest use:
HLo

BTW: After some further reading, thinking and experiments the reason why cfx crashes when CRESLT is defined as CHARACTER*(*) is clear:
As gfortran expects all variable string lengths added at the end of the argument list, but CFX never delivers them, a variable length definition of CRESLT therefore will pop off 4 bytes too much from stack.

Current status
 

Hello,
currently I got so far, that all code from examples and tutorials hasn't to be heavily changed, all modified arg lists and string pointer mangling is outsourced to 2 include files which are conditionally preprocessed, when using gfortran. They don't disturb if using ifort. There are only 1 global and 2 mods in each subroutine, everthing else, accompanying #define macros (aliases) for the CFX subroutines and functions are declared in header files:
1) #include "gfortran.h" at the beginning of the file has global #defines and macros for each CFX function and subroutine
2) #include "usercelargs.h" in each subroutine has the modified CEL arg definiton and a type specification for each used CFX function
3) each sub needs a gnuexport(MYSUBNAME) directive
I'll post the files and example when code and compilation script are "ripe"
@ njdyck
I went away from wrapping strings in another type, as this cannot be implemented with #define macros. I use the LOC(S) (wrapped in another function CPTR__(S) though) in combination with %VAL so a CFX argument string S is replaced with %VAL(CPTR__(S)),%VAL(LEN(S)). The predefined LOC(S) function in a 1-liner CFX call appends a hidden additional length for each S to the end of the arg list therefore puts them on the stack and the runtime behaviour is unpredictable (probably segm. fault). I assume that is a bug in the compiler. Proof is given in the assembly code. The first puts 4 args to the stacks, the latter 3, as desired (to proof: compilation with -S to assembly file)
or
with
Luckily, if calling a function which returns a string, both compilers put 2 hidden args in front of the arg list (string address and string length) in the same order, so no tricks there.

Remark to processor stack: on Win 64-bit systems the calling convention is FASTCALL, so the first 4 args are put to 8-byte registers, only the remaining args to conventional stack space. (can be seen in assembly code)

HLo

HLo,

I started working on my own fortran-based library and I'm not having any problems using #define macros in the code. For example the following code can be compiled into a library:

I appended _cfx to the necessary symbols in the mpi-XXX.a lib to avoid name clashes. I can then compile a dll from the code in my last post and it runs no problem.

This does, however, restrict the string length to fixed values. Lately I have not got any variable length string code to compile, so I have abandoned them altogether.

Anyway, I look forward to seeing your solution (sounds a little cleaner). Let me know of any way I can help.

Hello!
I am use ANSYS CFX 14.5.7.1 and Intel Composer Compiler SE 2011, Windows 7x64.

I am replace string:

CHARACTER CRESLT*(*)

on code:

TYPE CWRAP10
CHARACTER(LEN=10) str
END TYPE

TYPE(CWRAP10) :: CRESLT
..............................................

CRESLT%str = 'GOOD'


in cfx example #19, file Tstat_Control.F


I have errorr:


+--------------------------------------------------------------------+
| PROBLEM ENCOUNTERED WHEN EXECUTING CFX EXPRESSION LANGUAGE |
| |
| The CFX expression language was evaluating: |
| Static Temperature |
| |
| The problem was: |
| Internal Error - CEL ICODE array mismatch |
| |
| FURTHER INFORMATION |
| |
| The problem was encountered in executing the expression for: |
| ACOn |
| The complete expression is: |
| Thermostat Function(TSensor,TSet,TTol,atstep) |
| The error occurs on sub-expression: |
| TSensor |
| |
| BACKGROUND INFORMATION |
| |
| The error was detected at one location. The same problem may be |
| present at other locations - that has not been investigated. |
| The following values are for the first location which has the |
| problem. |
| |
| |
| END OF DIAGNOSTIC OUTPUT FOR CFX EXPRESSION LANGUAGE |
+--------------------------------------------------------------------+
Unexpected IP data copied: ABSCOEF_FL1


================================================== ====================
| Timestepping Information |
----------------------------------------------------------------------
| Timestep | RMS Courant Number | Max Courant Number |
+----------------------+----------------------+----------------------+
| 3.0000E+00 | 0.00 | 0.00 |
----------------------------------------------------------------------

================================================== ====================
TIME STEP = 1 SIMULATION TIME = 3.0000E+00 CPU SECONDS = 5.202E+00
----------------------------------------------------------------------
COEFFICIENT LOOP ITERATION = 1 CPU SECONDS = 5.202E+00
----------------------------------------------------------------------
| Equation | Rate | RMS Res | Max Res | Linear Solution |
+----------------------+------+---------+---------+------------------+
| U-Mom | 0.00 | 2.2E-02 | 8.9E-01 | 1.7E-04 OK|
| V-Mom | 0.00 | 0.0E+00 | 0.0E+00 | 0.0E+00 OK|
| W-Mom | 0.00 | 1.5E-01 | 7.6E-01 | 5.9E-05 OK|
| P-Mass | 0.00 | 1.5E-04 | 6.5E-03 | 39.4 7.4E-02 OK|
+----------------------+------+---------+---------+------------------+
----------------------------------
Error in subroutine GET_SPECVAR :
A recursion problem was encountered when evaluating the expression for
"Temperature" at the following location: "Inlet". Please carefully check that
the result of your expression does not depend on itself.
GETVAR originally called by subroutine CAL_TEMP_BCS

+--------------------------------------------------------------------+
| ERROR #001100279 has occurred in subroutine ErrAction. |
| Message: |
| Stopped in routine GV_ERROR |
| |
| |
| |
| |
| |
+--------------------------------------------------------------------+

+--------------------------------------------------------------------+
| An error has occurred in cfx5solve: |
| |
| The ANSYS CFX solver exited with return code 1. No results file |
| has been created. |
+--------------------------------------------------------------------+


What s wrong?

I need recompiling platform-mpi?


settings for compiler?

Ves,

The focus of this thread is to discuss techniques to compile CFX friendly user fortran code with the open source MinGW (gfortran) compiler for Windows. You said you are using the intel compiler, so you don't need to use any of the tricks here. Just compile the code as you normally would with variable length string arguments, and everything should line up.

One piece of advice I can give is make sure you are using the Intel Composer Command Prompt (look for it under the start menu). If you try and use the windows command prompt it can't find 'link' or something like that.

yes, I have windows command promt. CFX command promt compiler not found