[RanaP]

Hello,
I'm a student from India that recently has obtained a grant to work on CFD at an European university. Unfortunately my supervisor does not have CFD on his CV.
Can you guys please guide me into what means active CFD research today? I'm a decent programmer that can code in C++, C# and Java almost any numerical algorithm I can understand. My experience in CFD was in implementing a structured invisicid flow solver (Euler equations) in 2D.
Thanks and I'm waiting for your suggestions,

Rana

[RanaP]

I was think for example at implementing a CUDA parallel NS code ... Do you think this is a valid research domain ?

[cfd_newbie]

Originally Posted by RanaP

I was think for example at implementing a CUDA parallel NS code ... Do you think this is a valid research domain ?

That will be more of a extended Master's project.

For Phd, I suggest don't go to NS, it's a dead beaten horse.
Look into LES or DNS, they are the future.
DNS code with CUDA could be a good topic.

Regards,
Raashid

[RanaP]

Thanks Raashid.

[RanaP]

Originally Posted by cfd_newbie

That will be more of a extended Master's project.

For Phd, I suggest don't go to NS, it's a dead beaten horse.
Look into LES or DNS, they are the future.
DNS code with CUDA could be a good topic.

Regards,
Raashid

What do you think about ILES on CUDA ? The problem is that I have only one CUDA device and I don't know how relevant such a simulation will be. Maybe I can parallelize some parts of the code on the CPU using OpenMP and the most intensive parts on the GPU using CUDA.

Can you recommend me some bibliography ?

Thanks,

Rana

[cfd_newbie]

Can you drop me an email, I will try to connect you to someone who can tell you better in this area.

[cfdnewbie]

Originally Posted by cfd_newbie

That will be more of a extended Master's project.

For Phd, I suggest don't go to NS, it's a dead beaten horse.
Look into LES or DNS, they are the future.
DNS code with CUDA could be a good topic.

I suppose by NS you meant RANS, not just NS, right?
I agree, LES is a wide and open field for a PhD research focus....
but a NS solver with CUDA has already been done (see Andreas Kloeckner and colleagues http://mathema.tician.de/academic/research), so I wouldnt go that way....

multiscale numerics is the big thing at the moment, so look into LES / high order that kind of stuff!

[cfdnewbie]

Originally Posted by RanaP

What do you think about ILES on CUDA ? The problem is that I have only one CUDA device and I don't know how relevant such a simulation will be.

Rana

ONE GPU??? hm, truth be told you will not be able to do much (if any) meaningful LES on one GPU.... if you don't have access to massive computing power and will not in the future for your PhD, then stay away from anything turbulent...

[andy_]

Originally Posted by RanaP

I was think for example at implementing a CUDA parallel NS code ... Do you think this is a valid research domain ?

Not at a reasonable university and I suspect you might misunderstand the nature of a PhD in CFD. A PhD in CFD is about researching new approaches in CFD and not about coding. It is perfectly acceptable to use a third party to create the code. This is usually done by taking a previous researcher's code and modifying it to include the new techniques being studied but employing a programmer to write code would also be fine. Writing code during a PhD in CFD is usually a requirement but it is pure overhead.

If you were researching something new about parallel processing then developing most of the code on one processor is fine (or used to be when parallel computers were rare and expensive a few decades ago). Of course, when generating data for the PhD performing a set of runs on representative hardware would be expected and this is usually achieved by accessing remote supercomputing facilities freely available to university researchers after applying and filling in paperwork.

[cfdnewbie]

I have to agree with Andy on this (at least with most of what he said). A PhD might (and in my experience will) include coding, since that's what the C in CFD stands for, and since at one point in time you have to test your ideas. But the code itself is just a means to an end, i.e. it should verify your ideas, show that a new scheme works, prove your parallelization algorithm, validate your new turbulence model, do a DNS for you etc...
You implementing an algorithm sbd else thought up is not what a PhD is about....


There are occasions when big research codes are written from scratch, e.g. when a fundamental paradigm shift comes along (like going from FD to sth nicer, going from incomp, to comp and such), but that's usually done as a team effort by all the phd students in the working group, because writing a competitive DNS/LES state of the art code would be enough to keep a single researcher busy for most of his phd time....

[RanaP]

@cfdnewbie
@andy

Hello,
By one GPU device I've meant a single graphical card with 3GB of RAM on an 6 core machine. I could use this machine for developing my code (in this phase I consider an implicit large eddy simulation using a high order finite volume solver).
From what I've understand I could be allowed to use a 32 processors cluster, but I consider paying for a few hours of run on a private cluster made of a few GPU Tesla cards.
I know that implementing an already made algorithm is not a good PhD subject, so I'm also working at developing my own flavour of a space integration scheme (basically a blend of two methods).
Do you think that a parallel ILES code on CUDA with an "original" integration scheme is worth it ?
(Also, if you have any suggestions about the direction in which to concentrate my efforts I'm open to new ideas. As I've mentioned in my first post I'm a bit on my own here so I could use some constructive critiques).
Thanks,
Rana

[RanaP]

I put some emphasis on implementing a new code simply because I'm very fast at coding once I have a clear algorithm in my head.

[cfdnewbie]

Originally Posted by RanaP

@cfdnewbie
@andy

Hello,
By one GPU device I've meant a single graphical card with 3GB of RAM on an 6 core machine. I could use this machine for developing my code (in this phase I consider an implicit large eddy simulation using a high order finite volume solver).

for purely code development, that should be sufficient, however, you won't get any idea about how well your code scales, which is essential for high performance computing and writing to proposals to get time on the big machines...

You will be limited to very simple test cases at low Re, but as a proof of concept, it might work....

From what I've understand I could be allowed to use a 32 processors cluster, but I consider paying for a few hours of run on a private cluster made of a few GPU Tesla cards.

32 is a reasonable size for initial scaling tests, at least...the question is: how familiar are your with parallel programming, especially on GPUs? I have seen a number of attempts to construct GPU code that beats CPU codes, but most of them unsuccessful....I'm not saying it is impossible, it is just difficult if you have just beginner's knowledge...


my point is: if your goal is to do LES of a meaningful flow with high Re, then you will need to get access to the really big clusters. But to get time on these machines, you need to validate your code and prove its scaling on probably more than 32 nodes...

I know that implementing an already made algorithm is not a good PhD subject, so I'm also working at developing my own flavour of a space integration scheme (basically a blend of two methods).
Do you think that a parallel ILES code on CUDA with an "original" integration scheme is worth it ?

yes, if the integration scheme is indeed original, that in itself would make a reasonable phd project....but make sure your scheme is really new and original, and not sth done by sbd 20 years back (it happens quite often ).

I'm interested in what you mean by ILES....do you actually propose to play around with the approximation errors of your discretization, or do you just mean "LES without SGS model"??

(Also, if you have any suggestions about the direction in which to concentrate my efforts I'm open to new ideas. As I've mentioned in my first post I'm a bit on my own here so I could use some constructive critiques).
Thanks,
Rana

just a question: what do you mean by high order FV? ENO/WENO type stuff? or 2nd order.... because keep in mind: large stencils seriously reduce your parallelization efficiency....

[RanaP]

ILES - LES without SGS model (I'm still in the phase of reading articles about this, so I will probably change my mind).

High order 4th order or up ... I know a bigger stencil will hinder the parallelization this is why my code will be mostly a research code optimized for a a single machine with multiple processors and multiple GPUs. You can add 4-5 Tesla GPUs on a high performance desktop (2 quad code or 2 six cores).

For parallelizing some portions of the code on CPU I will use OpenMP for the GPUs I will use CUDA. I'm well versed in using both parallel APIs.

Using our 32 processors cluster doesn't seem to attractive to me, I will be forced to use MPI (didn't used this yet) and a slow (compared with sending data to a GPU on the same machine) network.

Rana

[andy_]

Do you intend to perform a PhD in Europe within a group that cannot supervise research in CFD or is it that your Indian supervisor is not familiar with CFD and you intend to visit an established CFD group in Europe?

Normally in Europe if a student chooses their own PhD topic they will have to find a supervisor that thinks the student is worth their time supervising. This usually means that the topic is relevant to the supervisor's area of research and that the student stands a reasonable chance of producing something worthwhile.

Active researchers normally have a list of projects they would like to work on in the future when resources become available and they also possess resources like existing codes, computers and people working on related topics. If you know the broad area you wish to work in then I would suggest you may be better off approaching the head of a few CFD groups with an outline of who you are and your areas of interest. Then if there is interest, talk around one or two topics potential supervisors put forward. I would also suggest making it clear in the first paragraph that you already have funding because they will almost certainly be receiving many requests from students seeking funding to come and study.

[RanaP]

With my supervisor I can research what CFD subject I want ... so you see my problem, I'm trying to clarify my ideas in this public forum, because I don't want to spend 3-4 years chasing dead subjects.

[cfdnewbie]

Originally Posted by RanaP

ILES - LES without SGS model (I'm still in the phase of reading articles about this, so I will probably change my mind).

That's more of dirty DNS, strictly speaking, implicit LES is when you actively modify your scheme to make sure your numerical error models the viscosity... just warning you that some people will take offense if you call a dirty DNS an implicit LES....

High order 4th order or up ... I know a bigger stencil will hinder the parallelization this is why my code will be mostly a research code optimized for a a single machine with multiple processors and multiple GPUs. You can add 4-5 Tesla GPUs on a high performance desktop (2 quad code or 2 six cores).

ok, 4th and up is ambitious for FV. why do you want to use FV schemes, then? I've always wondered why people stick with FV when using orders higher than 2...
Just keep in mind that with that rig you will not be able to compute really high RE flows in a reasonable time,,,so your chances of publication of your work will depend on your new space integration alone....just keep that in mind!

Using our 32 processors cluster doesn't seem to attractive to me, I will be forced to use MPI (didn't used this yet) and a slow (compared with sending data to a GPU on the same machine) network.

don't worry about MPI, all large scale codes use (at least 99%), so there's a lot of knowledge out there about it!


From my perspective, the space integration is the only novel thing about your project. CUDA and dirty DNS has been done before, and FV seems to me to be not as interesting at the moment. If however your space integrator is good, then you can build a demonstrator code on your GPU architecture. That would probably warrant a PhD.

However, you should also keep in mind that here in Europe, team work in research groups is a big thing. A professor will typically have a number of PhD students working on rather similar projects, supporting each other and sharing workload, coding and such. So a lot of professors not only look for PhD students with money, but for those who are willing to contribute to their group and maybe find a project that fits in.....

cheers!

[RanaP]

What do you suggest instead of FV for high order schemes, Finite elements (I was there I can go again) or spectral methods (this is new for me) ?

[cfdnewbie]

If you have any experience in FE, then I'd suggest looking into discontinuous FE! What country Europe are you looking at? There are some active groups in that area in the EU...

[RanaP]

UK.

I suppose you mean discontinuous Galerkin methods ...