[FLASH-USERS] 4000+ cpus on Franklin

[Klaus Weide]

On Mon, 31 Aug 2009, James Guillochon wrote:

> Hi all,
> 
> I'm trying to restart a FLASH simulation on Franklin. If I run on 2000 cpus,
> the job runs fine, however if I try to push the code to 4000 cpus, I get the
> following error:
> 
> abort_message [flash_convert_cc_hook] Trying to convert non-zero mass-specific
> variable to per-volume form, but dens is zero!

James, 

As you have probably noticed, the problem (or at least the symptom) is 
essentially the same as what you reported to flash-users in April: 
unwanted zero values in DENS_VAR.  The code in flash_convert_cc_hook, 
which triggers the abort, does essentially the same as the code in 
amr_prolong_gen_unk1_fun, where the problem showed up in your previous 
report.  I don't know what was ultimately the cause of the previous 
problem, but you have solved it somehow; could the cause be similar this 
time?

Please remind us whether you are using the latest released version of 
FLASH.  Also, 
 - does this problem occur immediately after restart, or some 
   time later?  
 - What are the last log file messages before the abort?
 - There should also have been a message on standard output, with 
   additional information (PE, ivar, and value). Do you have that?
 - Does the problem occur on several PEs at the same time (your should
   then see several of the standard output messages), or only on one CPU?


Klaus