Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeouts #60

Open
flatstik opened this issue Nov 30, 2021 · 7 comments
Open

Timeouts #60

flatstik opened this issue Nov 30, 2021 · 7 comments

Comments

@flatstik
Copy link

flatstik commented Nov 30, 2021

I keep getting these kind of errors with even the not-so-complex molecules:

 File "/scratch/project_2003067/DP5/DP5/PyDP4.py", line 841, in <module>
    main(settings)
  File "/scratch/project_2003067/DP5/DP5/PyDP4.py", line 317, in main
    Isomers = DFT.RunOptCalcs(Isomers, settings)
  File "/scratch/project_2003067/DP5/DP5/Gaussian.py", line 254, in RunOptCalcs
    Completed = RunCalcs(GausJobs, settings)
  File "/scratch/project_2003067/DP5/DP5/Gaussian.py", line 308, in RunCalcs
    outp = subprocess.check_output(GausPrefix + " < "  + f + ' > ' + f[:-3] + 'out', shell=True)
  File "/scratch/project_2003067/DP5/DP5_env/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/scratch/project_2003067/DP5/DP5_env/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '/appl/soft/chem/gaussian/G16RevC.01_new/g16/g16 < Smiles_Mol_2_0ginp008.com > Smiles_Mol_2_0ginp008.out' returned non-zero exit status 1.

Is there any way to increase the timeout @HowarthA ?

@HowarthA
Copy link
Collaborator

HowarthA commented Dec 6, 2021

How long does the gaussian job run for prior to this error being generated, does the job run at all? Can you run this gaussian job outside of DP5 on the same machine without any similar issues?

@flatstik
Copy link
Author

flatstik commented Dec 6, 2021

something like 30-180 minutes - I haven't been waiting. I can run gaussian job on the same machine w/o any issues

@HowarthA
Copy link
Collaborator

HowarthA commented Dec 6, 2021

ok sure, I'll have a look into this, its not a problem I've ever come across. If you rerun the calculation DP5 should pick up the geometry optimisation from where it was left after the timeout.

@HowarthA
Copy link
Collaborator

I've made a change that may fix this, give it a try

@flatstik
Copy link
Author

flatstik commented Dec 22, 2021

I've made a change that may fix this, give it a try

Still the same issue, but it did finish the first calculation. timeout= 86400:

/appl/soft/chem/gaussian/G16RevC.01_new/g16/g16 < Smiles_Mol_0_1ginp001.com > Smiles_Mol_0_1ginp001.out
Gaussian job 1 of 318 completed.
/appl/soft/chem/gaussian/G16RevC.01_new/g16/g16 < Smiles_Mol_0_1ginp002.com > Smiles_Mol_0_1ginp002.out
Traceback (most recent call last):
  File "/scratch/project_2003067/DP5/DP5/PyDP4.py", line 841, in <module>
    main(settings)
  File "/scratch/project_2003067/DP5/DP5/PyDP4.py", line 317, in main
    Isomers = DFT.RunOptCalcs(Isomers, settings)
  File "/scratch/project_2003067/DP5/DP5/Gaussian.py", line 254, in RunOptCalcs
    Completed = RunCalcs(GausJobs, settings)
  File "/scratch/project_2003067/DP5/DP5/Gaussian.py", line 308, in RunCalcs
    outp = subprocess.check_output(GausPrefix + " < "  + f + ' > ' + f[:-3] + 'out', shell=True,timeout= 86400)
  File "/scratch/project_2003067/DP5/DP5_env/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/scratch/project_2003067/DP5/DP5_env/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '/appl/soft/chem/gaussian/G16RevC.01_new/g16/g16 < Smiles_Mol_0_1ginp002.com > Smiles_Mol_0_1ginp002.out' returned non-zero exit status 1.

@flatstik
Copy link
Author

flatstik commented Jan 4, 2022

Update: It only hangs up with -gnomesw but not with -gnmesw

@flatstik
Copy link
Author

flatstik commented Jan 8, 2022

And after two weeks of preparing all the isomers for two smiles strings, even the -gnmesw hangs due to memory limit (and I cannot increase it more than that):

Reading experimental NMR data...
[PosixPath('Proton'), PosixPath('Carbon')]
Processing Proton Spectrum
slurmstepd: error: StepId=10077369.0 exceeded memory limit (67121145856 > 67108864000), being killed
srun: Exceeded job memory limit
slurmstepd: error: *** STEP 10077369.0 ON r07c52 CANCELLED AT 2022-01-08T02:02:11 ***
slurmstepd: error: StepId=10077369.0 exceeded memory limit (67121145856 > 67108864000), being killed
srun: Exceeded job memory limit
srun: Exceeded job memory limit
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: r07c52: task 0: Killed
srun: launch/slurm: _step_signal: Terminating StepId=10077369.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants