Can't redirect stdout from calculator singlepoint()

[craldaz]

I have an application that will be running a lot of xtb calculations. I don't want my main output to be clogged with a lot of output from xtb itself but instead would like to write this information to a different file using redirection.

Normally this is possible with contextlib in python but contextlib can't capture the stdout from Calculator.

For example, the following code still prints out the SCC iteration information to the standard output (which I don't want). It should be written to log.txt instead.

import contextlib
import numpy as np
from xtb.interface import Calculator
from xtb.utils import get_method
from xtb.interface import Environment

numbers = np.array([8, 1, 1]) 
positions = np.array([
[ 0.00000000000000, 0.00000000000000,-0.73578586109551],
[ 1.44183152868459, 0.00000000000000, 0.36789293054775],
[-1.44183152868459, 0.00000000000000, 0.36789293054775]])

calc = Calculator(get_method("GFN2-xTB"), numbers, positions)

with open('log.txt','a') as f:
    with contextlib.redirect_stdout(f):
        res = calc.singlepoint()  # energy printed is only the electronic part
E = res.get_energy()
print(E)
g = res.get_gradient()
print(g)

c = res.get_charges()
print(c) 

[awvwgk]

The Fortran library is writing to a formatted unit, not sure if Python is actually able to intercept this.

For this purpose you can create a formatted unit by an API call instead:

calc.set_output('log.txt')
res = calc.singlepoint()
calc.release_output()

[craldaz]

Thanks for the response.

This prevents output for a little while then it all dumps to stdout anyway.

For example, here I am doing a geometry optimization that calls xTB, the xTB output is suppressed until optimization step 3.

It's not super important.

 On opt step 1 for node 0
 In backtrack
 Iter: 1 Err-dQ = 5.91717e-04 RMSD: 1.61973e-02 Damp: 1.00000e+00
 Iter: 2 Err-dQ (Best) = 1.07559e-08 (5.91717e-04) RMSD: 5.02527e-05 Damp: 1.00000e+00 (Good)
 Cartesian coordinates obtained after 2 microiterations (rmsd = 5.025e-05 |dQ| = 1.076e-08)
 Decreasing DMAX to 0.05919417044389267
 dEstep=-0.7384
 Node: 0 Opt step: 1 E: -0.7384 predE: -1.1099 ratio: 0.665 gradrms: 0.00158 ss: 0.059 DMAX: 0.059
 gmax 0.0163 disp 0.6662 Ediff -0.7384 gradrms 0.0016

 updating coord basis
 Timings: Build G: 1.412 Eig: 0.465

 On opt step 2 for node 0
 updating prim hess
 In backtrack
 Iter: 1 Err-dQ = 6.82791e-04 RMSD: 1.39723e-02 Damp: 1.00000e+00
 Iter: 2 Err-dQ (Best) = 2.73348e-08 (6.82791e-04) RMSD: 5.48664e-05 Damp: 1.00000e+00 (Good)
 Cartesian coordinates obtained after 2 microiterations (rmsd = 5.487e-05 |dQ| = 2.733e-08)
 [INFO] decreasing step: does not satisfy sufficient decrease condition.
 step 0.02960
 Iter: 1 Err-dQ = 1.70698e-04 RMSD: 6.98613e-03 Damp: 1.00000e+00
 Iter: 2 Err-dQ (Best) = 1.70834e-09 (1.70698e-04) RMSD: 1.37165e-05 Damp: 1.00000e+00 (Good)
 Cartesian coordinates obtained after 2 microiterations (rmsd = 1.372e-05 |dQ| = 1.708e-09)
 Decreasing DMAX to 0.029597085221946335
 dEstep=-0.1123
 decreasing DMAX
 Node: 0 Opt step: 2 E: -0.8507 predE: -0.7656 ratio: 0.147 gradrms: 0.00278 ss: 0.030 DMAX: 0.025
 gmax 0.0400 disp 0.2874 Ediff -0.1123 gradrms 0.0028

 updating coord basis
 Timings: Build G: 1.419 Eig: 0.466

 On opt step 3 for node 0
 updating prim hess
 In backtrack
 Iter: 1 Err-dQ = 8.62347e-05 RMSD: 5.76099e-03 Damp: 1.00000e+00
 Iter: 2 Err-dQ (Best) = 2.43285e-10 (8.62347e-05) RMSD: 6.45243e-06 Damp: 1.00000e+00 (Good)
 Cartesian coordinates obtained after 2 microiterations (rmsd = 6.452e-06 |dQ| = 2.433e-10)
   1   -955.3072079 -0.955307E+03  0.598E+00    0.82       0.0  T
   2   -960.0498634 -0.474266E+01  0.316E+00    0.60       1.0  T
   3   -959.9884823  0.613810E-01  0.159E+00    0.71       1.0  T
   4   -959.8016077  0.186875E+00  0.751E-01    0.63       1.0  T
   5   -960.2461265 -0.444519E+00  0.191E-01    0.67       1.0  T
   6   -960.2606855 -0.145590E-01  0.101E-01    0.67       1.0  T
   7   -960.2639024 -0.321684E-02  0.640E-02    0.67       1.0  T
   8   -960.2670814 -0.317897E-02  0.287E-02    0.67       1.0  T
   9   -960.2677006 -0.619286E-03  0.796E-03    0.67       1.0  T
  10   -960.2676958  0.483282E-05  0.599E-03    0.67       1.0  T
  11   -960.2677149 -0.190545E-04  0.232E-03    0.67       1.8  T
  12   -960.2677166 -0.177235E-05  0.128E-03    0.67       3.3  T
  13   -960.2677168 -0.143441E-06  0.799E-04    0.67       5.3  T
  14   -960.2677170 -0.178110E-06  0.359E-04    0.67      11.7  T
     SCC iter.                  ...        1 min, 15.246 sec
     gradient                   ...        0 min,  9.483 sec
   1   -955.2950039 -0.955295E+03  0.598E+00    0.84       0.0  T
   2   -960.0359753 -0.474097E+01  0.316E+00    0.62       1.0  T
   3   -959.9602917  0.756836E-01  0.159E+00    0.73       1.0  T
   4   -959.7398188  0.220473E+00  0.795E-01    0.66       1.0  T
   5   -960.2353678 -0.495549E+00  0.195E-01    0.69       1.0  T
   6   -960.2509581 -0.155903E-01  0.101E-01    0.69       1.0  T
   7   -960.2540865 -0.312841E-02  0.653E-02    0.69       1.0  T
   8   -960.2574356 -0.334910E-02  0.286E-02    0.69       1.0  T
   9   -960.2580375 -0.601917E-03  0.815E-03    0.69       1.0  T
  10   -960.2580358  0.170290E-05  0.604E-03    0.69       1.0  T
  11   -960.2580550 -0.192064E-04  0.225E-03    0.69       1.9  T
  12   -960.2580563 -0.130665E-05  0.126E-03    0.69       3.3  T
  13   -960.2580564 -0.934145E-07  0.810E-04    0.69       5.2  T
  14   -960.2580566 -0.198629E-06  0.364E-04    0.69      11.6  T
     SCC iter.                  ...        1 min, 15.334 sec
     gradient                   ...        0 min,  9.410 sec
   1   -955.3017034 -0.955302E+03  0.598E+00    0.83       0.0  T
   2   -960.0433151 -0.474161E+01  0.316E+00    0.61       1.0  T
   3   -959.9751101  0.682051E-01  0.159E+00    0.72       1.0  T
   4   -959.7722861  0.202824E+00  0.772E-01    0.64       1.0  T
   5   -960.2410138 -0.468728E+00  0.193E-01    0.68       1.0  T
   6   -960.2560624 -0.150487E-01  0.101E-01    0.68       1.0  T
   7   -960.2592336 -0.317115E-02  0.647E-02    0.68       1.0  T
   8   -960.2625164 -0.328285E-02  0.286E-02    0.68       1.0  T
   9   -960.2631249 -0.608503E-03  0.801E-03    0.68       1.0  T
  10   -960.2631215  0.338383E-05  0.597E-03    0.68       1.0  T
  11   -960.2631402 -0.187053E-04  0.228E-03    0.68       1.8  T
  12   -960.2631418 -0.151996E-05  0.127E-03    0.68       3.3  T
  13   -960.2631419 -0.127363E-06  0.803E-04    0.68       5.2  T
  14   -960.2631421 -0.190713E-06  0.355E-04    0.68      11.9  T
     SCC iter.                  ...        1 min, 10.393 sec
     gradient                   ...        0 min, 11.980 sec
   1   -955.3038100 -0.955304E+03  0.598E+00    0.83       0.0  T
   2   -960.0450368 -0.474123E+01  0.316E+00    0.61       1.0  T
   3   -959.9757359  0.693009E-01  0.159E+00    0.72       1.0  T
   4   -959.7705083  0.205228E+00  0.775E-01    0.65       1.0  T
   5   -960.2428269 -0.472319E+00  0.193E-01    0.68       1.0  T
   6   -960.2579378 -0.151109E-01  0.101E-01    0.68       1.0  T
   7   -960.2611627 -0.322496E-02  0.653E-02    0.68       1.0  T
   8   -960.2645399 -0.337711E-02  0.284E-02    0.69       1.0  T
   9   -960.2651256 -0.585756E-03  0.794E-03    0.68       1.0  T
  10   -960.2651225  0.306501E-05  0.583E-03    0.68       1.0  T
  11   -960.2651403 -0.177369E-04  0.222E-03    0.68       1.9  T
  12   -960.2651416 -0.128284E-05  0.124E-03    0.68       3.4  T
  13   -960.2651416 -0.752927E-07  0.801E-04    0.68       5.3  T
  14   -960.2651419 -0.214702E-06  0.342E-04    0.68      12.3  T
     SCC iter.                  ...        1 min, 11.030 sec
 dEstep=-0.3042
 Node: 0 Opt step: 3 E: -1.1550 predE: -0.4588 ratio: 0.663 gradrms: 0.00248 ss: 0.025 DMAX: 0.025
 gmax 0.0274 disp 0.2370 Ediff -0.3042 gradrms 0.0025

[awvwgk]

There is a way to workaround the discrepancy between C/Fortran output and Python file handle described in grimme-lab/xtb#160 (comment) for the old Python API, it should work with the new one as well.

[kjelljorner]

I encountered a similar issue in relation to the QCEngine implementation of xtb, where the default is to write output to stdout. Maybe one solution would be to allow different levels of verbosity. Currently, the harness defaults to VERBOSITY_FULL. VERBOSITY_MUTED does seem to do the trick for GFN2-xTB, but for GFN-FF there is still output printed.

xtb-python/xtb/qcschema/harness.py

Line 141 in 43a680b

calc.set_verbosity(VERBOSITY_FULL)

[awvwgk]

For the tblite library I circumvented this problem by allowing the Python API to provide the IO functionality via a callback:

https://github.com/awvwgk/tblite/blob/6396909a4469146e4b2dfacc3dd890ee06223ba9/python/tblite/library.py#L77-L81

This solution will be adopted in xtb as well once we switch to the library as backend for implementing the xTB Hamiltonian. For GFN-FF this would completely resolve the problem that the Calculator can write in the __init__ step before we can actually set the verbosity or redirect the output to a file.

[kjelljorner]

Ok, sounds good!