enable non-verbose logging of relaxation (e.g. `Relax(verbose=0)`)

[materialsvirtuallab]

enable non-verbose logging of relaxation (e.g. `Relax(verbose=0)`)

[shyuep]

The tensorflow warnings can be captured with a catch warnings. But the FIRE: is output from ASE. I am trying to figure out how to control the verbosity, but I can't see an obvious way to do it (beyond supplying a log file, which I don't want to do).

[shyuep]

I actually found a more elegant solution. Using contextlib.redirect_stdout....

[shyuep]

Yeah for the TF, I will leave them for now since it is relatively easy to silence by the end user if you wish. Tensorflow has a lot of deprecations and warnings and sometimes those are useful, especially for the developer end....

[shyuep]

Ok, I just released v0.06 that should have the verbosity control.

[sgbaird]

For the FIRE:, if nothing on the ASE side, maybe filter out lines that begin with FIRE: and then reset the stdout after the function is called? Not the most elegant, but I verified it and it seems to works as expected:

import re


class Filter(object):
    """https://stackoverflow.com/a/63662744/13697228
    
    For reassigning default, https://stackoverflow.com/a/51340381/13697228"""

    def __init__(self, stream, re_pattern):
        self.stream = stream
        self.pattern = (
            re.compile(re_pattern) if isinstance(re_pattern, str) else re_pattern
        )
        self.triggered = False

    def __getattr__(self, attr_name):
        return getattr(self.stream, attr_name)

    def write(self, data):
        if data == "\n" and self.triggered:
            self.triggered = False
        else:
            if self.pattern.search(data) is None:
                self.stream.write(data)
                self.stream.flush()
            else:
                # caught bad pattern
                self.triggered = True

    def flush(self):
        self.stream.flush()

        if self.relax_on_decode:
            sys.stdout = Filter(sys.stdout, r"^FIRE:|     Step     Time          Energy         fmax|\*Force-consistent energies used in optimization.")  # type: ignore # noqa: E501
            relaxer = Relaxer()  # This loads the default pre-trained model

        # relaxation

        if self.relax_on_decode:
            # restore default https://stackoverflow.com/a/51340381/13697228
            sys.stdout = sys.__stdout__

2022-06-16 20:35:23.393379: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found  
2022-06-16 20:35:23.397911: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-06-16 20:35:23.405877: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:4", shape=(None,), dtype=int32), values=Tensor("PartitionedCall:3", shape=(None, 3, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:5", shape=(3,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:1", shape=(3024,), dtype=int32), values=Tensor("Neg:0", shape=(3024, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:2", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/concat_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/concat:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_2_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_5_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_5_grad/Reshape:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_5_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.       

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_6_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_6_grad/Reshape:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_6_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.       

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_3_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_3_grad/Reshape:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_3_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.     

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_4_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_4_grad/Reshape:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_4_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.     

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:1", shape=(None,), dtype=int32), values=Tensor("Neg:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:2", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_1_grad/Reshape:0", shape=(None, 9), dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_1_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.   

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_1_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_1_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.      

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer/concat_atoms/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer/concat_atoms/GatherV2_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer/concat_atoms/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

[sgbaird]

For the tensorflow warnings, I went with:

import tensorflow as tf

...

if not self.verbose:
    tf.get_logger().setLevel(logging.ERROR)

which just leaves:

2022-06-16 20:48:47.442172: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2022-06-16 20:48:47.446997: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-06-16 20:48:47.460638: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

I'm pretty sure this is just during the one call to Relax() (not the repeated calls to relaxer.relax()), and I'm fine with that.

[sgbaird]

@shyuep great! That works for me. The contextlib solution is definitely a lot cleaner for the FIRE lines.