Occasional unit test failure of `test_classifier_with_circuit_qnn_and_cross_entropy`

[qiskit-community]

Occasional unit test failure of `test_classifier_with_circuit_qnn_and_cross_entropy`

[manoelmarques]

I created a simpler sample to recreate the error. It seems to only happen for python 3.7 and it seems to be caused by Aer.
I couldn't recreate the error with BasicAer.
I could't recreate the error with python 3.8 either.
@woodsp-ibm couldn't recreate the error on Windows with python 3.7.

For python 3.7 It failed on Ubuntu after 14 and 116 iterations and on MacOS after 161 iterations.

import numpy as np
from qiskit import Aer, QuantumCircuit, BasicAer
from qiskit.algorithms.optimizers import COBYLA
from qiskit.circuit.library import RealAmplitudes, ZZFeatureMap
from qiskit.utils import QuantumInstance, algorithm_globals

from qiskit_machine_learning.algorithms.classifiers import NeuralNetworkClassifier
from qiskit_machine_learning.neural_networks import CircuitQNN
from qiskit_machine_learning.utils.loss_functions import CrossEntropyLoss

def _classifier_with_circuit_qnn_and_cross_entropy():
    count = 0
    while True:
        algorithm_globals.random_seed = 12345
        quantum_instance = QuantumInstance(
            # BasicAer.get_backend("qasm_simulator"),
            Aer.get_backend("aer_simulator"),
            shots=100,
            seed_simulator=algorithm_globals.random_seed,
            seed_transpiler=algorithm_globals.random_seed,
        )

        optimizer = COBYLA(maxiter=25)

        loss = CrossEntropyLoss()

        num_inputs = 2
        feature_map = ZZFeatureMap(num_inputs)
        ansatz = RealAmplitudes(num_inputs, reps=1)

        # construct circuit
        qc = QuantumCircuit(num_inputs)
        qc.append(feature_map, range(2))
        qc.append(ansatz, range(2))

        # construct qnn
        def parity(x):
            return "{:b}".format(x).count("1") % 2

        output_shape = 2
        qnn = CircuitQNN(
            qc,
            input_params=feature_map.parameters,
            weight_params=ansatz.parameters,
            sparse=False,
            interpret=parity,
            output_shape=output_shape,
            quantum_instance=quantum_instance,
        )
        # classification may fail sometimes, so let's fix initial point
        initial_point = np.array([0.5] * ansatz.num_parameters)
        # construct classifier - note: CrossEntropy requires eval_probabilities=True!
        classifier = NeuralNetworkClassifier(
            qnn,
            optimizer=optimizer,
            loss=loss,
            one_hot=True,
            initial_point=initial_point,
            # callback=None,
        )

        # construct data
        num_samples = 5
        x = algorithm_globals.random.random(
            (num_samples, num_inputs)
        )
        y = 1.0 * (np.sum(x, axis=1) <= 1)
        y = np.array([y, 1 - y]).transpose()

        # fit to data
        classifier.fit(x, y)

        # score
        score = classifier.score(x, y)
        count += 1
        print(f"\n{score} {initial_point} {x} {count}")
        if score > 0.5:
            print("Success")
        else:
            print("Failed")
            return


if __name__ == "__main__":
    _classifier_with_circuit_qnn_and_cross_entropy()

[manoelmarques]

Also, this unit test method in the same file failed too under python 3.7 or python 3.8:

test.algorithms.classifiers.test_neural_network_classifier.TestNeuralNetworkClassifier.test_classifier_with_opflow_qnn_19___bfgs____squared_error____statevector___True_ [5.494827s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/ddt.py", line 182, in wrapper
    return func(self, *args, **kwargs)

      File "/home/runner/work/qiskit-machine-learning/qiskit-machine-learning/test/algorithms/classifiers/test_neural_network_classifier.py", line 127, in test_classifier_with_opflow_qnn
    self.assertGreater(score, 0.5)

      File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/unittest/case.py", line 1251, in assertGreater
    self.fail(self._formatMessage(msg, standardMsg))

      File "/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/unittest/case.py", line 693, in fail
    raise self.failureException(msg)

    AssertionError: 0.4 not greater than 0

I created a sample script for the error and was able to reproduce in Ubuntu python 3.7 and MacOS python 3.7/3.8. Unfortunately for this method, I was able to recreate the error with Aer and BasicAer.

import numpy as np
from qiskit import Aer, BasicAer
from qiskit.algorithms.optimizers import L_BFGS_B
from qiskit.circuit.library import RealAmplitudes
from qiskit.utils import QuantumInstance, algorithm_globals

from qiskit_machine_learning.algorithms.classifiers import NeuralNetworkClassifier
from qiskit_machine_learning.neural_networks import TwoLayerQNN, CircuitQNN
from qiskit_machine_learning.utils.loss_functions import CrossEntropyLoss


def _classifier_with_opflow_qnn():
    loss = "squared_error"

    count = 0
    while True:
        algorithm_globals.random_seed = 12345
        quantum_instance = QuantumInstance(
            # BasicAer.get_backend("statevector_simulator"),
            Aer.get_backend("aer_simulator_statevector"),
            seed_simulator=algorithm_globals.random_seed,
            seed_transpiler=algorithm_globals.random_seed,
        )

        optimizer = L_BFGS_B(maxiter=5)

        history = {"weights": [], "values": []}

        def callback(objective_weights, objective_value):
            history["weights"].append(objective_weights)
            history["values"].append(objective_value)

        num_inputs = 2
        ansatz = RealAmplitudes(num_inputs, reps=1)
        qnn = TwoLayerQNN(num_inputs, ansatz=ansatz, quantum_instance=quantum_instance)
        initial_point = np.array([0.5] * ansatz.num_parameters)

        classifier = NeuralNetworkClassifier(
            qnn, optimizer=optimizer, loss=loss, initial_point=initial_point, callback=callback
        )

        # construct data
        num_samples = 5
        x = algorithm_globals.random.random(
            (num_samples, num_inputs)
        )
        y = 2.0 * (np.sum(x, axis=1) <= 1) - 1.0

        # fit to data
        classifier.fit(x, y)

        # score
        score = classifier.score(x, y)
        count += 1
        print(f"\n{score} {initial_point} {x} {count}")
        if score > 0.5:
            print("Success")
        else:
            print("Failed")
            return


if __name__ == "__main__":
    _classifier_with_opflow_qnn()

[adekusar-drl]

I have not seen this issue for a long time. If I recall correctly randomization was fixed in the unit tests and perhaps some randomization was also fixed in simulators.
@woodsp-ibm should we close it for now?

[woodsp-ibm]

As @manoelmarques sees info from nightlies etc maybe he has some thought. On my part if its not been seen in a while I would say close it it - if it starts again this or another new issue can always be (re-)opened etc.

[manoelmarques]

@adekusar-drl @woodsp-ibm I haven't seen any random error in the nightly builds in months. Also, I ran both samples above locally in my MacOS and they didn't fail even after more than 600 iterations while they would fail fast before.
We can close this issue. The problem seems to have been fixed.

[adekusar-drl]

@manoelmarques Thanks for looking into this. Closing the issue.