Test results exported to result.json don't map to the correct server/client

[jaikiran]

Test results exported to result.json don't map to the correct server/client

[marten-seemann]

Can you explain where the non-determinism comes from?

[jaikiran]

Hello Marten, _client_server_pairs is an ordered list of client, server tuples. That ordered list loses its order when converted to a set() on these two lines https://github.com/quic-interop/quic-interop-runner/blob/master/interop.py#L287C57-L288. set() is unordered as noted in the documentation https://docs.python.org/3/tutorial/datastructures.html#sets. Those unordered clients, servers are then written out to the "servers" and "clients" JSON arrays in the result.json. Then a few lines later in that _export_results method, we iterate over an ordered list of tuples of clients and servers https://github.com/quic-interop/quic-interop-runner/blob/master/interop.py#L309 to write out the "results" attribute in the json content. As a result, this iteration returns a different order of clients and servers as compared to the clients, servers variables constructed out of the set(). To explain this more clearly, consider this example in the python interpreter:

>>> client_server_pair = []
>>> client_list = []
>>> client_list.append("quic-go")


>>> server_list = []
>>> server_list.append("ngtcp2")
>>> server_list.append("quic-go")
>>> server_list.append("nginx")
>>> server_list.append("neqo")
>>> server_list.append("haproxy")
>>> server_list.append("quiche")


>>> client_list # print the client list
['quic-go']

>>> server_list # print the ordered server list
['ngtcp2', 'quic-go', 'nginx', 'neqo', 'haproxy', 'quiche']



>>> client_server_pairs = []
>>> for client in client_list:
...     for server in server_list:
...             client_server_pairs.append((client, server)) # form a tuple of client, server and add it to a list
... 
>>> client_server_pairs # print the ordered list of tuples
[('quic-go', 'ngtcp2'), ('quic-go', 'quic-go'), ('quic-go', 'nginx'), ('quic-go', 'neqo'), ('quic-go', 'haproxy'), ('quic-go', 'quiche')]


>>> clients = list(set(client for client, _ in client_server_pairs)) # convert to a set() 
>>> clients # print the clients 
['quic-go']


>>> servers = list(set(server for _, server in client_server_pairs)) # convert to a set()
>>> servers # print the servers and notice the original order is lost
['nginx', 'quiche', 'quic-go', 'ngtcp2', 'neqo', 'haproxy']