GaussianObservation.probability always returns zero

[lucasmaystre]

GaussianObservation.probability always returns zero

[bpisaacoff]

AHHHHH! It's usually something totally dumb like that isn't it? That resolved it, and is a lesson to all of us to always use keyword arguments. Thank you so much for your help, I really appreciate it.

[lucasmaystre]

Hi @bpisaacoff here is a minimal example

model = ks.DifferenceModel()
kernel = ks.kernel.Constant(var=1.0)

model.add_item("a", kernel=kernel)
model.add_item("b", kernel=kernel)

model.observe(items1=["a"], items2=["b"], diff=3, t=0.0)
converged = model.fit()

model.probabilities(["a"], ["b"], t=1.0)

This returns (0.9047848680872379, 0.0952151319127621) which is not zero.

It is difficult to say without looking at the data that you are using, but perhaps you could try increasing the observation noise:

model = ks.DifferenceModel(var=10.0)

As a rule of thumb I would start with setting var to diff**2, where diff is a typical difference you would see for a match between two teams of equal strength.

Hope this helps!

[bpisaacoff]

Hey @lucasmaystre, thanks for the quick response! Indeed your minimal example works on my machine so there isn't any install problem or something along those lines. So I think it's just an issue with hyperparameters/model choices. To try and elucidate this I tried to get your nba-history notebook working with the Difference Model and am having trouble getting the model to successfully fit. Would you be so kind as to suggest some hyperparameters that might get that to work? What I did is below:

I modified the data setup to the below

teams = set()
observations = list()

cutoff = datetime(2019, 6, 1).timestamp()

with open("nba_elo.csv") as f:
    for row in csv.DictReader(f):
        curdate = datetime.strptime(row["date"], "%Y-%m-%d")
        # I'm on Windows, so can only use timestamps after 1970
        if curdate.year < 1971: continue
        t = datetime.strptime(row["date"], "%Y-%m-%d").timestamp()

        if t > cutoff:
            break
        teams.add(row["team1"])
        teams.add(row["team2"])
        if int(row["score1"]) > int(row["score2"]):
            observations.append({
                "items1": [row["team1"]],
                "items2": [row["team2"]],
                "diff": int(row["score1"]) - int(row["score2"]),
                "t": t,
            })
        else:
            observations.append({
                "items1": [row["team2"]],
                "items2": [row["team1"]],
                "diff": int(row["score2"]) - int(row["score1"]),
                "t": t,
            })

Then the only modification I made from the original notebook was changing the model. With this choices of var = 10 the model won't fit and I get an error in np.linalg that the Matirx is singular to machine precision. If var = 1.0 is used then the fit method won't error out, but rather fails to converge.

# It is a bit more convenient to specify lengthscales in yearly units.
seconds_in_year = 365.25 * 24 * 60 * 60

model = ks.DifferenceModel(var = 10)
kernel = (ks.kernel.Constant(var=0.03)
        + ks.kernel.Matern32(var=0.138, lscale=1.753*seconds_in_year))

for team in teams:
    model.add_item(team, kernel=kernel)

for obs in observations:
    model.observe(**obs)

%%time
converged = model.fit()
if converged:
    print("Model has converged.")

[lucasmaystre]

The exact code in your last message runs when I execute it in a google colab, modifying the example here:
https://colab.research.google.com/github/lucasmaystre/kickscore/blob/master/examples/nba-history.ipynb

Probably something due to your platform?

Note that the NBA dataset is an example where the difference between two teams of similar strength can easily be in the double digits, so setting var=10**2 or even higher might make it more stable.

(EDIT: changed var)

[bpisaacoff]

Ah my bad it didn't converge bc the variance was too low, setting it to 10**2 has gotten the converge flag to return True ... However, it's still returning the probability of (0,1) no matter what. Full notebook here https://colab.research.google.com/drive/1D85Ds_kDWYfH1K5xXURom5vWBcTa27cI?usp=sharing

To see the problem, note that simply just switching the teams doesn't change the probability from (0,1)

print(model.probabilities([row["team2"]],[row["team1"]],t+1))
print(model.probabilities([row["team1"]],[row["team2"]],t+1))

(0.0, 1.0)
(0.0, 1.0)

and just to double check they are in fact different teams

print(row["team1"])
print(row["team2"])

TOR
GSW

[lucasmaystre]

Ah, got it. The issue is that the third argument of model.probabilities() is not t, but some threshold that can be used to rescale the observations, c.f.:
https://github.com/lucasmaystre/kickscore/blob/master/kickscore/observation/gaussian.py#L34

If you change your code to

print(model.probabilities([row["team2"]],[row["team1"]], t=t+1))
print(model.probabilities([row["team1"]],[row["team2"]], t=t+1))

that should do the trick.

[lucasmaystre]

Glad we managed to track it down!