[SPOILER] I FIGURED OUT (at least part of) THE EXPERIMENT GIMMICK [Formerly: What's up with the other branch?]

[carykh]

[SPOILER] I FIGURED OUT (at least part of) THE EXPERIMENT GIMMICK [Formerly: What's up with the other branch?]

[nobody5050]

I wonder if Cary took into account that people are going to use these issues not only as a forum but also that they’re going to try figure out the gimmick

[carykh]

oh my gosh I love how much discussion there is here trying to figure out what my plan is! I don't want to reveal too much haha. And also, yeah I'm aware that people can discover quite a bit through looking through the source code and all the GitHub branches. That's okay with me! I'm hoping that if at least 90%+ of people don't do that, then the statistical trends will still be there...

As for people making their own branches and then discovering the code that came from "main", I didn't realize that would happen... I'm not super familiar with GitHub, so my goal was to make two branches that were completely separate unless you dug deeper, but oh well! The good thing is that the tournament-running script can interpret all three types of input in the same way, so even if you switched branches, it's still compatible.

[astropingo]

hehe I like the idea he is watching our comments and making notes about each one of us.

[astropingo]

The form asks for the word. For me it was broccoli. When I open in incognito, it said earthquake.

The difference in the stories were the item robbed and the gender of the people robbing it. This reminded me that the initial experiments on the prisoners's dilemma when they tried to see if there were a difference in the expectation/outcome when the opponent were gender specific. There is a paper of a biology class where the professor did some real implementation of the problem testing which were the most used strategy (if Tit for Tat or something else), and they found that (on that particular case) the gender didn't matter.

So I believe Cary might be interested in seeing if the story can have an effect on the average strategy chosen? Maybe?

[peterHoburg]

The one big issue I see with this experiment is if I get the police station story, come to the repo and end up on the police station branch, fork the repo, pull it locally, I will end up on main because git defaults.

So, you can have the police station story, the main branch, then when you go back to the website to find the form you are not guaranteed to have the same story + word at the bottom of the page as when you first read the story and wrote your code.

It is 100% possible to read the police station story, write the code on main, and give them gem word on the form.

[nobody5050]

This is an awesome analysis. Looks like it confirms that he’s most likely looking for how differences in the story affect the algorithms

[nobody5050]

hehe I like the idea he is watching our comments and making notes about each one of us.

hehe same

[cai-lw]

A fair amount of people already know prisoner's dilemma and some others like getting their hands dirty rather than reading a wall of text. They will skip the story on the website entirely and the gimmick will not work.

[astropingo]

You would assume that is the incentive. I bet that is what Cary is testing.

Not trying to be the different one here... But to me, the incentive is the 1000 dollar prize 😂. I mean... the fun of it!

[jherndon8]

That's okay with me! I'm hoping that if at least 90%+ of people don't do that, then the statistical trends will still be there...

On the bright side, I definitely looked into trying solutions and submitted before snooping around into what the passcode meant. On the other hand, I think I might have skipped over most of the site, and scrolled straight to the source code when I read “prisoners dilemma” haha

[peterHoburg]

Honestly, I think figuring out the point of the contest is more interesting. Spending a bunch of time coming up with an algorithm only using the standard python lib and Numpy is cool, but I think it would be a lot more fun if you could pull in some ML libs like TensorFlow. I am not nearly smart enough to make a decent ML model from scratch...

[nobody5050]

I actually just assumed gemstone was cary’s staging branch and he’d accidentally linked the wrong one. Even though I’ve read this I’m still using it because I like the branch haha

[CGjupoulton]

What tipped me off to the gimmick wan't the source code, or the branches. I totally accidentally found this zip on the game's website and seeing all the different images got me thinking. The source code confirmed it all.

When I submitted my code, I used the first version I saw to try and stay as biased as possible for you cary :D

[nicholaskross]

GUYS I FIGURED OUT THE EXPERIMENT GIMMICK

It's where the game designer randomly reframes the coop/defect outcomes as gains or losses. The reframing affects how you play; we take more risks to avoid losses and take fewer risks to secure gains.

[peterHoburg]

Now the real question is: Is us finding this the point of the experiment? Also, here is a link to the class this project is for.

CS 269I: Incentives in Computer Science. Hmmmmmmmmmmm
https://timroughgarden.org/f18/f18.html

[nobody5050]

The one big issue I see with this experiment is if I get the police station story, come to the repo and end up on the police station branch, fork the repo, pull it locally, I will end up on main because git defaults.
So, you can have the police station story, the main branch, then when you go back to the website to find the form you are not guaranteed to have the same story + word at the bottom of the page as when you first read the story and wrote your code.
It is 100% possible to read the police station story, write the code on main, and give them gem word on the form.

Exactly, the default mode is not "This is an ARG and I'm in the Matrix of an A/B test", it's "He linked the wrong branch, better use main... wait, the branches have different output types... is this a glitch?"

I wonder if this could almost have been avoided by linking to different repositories, that way at least on the first and longer glances you think no mistake has been made

[nicholaskross]

NEW THEORY: whoever clicks the issues with [SPOILER], automatically wins. Or loses. Or both! Or neither! Or both and neither!

[nicholaskross]

Well see now I'm not sure what to actually submit, except to double-check that I'm on the branch of the page with the word that I clicked in the dictionary of the words that match to the form words in the

[hand-burger]

Seems like you've already figured this much out but ill comment this anyways:

I figured out how it changes, there's is a localStorage variable called choice which is generated when you first go to the website. It can have a value from 0-3, which means there are four different challenges. These different choices are what also change the code word element on the bottom of the page and is why he asks for them. To see each different one you can go into the dev tools and go to storage, open local storage and edit the number which you got and change to be a different one and reload, you'll get a new one!

[EFHIII]

A fair amount of people already know prisoner's dilemma and some others like getting their hands dirty rather than reading a wall of text. They will skip the story on the website entirely and the gimmick will not work.

Speaking personally, that's how it went for me. As soon as I saw the story, I knew it was the prisoner's dilemma, and my strategizing hasn't been impacted in the slightest by what the story said. I honestly can't remember what the story was because it seemed like pointless flavor text.

I'm probably bias towards being that way since I program a lot, and that's what I was focused on, but I imagine that's going to be the case for a lot of people who enter.

I imagine people less proficient in programming won't bother submitting, particularly if they can't beat all the exampleStrats, and I feel like the fact that the random strategy was mentioned in the video will influence a fair number of people into submitting that blindly.

Maybe there will still be a statistically significant trend though.

[OnesAndZer0s]

To add onto this, HEAVY SPOILERS BY THE WAY

Okay for real, spoilers

once going into the html file for the competition website, for the secret word, 
there are four words to choose, being POWDER, BROCCOLI, EARTHQUAKE, 
and EYEBROW, with each story to go along with it. There are two stories, 
a gem swapping story, and a robbing story, with each containing a pair 
of men and a pair of women.

( for context, i opened the website on my phone, then my laptop, 
getting me the different result. It is stored in the cookies. )  

[peterHoburg]

To add onto this, HEAVY SPOILERS BY THE WAY

I didn't even notice this. I just skipped to the repo link and didn't read anything on the page.

[astropingo]

The incentive is not that, though. The incentive is to solve the computational problem, which apparently is the same for everyone regardless the main story they get. That is, to beat the strategies from other players. It might provide data for the author, but I don't know how it would directly interfere with our solutions.

[peterHoburg]

Honestly, I think figuring out the point of the contest is more interesting. Spending a bunch of time coming up with an algorithm only using the standard python lib and Numpy is cool, but I think it would be a lot more fun if you could pull in some ML libs like TensorFlow. I am not nearly smart enough to make a decent ML model from scratch...

I was actually wondering about this earlier and I think you could probably train a ml model using the aforementioned libraries before hand but submit a python file which only runs the pre-trained model. Running a model could “easily” be achieved via numpy

ROFL I was thinking about the same thing. My main issue is how to store the model? You could put it in your .py file and have it be huge. Just have a random pickle string at the bottom of the file that you use. You could also use a model to create a simple equation based off the totals and that could be stored directly as code.

You could also go HAM and pickle a bunch of lib binaries, load them into memory at runtime and use it then... Fun thoughts. I have no idea how to even get started doing that.

[nicholaskross]

(See also, demand characteristics and Any Poll's a Goal.)

[peterHoburg]

Wait until people start doing diffs on the branches...

[nobody5050]

Ok I did some diffs, and without looking at the actual strategies, it looks like there’s some terminology changes in comments/variables. In main most code is referring to the classic prisoners dilemma with a crime needing to be confessed, whereas gemstone seems to be referring to the gemstone story. Interesting.

[nobody5050]

This seems to confirm what we’ve seen on the website so far

[nicholaskross]

And the strategy() output is different depending on which branch you look at.

[nobody5050]

After reading that, I almost wonder if the actual project is actually relating to game theory at all, or rather trying to see what conditions emerge in an open forum setting. Of course this is all pure speculation but I do think Cary wouldn’t accidentally leave issues enabled on this repo.

[peterHoburg]

Also, click the Google Form link on the main comp page and read the questions.

[nobody5050]

One thing I find interesting is that the form link is shared across all of the branches and stories, so it looks like he has no way of knowing which story lead to which strategy. Kinda odd

[peterHoburg]

Now, assuming the different stories is actually the point of this tournament, how do you guarantee (or make it harder) for someone to get more than one story or be aware that there are multiple?

Use one site that you log in to, it gives you a .zip of the code, and take you to the form? You could still make multiple accounts... Hm. Hard problem.

[nobody5050]

I’d assume that the error caused by encountering a different story would be negligible when you have an audience the size of Cary’s. It looks like he’s using local browser storage currently. One thing that could skew this potentially is that mobile browsers clear the local storage after 7 days by default.

[peterHoburg]

I’d assume that the error caused by encountering a different story would be negligible when you have an audience the size of Cary’s. It looks like he’s using local browser storage currently. One thing that could skew this potentially is that mobile browsers clear the local storage after 7 days by default.

I was just messing with that. It looks like it only goes to 3.

[nobody5050]

The one big issue I see with this experiment is if I get the police station story, come to the repo and end up on the police station branch, fork the repo, pull it locally, I will end up on main because git defaults.

oh also it looks like he’s dealt with this by not including the google form on the main branch, thus forcing you to use the branch you got sent by the website’s README to access the form

[nobody5050]

I’d assume that the error caused by encountering a different story would be negligible when you have an audience the size of Cary’s. It looks like he’s using local browser storage currently. One thing that could skew this potentially is that mobile browsers clear the local storage after 7 days by default.

I was just messing with that. It looks like it only goes to 3.

Huh, neat

[peterHoburg]

The one big issue I see with this experiment is if I get the police station story, come to the repo and end up on the police station branch, fork the repo, pull it locally, I will end up on main because git defaults.

oh also it looks like he’s dealt with this by not including the google form on the main branch, thus forcing you to use the branch you got sent by the website’s README to access the form

You still end up with the same issue. I wrote my code on main, went back to the site to get the form as it was not on main, and clicked through to the form there. Also, the form is assuming you navigated to the form via the webpage. Hence, the words at the bottom.

[peterHoburg]

The incentive is not that, though. The incentive is to solve the computational problem, which apparently is the same for everyone regardless the main story they get. That is, to beat the strategies from other players. It might provide data for the author, but I don't know how it would directly interfere with our solutions.

You would assume that is the incentive. I bet that is what Cary is testing.

[nobody5050]

This is a really interesting comment chain, there’s a few similar but competing theories for what Cary is actually testing.

[nobody5050]

Honestly, I think figuring out the point of the contest is more interesting. Spending a bunch of time coming up with an algorithm only using the standard python lib and Numpy is cool, but I think it would be a lot more fun if you could pull in some ML libs like TensorFlow. I am not nearly smart enough to make a decent ML model from scratch...

I was actually wondering about this earlier and I think you could probably train a ml model using the aforementioned libraries before hand but submit a python file which only runs the pre-trained model. Running a model could “easily” be achieved via numpy

[nicholaskross]

The one big issue I see with this experiment is if I get the police station story, come to the repo and end up on the police station branch, fork the repo, pull it locally, I will end up on main because git defaults.

So, you can have the police station story, the main branch, then when you go back to the website to find the form you are not guaranteed to have the same story + word at the bottom of the page as when you first read the story and wrote your code.

It is 100% possible to read the police station story, write the code on main, and give them gem word on the form.

Exactly, the default mode is not "This is an ARG and I'm in the Matrix of an A/B test", it's "He linked the wrong branch, better use main... wait, the branches have different output types... is this a glitch?"

[astropingo]

Honestly, I think figuring out the point of the contest is more interesting. Spending a bunch of time coming up with an algorithm only using the standard python lib and Numpy is cool, but I think it would be a lot more fun if you could pull in some ML libs like TensorFlow. I am not nearly smart enough to make a decent ML model from scratch...

I was actually wondering about this earlier and I think you could probably train a ml model using the aforementioned libraries before hand but submit a python file which only runs the pre-trained model. Running a model could “easily” be achieved via numpy

We are not going to run against the libraries. At least not just against them. We are going to run against other users submissions too. Hard to train with what we don't know what to expect :)

[nobody5050]

Honestly, I think figuring out the point of the contest is more interesting. Spending a bunch of time coming up with an algorithm only using the standard python lib and Numpy is cool, but I think it would be a lot more fun if you could pull in some ML libs like TensorFlow. I am not nearly smart enough to make a decent ML model from scratch...

I was actually wondering about this earlier and I think you could probably train a ml model using the aforementioned libraries before hand but submit a python file which only runs the pre-trained model. Running a model could “easily” be achieved via numpy

ROFL I was thinking about the same thing. My main issue is how to store the model? You could put it in your .py file and have it be huge. Just have a random pickle string at the bottom of the file that you use. You could also use a model to create a simple equation based off the totals and that could be stored directly as code.

For including in the py file, you could use tensorflow lite which will give a smaller output at cost of accuracy. Then again there doesn’t seem to be a filesize limit on the form...

[astropingo]

But the rules specifically said to use the standard random and numpy libraries and not import anything else. Unless you want to write all the functions on the file, but that is just importing with extra steps.

[peterHoburg]

Honestly, I think figuring out the point of the contest is more interesting. Spending a bunch of time coming up with an algorithm only using the standard python lib and Numpy is cool, but I think it would be a lot more fun if you could pull in some ML libs like TensorFlow. I am not nearly smart enough to make a decent ML model from scratch...

I was actually wondering about this earlier and I think you could probably train a ml model using the aforementioned libraries before hand but submit a python file which only runs the pre-trained model. Running a model could “easily” be achieved via numpy

But we are not going to run against the libraries. At least not just against them. We are going to run against other users submissions too. Hard to train with what we don't know what to expect :)

I would probably go with start the training vs the existing ones and maybe a couple more I write. Then start doing generational training vs it's self.

[nobody5050]

Honestly, I think figuring out the point of the contest is more interesting. Spending a bunch of time coming up with an algorithm only using the standard python lib and Numpy is cool, but I think it would be a lot more fun if you could pull in some ML libs like TensorFlow. I am not nearly smart enough to make a decent ML model from scratch...

I was actually wondering about this earlier and I think you could probably train a ml model using the aforementioned libraries before hand but submit a python file which only runs the pre-trained model. Running a model could “easily” be achieved via numpy

But we are not going to run against the libraries. At least not just against them. We are going to run against other users submissions too. Hard to train with what we don't know what to expect :)

I would probably go with start the training vs the existing ones and maybe a couple more I write. Then start doing generational training vs it's self.

Same. I’ve already devised one model which is slightly better than TFT but it’s nowhere near good enough to win

[duckboycool]

In terms of the different versions you're given, gender and story are independent, and are the only two variables.

You're given a variable choice from 0 to 3. 0 and 1 are police station, and 2 and 3 are gemstone. 0 and 2 are women, and 1 and 3 are men. The possible words are POWDER, BROCCOLI, EARTHQUAKE, and EYEBROW, where they're picked in that order from choice. So POWDER is women and police station for example.

Also I believe all links except the github branch are the same for all versions.

[seba-eng]

Very interesting, I never thought this was more about psychology than about computer science. That also explains why the strategy functions on the gemstone and police station branches return strings and not just zero or one.

[hand-burger]

One thing I find interesting is that the form link is shared across all of the branches and stories, so it looks like he has no way of knowing which story lead to which strategy. Kinda odd

Theres the code words, which he asks for in the google form to identify the different challenges

[SunCatMC]

I managed to see 2 versions of the story 'cos of first using a tablet, and then going to laptop to figure out the actual code

and only after reading this issue thread I realized what it was all about xD

[Silivek]

The differing stories never actually affected my code, but when I was trying to explain to people what I was trying to do I was showing them the website on my phone, which was different from my computer's website and I thought I was going crazy at first. It was very odd figuring out that I wasn't the crazy one