General Policy Review

[kleros]

General Policy Review

[greenlucid]

0. a. "Author": I suggest "submitter" to keep in line with existing language. Also the actual author and the submitter could always be different people and this could be confusing.

Looks good

b. "MetaList": Should be named "ListMeta" instead, or better still perhaps, "ListMetadata".

I like MetaList because it's like MetaEvidence, whose name was also a mistake. It's minor, will think about it.

c. "Challenge: process in which the inclusion of an Item, in regards to an Edition, is put into question, and it creates a Dispute." If I've understood correctly, I suggest the following instead which should be a bit clearer: "Process by which an edition of an item is disputed. If the challenge succeeds, the Item is removed." (but is it the Item or the Edition?)

Now I'm thinking both statements are confusing. Maybe just "the inclusion of an item is disputed", without mentioning the Editions at all. Since, the policy explains this later anyway.

0'. You don't define when an item is to be considered registered. From what you've said previously I'm guessing this is up to the downstream dapps to decide but this is quite confusing to me.

The item considered registered is never mentioned in the policy because:

• at the contract level, the item is registered at the time of submission. if an item can be challenged, then it's registered.
• it doesn't (shouldn't) affect the policy anyway

MetaList contains a defaultAgeForInclusion field to let dapps know when the item can be considered included after a certain age after commitTimestamp in the item.

0''. Instead of giving an unstructured list of definitions, I think an explanatory paragraph or two (where you can bold the keywords) would be more helpful.

No idea how to do it cleanly

1. "This document will refer to the General Policy as Policy from this point onwards." => "as the Policy" but you could remove this passage and just refer to it as "The General Policy".

originally i just wanted to make the policy less verbose. but this self reference was not used much anyway, so it makes sense to remove this variable name.

2. "This document will refer to the General Policy as Policy from this point onwards. This also refers to all documents this Policy holds authority over." Second sentence is not clear at all. What's "this", and how can a document hold authority over other documents? Maybe you simply meant: "The combination of this document (the General Policy) and the List Policy of the specific list under consideration (as defined in the previous paragraph) will be referred to as the Policy (bold)."

what's this

the General Policy

authority

I thought it was an easy way to convey the point of a heirarchy of policies the arbitrator should consider.

3. Nitpick: You can refactor this: "The terms must, must not, shall, shall not, should, should not, may, required, recommended, not recommended and optional, have their meaning specified in RFC2119.
   To ease understanding, the Policy may only use the following subset of terms: must, must not, should, should not and may." => "The terms must, must not, should, should not and may have their meaning specified in RFC2119. This applies to List Policies as well.".

Fair simplification. I'm not sure though. But putting an optional suggestion for policy building is not appropriate for the General Policy, so it can leave.

3'. You say "the Policy may only use the following subset of terms" but what if I create a List Policy that uses other terms? Surely we don't want to consider it void in that case. Better not to insist on that point IMO.

Well, it's a may so you can skip it. but yeah I can just let it go.

4. "Upon creation of the Dispute, the Challenger must give a reason for removing the Item. It must be a valid reason according to the Policy. If this reason is not valid, the Arbitrator must rule to Keep the Item. This is the case even if the Item does not belong for any other reason, even if implied otherwise on the following clauses."
   a. Upon is vague. Must the full reason be given in the same transaction as the one creating the dispute? Personally, I think it's good to give the challenger some time to formulate their reasoning after challenging. Otherwise, if the reasoning is complex, this could lead to multiple challengers spending time and effort writing their justification in parallel while only the fastest one will get to reap the benefits. I suggest you allow 24 hours to submit a justification.

Challenger has to provide the reason in the method challengeItem. The reason why it's provided immediately, is to avoid frontrunning the edits and figuring out why they're wrong later.

b. "The reason should be clear and exhaustive. The reason must not be ambiguous or open to interpretation." I would add: "The reason must be reasonably specific (not e.g. "The policy does not abide by one of the rules.", or "The item is a duplicate of another item in the list." (without specifying the duplicate item))".

No, I think the exact reason should be provided ("exact" taken in a reasonable meaning). Reason + evidence. I will amend this.
however asking for more from challengers may disincentivize challenges... unless the stake is high. but, maybe this restriction is too large for the general policy.

c. Are parallel challenges possible? I don't see how they could be. Which means one can easily slip an item in by self-challenging with an invalid reason. Or perhaps an item is not considered to be part of the list until it has been unchallenged for a sufficient amount of time, but in that case, the item can be attacked so it never makes the list. I personally dislike the current challenge format as well, from a juror's perspective especially, since, in case no reason is given or the reason given is bad, jurors have to check every single detail of the submission, but there's no simple way around it.

Parallel challenges are not possible. Yes you can self challenge.
The item does not "slip". There is never a period. The item is instantly registered when you call addItem. If the challenge rules to Keep, you can just make another one later.

if no reason is given or wrong, jurors have to check...

No, opposite. arbitrator just rules to keep in that case, no need to read the List Policy.

5. "Required Props" If you mean properties, I'd say "properties" in full. Prop already has several very different meanings in English.

"Property" sounds too generic and I rather use tech or make up words. That removes semantical collisions. "Prop" is shorter than "Property of the Item".

6. "An Item must be removed if it lacks Props whose Column is marked as required." => "if it lacks a Prop/Property"

Fair

7. Why "column" instead of "field"? I'm also confused by the data structures here. You say a Prop is a (label, value) pair. Do you mean (field name, field value) pair then? If so, I'd drop the Prop/Column language altogether. E.g. "Required Fields" "An Item must be removed if it lacks a field marked as required", "Intrusive Fields" (maybe "Unspecified Field" would be a better name) "An item must be removed if it contains a field unspecified in the List Policy" (not sure if that was your intended meaning).

column / field

No real reason. I'm not sure about this one, since field is very generic, probably people would use it for the Props and generate confusing language. Will gather opinions

8. Regarding "Frontrunning, baiting and Editions", I'm sure you've thought about it, but why not have the edition be part of the txdata? In practice, being allowed to push a new edition once the challenge transaction is in the mempool will definitely allow attacks. And relying on flashbots is not acceptable since this service is not guaranteed to remain accessible and otherwise relies on trust assumptions.

Not sure how this works anymore. (context for other participants, we had a chat on telegram before I replied). Maybe the liveliness guarantees of the rollup allow to make this work.

Tldr: the "frontrunning protection" of the general policy didn't take into account that a tx being public does not guarantee it gets into the block.

I guess a terrible way to solve it is, do calls through a contract that will revert if it doesn't occur before the designated block. (bad ux in multiple levels), but it doesn't save editors.

I'm also confused: what happens if multiple editions are published in quick succession? Can only the last one be challenged? What happens to the previous ones? If the last edition is removed, is the item itself removed?

Yes, only the last edition, under normal circumstances, is challenged.
When a new edition is made, the previous one is overwritten. This produces two changes in the smart contract (update commitTimestamp and harddata).
The subgraph keeps all of the history, so everyone can access it.
The contract has no struct for editions. Editions are the basis of the challenge, but not what's removed (that's the item)

8'. TODO after you answer 8..

done

8''. Disregarding previous points: "the previous block in which the Challenge was created" ("in which" somewhat confusing) => "the block immediately preceding the one in which the Challenge was created (henceforth Previous Block and Challenge Block respectively)"

9. Not sure what the point of the "Challenge Cooldown" clause is since the item will still have "disputed" status. Is it just to lift the burden of stakers having to defend themselves too often?

well, it's an optional feature that the subgraph is supporting, so it may warrant be at the general policy.
I don't want to prescribe too much about the lists. the intent of this feature is, after a Keep dispute, give the owner some time to edit the item. imagine a challenge is made right after the item is ruled to Keep.

it was wrong from the beginning but reason was wrong, so Keep. challenger can put the reason now if he gets the next challenge.

[mizu-eth]

2. "This document will refer to the General Policy as Policy from this point onwards. This also refers to all documents this Policy holds authority over." Second sentence is not clear at all. What's "this", and how can a document hold authority over other documents? Maybe you simply meant: "The combination of this document (the General Policy) and the List Policy of the specific list under consideration (as defined in the previous paragraph) will be referred to as the Policy (bold)."

what's this

the General Policy

authority

I thought it was an easy way to convey the point of a heirarchy of policies the arbitrator should consider.

Not clear enough for a policy IMO.

3'. You say "the Policy may only use the following subset of terms" but what if I create a List Policy that uses other terms? Surely we don't want to consider it void in that case. Better not to insist on that point IMO.

Well, it's a may so you can skip it. but yeah I can just let it go.

It's actually worded as "may only" which is a totally different meaning from "may". "you may only do x" means that all alternatives to x are prohibited. So in this case, whether that was your intent or not, you were actually prohibiting the use of the other RFC2119 words.

4. "Upon creation of the Dispute, the Challenger must give a reason for removing the Item. It must be a valid reason according to the Policy. If this reason is not valid, the Arbitrator must rule to Keep the Item. This is the case even if the Item does not belong for any other reason, even if implied otherwise on the following clauses."
   a. Upon is vague. Must the full reason be given in the same transaction as the one creating the dispute? Personally, I think it's good to give the challenger some time to formulate their reasoning after challenging. Otherwise, if the reasoning is complex, this could lead to multiple challengers spending time and effort writing their justification in parallel while only the fastest one will get to reap the benefits. I suggest you allow 24 hours to submit a justification.

Challenger has to provide the reason in the method challengeItem. The reason why it's provided immediately, is to avoid frontrunning the edits and figuring out why they're wrong later.

I don't get the problem here. What frontrunning can happen? Either the "edit" is wrong and the challenger can justify why or it's not and they've just played themselves by "frontrunning" a valid entry. So "frontrunning" blindly is a losing strategy on average (unless the vast majority of submitted entries are invalid). If you just give one day to provide a reason during the evidence period, as long as the reason doesn't change passed one day, it's equivalent, except for the fact that the submitter has 1 day less to submit their own evidence.

b. "The reason should be clear and exhaustive. The reason must not be ambiguous or open to interpretation." I would add: "The reason must be reasonably specific (not e.g. "The policy does not abide by one of the rules.", or "The item is a duplicate of another item in the list." (without specifying the duplicate item))".

No, I think the exact reason should be provided ("exact" taken in a reasonable meaning). Reason + evidence. I will amend this. however asking for more from challengers may disincentivize challenges... unless the stake is high. but, maybe this restriction is too large for the general policy.

I don't get what your objection is. I only ask that "the reason must be reasonably specific".

c. Are parallel challenges possible? I don't see how they could be. Which means one can easily slip an item in by self-challenging with an invalid reason. Or perhaps an item is not considered to be part of the list until it has been unchallenged for a sufficient amount of time, but in that case, the item can be attacked so it never makes the list. I personally dislike the current challenge format as well, from a juror's perspective especially, since, in case no reason is given or the reason given is bad, jurors have to check every single detail of the submission, but there's no simple way around it.

Parallel challenges are not possible. Yes you can self challenge. The item does not "slip". There is never a period. The item is instantly registered when you call addItem. If the challenge rules to Keep, you can just make another one later.

if no reason is given or wrong, jurors have to check...

No, opposite. arbitrator just rules to keep in that case, no need to read the List Policy.

Then, your lists lose a lot of utility. Consider an NFT registry for instance. If I'm an NFT marketplace, I might want to only allow Kleros-certified NFTs to be sold. But if a minter can self-challenge with a bad reason, their NFT will exceed the defaultAgeForInclusion and be considered certified despite having not gone through the proper verification process.

5. "Required Props" If you mean properties, I'd say "properties" in full. Prop already has several very different meanings in English.

"Property" sounds too generic and I rather use tech or make up words. That removes semantical collisions. "Prop" is shorter than "Property of the Item".

Prop already has semantic collisions, but suit yourself.

7. Why "column" instead of "field"? I'm also confused by the data structures here. You say a Prop is a (label, value) pair. Do you mean (field name, field value) pair then? If so, I'd drop the Prop/Column language altogether. E.g. "Required Fields" "An Item must be removed if it lacks a field marked as required", "Intrusive Fields" (maybe "Unspecified Field" would be a better name) "An item must be removed if it contains a field unspecified in the List Policy" (not sure if that was your intended meaning).

column / field

No real reason. I'm not sure about this one, since field is very generic, probably people would use it for the Props and generate confusing language. Will gather opinions

I don't think "generic" names are an issue. Just define them in the definitions if you're afraid they might be misunderstood.

8. Regarding "Frontrunning, baiting and Editions", I'm sure you've thought about it, but why not have the edition be part of the txdata? In practice, being allowed to push a new edition once the challenge transaction is in the mempool will definitely allow attacks. And relying on flashbots is not acceptable since this service is not guaranteed to remain accessible and otherwise relies on trust assumptions.

Not sure how this works anymore. (context for other participants, we had a chat on telegram before I replied). Maybe the liveliness guarantees of the rollup allow to make this work.

Tldr: the "frontrunning protection" of the general policy didn't take into account that a tx being public does not guarantee it gets into the block.

I guess a terrible way to solve it is, do calls through a contract that will revert if it doesn't occur before the designated block. (bad ux in multiple levels), but it doesn't save editors.

I'm also confused: what happens if multiple editions are published in quick succession? Can only the last one be challenged? What happens to the previous ones? If the last edition is removed, is the item itself removed?

Yes, only the last edition, under normal circumstances, is challenged. When a new edition is made, the previous one is overwritten. This produces two changes in the smart contract (update commitTimestamp and harddata). The subgraph keeps all of the history, so everyone can access it. The contract has no struct for editions. Editions are the basis of the challenge, but not what's removed (that's the item)

8'. TODO after you answer 8..

done

So, I looked at the code and I now see that only an item's "owner" can edit it. That makes things a lot simpler indeed. In that case, my suggestion is to drop the block-by-block requirement and replace it with a 1-hour delay (although you might want to make that configurable in the ListMeta/MetaList):

"The challenged edition's IPFS URI must be recorded as part of the initial challenge reason (in the appropriate json field (for instance)) and must have been the active edition at the start of at least one block in the hour preceding the confirmation of the challenge transaction (including the challenge transaction's own block). More precisely, the following must hold true: C ≤ E + 3600 where C is the challenge block's timestamp in seconds and E is the timestamp in seconds of the latest block preceding or at C at the start of which the challenged edition was the active edition."

[greenlucid]

It's actually worded as "may only" which is a totally different meaning from "may".

RFC2119 "may" means optional, and RFC2119 is in place for the General Policy as well

What frontrunning can happen?...

my naive thought on edits was that, when you are editing an item it is likely that there's something wrong with it, on the context of list the version it was launched with, and you want to fix it. if this list version is the latest one, then the item is challengeable.

because (in the version of the General Policy you reviewed) edit + challenge on the same block means you consider the edition available at the previous block, if it was challengeable, when you see the edit coming, you may want to challenge immediately to get the opportunity of finding the reason after the fact, then you can frontrun the challenge and figure out why its wrong later.

you dont want this to happen, because you want item owners to be able to fix their wrong item instead of punishing them for it.

if this wasn't added, then item owners and frontrun challengers have to play an arms race in this dark forest: owners edit their items which are correct both in the previous and new editions, and challengers have to figure out if the edit is a bait / enhancement, or a fix.

i would also be fine for that. but the item owners would have more incentives for baiting if the challenge had to put a stake to be awarded to the owner if "Keep" is ruled. unfortunately, this is not the case in the contract. but, maybe it should be.

I don't get what your objection is. I only ask that "the reason must be reasonably specific".

I misread, sorry.

Then, your lists lose a lot of utility. Consider an NFT registry for instance. If I'm an NFT marketplace, I might want to only allow Kleros-certified NFTs to be sold. But if a minter can self-challenge with a bad reason, their NFT will exceed the defaultAgeForInclusion and be considered certified despite having not gone through the proper verification process.

when I was writing that response I also thought about this edge case, and I wrote an idea to handle it but then removed to not clutter the response.
but, since you brought it up: when editing, challenging the item, or any one of these notable events, you can keep a field for it in the subgraph, distinct from the "commitTimestamp" that is already used for other purposes. something like a "born" timestamp, that is used to figure out canonical inclusion with the defaultAgeForInclusion.

challenged items are not canonically included, either. for lists in which this creates an incentive to wrongly challenge to temporally remove items for griefing, ways to mitigate the damage are allowing duplicates and raising the amounts at stake. again, here it would be interesting to allow lists to force challengers to put skin in the game, because right now they only pay the arbitration fee and item owners get nothing out of it.

So, I looked at the code and I now see that only an item's "owner" can edit it. That makes things a lot simpler indeed. In that case, my suggestion is to drop the block-by-block requirement and replace it with a 1-hour delay (although you might want to make that configurable in the ListMeta/MetaList):

doesn't this allow a challenger to listen to editions taking place, and challenge an item to use a previous wrong edit?

• im a challenger. have notification bot set up to listen to edits
• someone edits a tag in address tags list
• i check the diff...
• seems like the number of characters of one of the props was too large. it was just ammended and its now fine
• i challenge before 1h
• arbitrator will resolve by looking at the previous edition, which was broken
• owner is punished for fixing an item and loses the stake supporting that item

[mizu-eth]

seems like the number of characters of one of the props was too large. it was just ammended and its now fine
owner is punished for fixing an item and loses the stake supporting that item

You're right that's a perverse incentive not to self-correct mistakes, but on the other hand it means there was a mistake in the first place which should have been challenged. So although not optimal, this does not break the registry.

[mizu-eth]

i would also be fine for that. but the item owners would have more incentives for baiting if the challenge had to put a stake to be awarded to the owner if "Keep" is ruled. unfortunately, this is not the case in the contract. but, maybe it should be.

I think pretty much all the other "challengeable" smart contracts have a challenger deposit (in addition to the juror fees that is). It's usually set to 0, but configurable.

challenged items are not canonically included, either. for lists in which this creates an incentive to wrongly challenge to temporally remove items for griefing, ways to mitigate the damage are allowing duplicates and raising the amounts at stake. again, here it would be interesting to allow lists to force challengers to put skin in the game, because right now they only pay the arbitration fee and item owners get nothing out of it.

Raising the amount required to challenge is one mitigation (and only a mitigation) and needs to be supported by the smart contract. Perhaps a better way to go about this would be an ad hoc appeal mechanism of sorts, where the number of jurors is increased *2+1 on each new challenge, either compared to the previous challenge first round, or compared to the last round in the Arbitrator. Both options have nasty edge cases though.
Another approach would be to allow multiple challenges to be proposed in parallel and to ask the jury to choose which reason is the most valid if any, which is how the governor works, but multiple choice votes don't really work well in Kleros v1.

At the end of the day, I feel like you're trying to solve two problems at once (stake curate and the challenge problem) while proposing a half-baked solution (or non-solution) to the second one. Perhaps it would be better to just go with the status-quo regarding challenge mechanics for now. At a minimum, I don't think this should be enforced by the General Policy, and there should at the very least be the option to override this in the List Policies.

[greenlucid]

I think pretty much all the other "challengeable" smart contracts have a challenger deposit (in addition to the juror fees that is). It's usually set to 0, but configurable.

I will add this feature first, then I'll apply all this feedback onto the new policy if you want to review it then. I'll ping you by then.

On the frontrunning mechanism, I thought about two completely different ways to make a good compromise:

• revert challenges on the same block / on the same X min period. the issue is the data structure doesn't support this at the moment but I might be able to find space for it. so now:
  • baiting is possible, but at least the consequence aren't a lost challenge
  • an item owner could continuously spam editions to become unchallengeable, but the inclusion would never realize as the timestamp is kept being reset, so at least the item is not canonically included. also, it wastes gas the item owner spammer.
• keep status quo, with a small period (1-5 minutes) instead of same block. here, the challenger can protect themselves by using a function that reverts after the period, uniswap style, in case the tx is not mined. this utility would be provided by the default frontend.

I'm leaning towards 2nd option because it seems simple (even if it's offloading the complexity to a general policy). The default frontend and injected evidence will certainly help with this so it should not be an issue.

The bad thing about using 2nd option is, (getting dystopian) people can run AIs and figure it out in a short amount of time, and we go back to the dark forest issue. But, tbh this can also happen in a block by block basis to it should just be embraced. It's not perfect but status quo could work

(grief challenging denial of service)

I think a very adamant way to fight it is to allow duplicates in the lists. The attacker would have to spend more and more resources to dunk the items, and the submitter gets more and more funds from the wrongful challenges.

Keeping disputes in parallel would be difficult (the structure cannot afford to have an extra ~32-56 bits to keep a counter of disputes without an extra slot). I don't see how to make it work without keeping track of the counter. This is due to locked amounts, etc.
But, since when you create the item, the dispute counter is zero (unmodified), it would be fine, there could be a way to make this work with parallel disputes. It would delay Stake Curate, though, so I think it's not desirable. Maybe for a future version. I'll think about how serious could not having parallel disputes be. But I don't quite see what the issue is right now

Also, one thing: the challenges, if successful, can only remove the item. A dispute that rules to keep the item does not mean you cannot remove it with a new dispute, just making that clear. The item is always collaterized.