Feature request: "which" attribute

[fixtradingcommunity]

Feature request: "which" attribute

[donmendelson]

@YuvalCohen +1

The only amendment is that required is no longer a boolean but rather an enumeration of presence = required/optional (default)/forbidden/ignored/constant. Conditionally required is expressed as presence=optional plus a rule to override it to required. Agree that a required member of a oneOf group doesn't make sense. However, a rule for optionally required could be allowed, based on the value of another field, perhaps security type in the example.

[donmendelson]

Proposed usage in repeating groups: if OrderQtyData has which="oneOf", meaning exactly one of its members must be present, and the component is contained by a repeating group like ListOrdGrp, then one member must be present in each instance of the group, but not necessarily the same member in each instance. It might be OrderQty in one instance and CashOrderQty in another since theoretically, the instrument types could be different. In general, FIX does not require instances of a repeating group to be homogeneous.

[kleihan]

@donmendelson agree, I would add one bit to your description above: "...and the component is contained by a repeating group like ListOrdGrp which requires OrderQtyData to be present, then...". I believe FIX has very few cases such as OrderQtyData.

Even though FIX does not require instances to be homogeneous, high performance implementations using fixed length encodings (e.g. SBE) will have an issue to be FIX compliant for OrderQtyData. Compliance means that you have to define 3 qty fields into your schema and space for two of them is wasted in every message, or worse, in every instance of ListOrdGrp. We just need to be aware of that. Then again, high performance implementations will probably focus on a single asset class, either equities or fixed income or CIV. We should still offer the proposed feature.

[donmendelson]

I believe FIX has very few cases such as OrderQtyData.

@kleihan, that's true, but if we offer this feature it could become more common. @YuvalCohen's motivation for this feature is JSON Schema's equivalent, which is commonly used.

[donmendelson]

@YuvalCohen, this idea was brought up once before but was not accepted in the Orchestra standard, as there was no precedent for it in FIX Repository. Could you please provide a business use case that requires it?

There is a mechanism in Orchestra now to make a field or component conditionally required based on an expression, e.g. StopPx is required if OrdType is Stop or StopLimit.

[YuvalCohen]

The FIX repository is not used only for FIX messages. There are other businesses (as you may know) that uses the FIX repository. This feature is required for such businesses.
Without it, I'm afraid, this 2016 repository will remain only for the purposes of FIX messages.

Whilst it is possible (and easy to have: 'allOf' (simply by placing required)
'anyOf' and 'oneOf' are not so simple to define without the explicit keywords.

Think about FpML structures, these have few occurrences and examples of such.

Whilst I cannot provide an explicit example (not available for public yet), I hope to have them later this year. I'll be happy to provide few drafts privately.

I hope it helps.

[donmendelson]

Not disagreeing with your request, but I want to point out that there is one more attribute in Orchestra that may help. The "rendering" attribute is free-form and may be used as a hint to translators for any message format. It was originally conceived to help mapping to web API protocols.

[YuvalCohen]

Thanks, the rendering is not what I am after.

Yet, speaking about translators, it will be nice to have sample translators e.g. the ISIN-mate translator (e.g. a translator from FIX repository to ISIN-mate html files).

[jimnup]

@YuvalCohen do you think the use of the Which entity would have applicability beyond APIs to where it would be usable for other protocols and FIX protocols? We think there might be cases in FIX messages where we say either this field or that field, but can't recall any specific ones.

[YuvalCohen]

@jimnup
In FIX, the obvious cases is when users want to specify either the yield or the price... (this may be as either 'oneOf' or 'anyOf' - to indicate that both are acceptable ).

Another case for 'allOf' is the Parties structure, all attributes in the parties structure have a comment: Required if NoPartyIDs(453) > 0.
Is there a way to code it? Yes there is! the 'which' attribute should be equal to 'allOf' in that component.

In the Instrument structure there is the case where users may want to sent either SecurityID or SecurityXML (or both).

There are many cases for this feature both in FIX as well as in other protocols.

These rules are currently written in comments in the FIX protocol and later agreed (/nailed) on the rules of engagements (again non-machine readable format). Hopefully Orchestra resolve such out-of-band agreements and convert them to machine readable.

[donmendelson]

Orchestra already has a mechanism to express a rule for a conditionally required field based on other values in a message. Also, it has scenarios to express different use cases, for instance, different fields used by asset class.

@YuvalCohen, in your example about SecurityID/SecurityXML, are you saying that a firm would accept either in an incoming message? If so, I agree with the need. (That's different than a firm deciding which of many alternatives from the FIX standard to use as their rules of engagement.)

[kleihan]

'allOf' does not apply to the Parties component in general. PartyRoleQualifier(2376) and PtysSubGrp are optional. However, if only PartyID, PartyIDSource, PartyRole are used in a given implementation, then it would apply. My view is that the "Req'd" setting of a field in a (repeating) group should only apply to that group. The semantic is then: "for any given instance of Parties, this field is required". Today we only have a message scope for nested fields which is insufficient. The Parties component as a whole has its own "Req'd" setting which applies to the message.

I do agree that we have cases: CommissionData and CommissionDataGrp (The CommissionDataGrp component block is used to carry commission information such as the type of commission and the rate. It provides an alternative to the CommissionData component if multiple commissions or enhanced attributes are needed.)

[kleihan]

Further example: OrderQtyData, see field usage comment of OrderQty(38) is a case for "oneOf":

One of CashOrderQty, OrderQty, or (for CIV only) OrderPercent is required. Note that unless otherwise specified, only one of CashOrderQty, OrderQty, or OrderPercent should be specified.

[donmendelson]

• Wouldn't "allOf" a group of fields be the same thing as making them all required? I understand the need for "oneOf" since it implies mutually exclusive choice as in the OrderQtyData example.
• I can't think of an example, but I suppose we need to allow for more than one such group in a message or component.

Syntax thoughts:
If a rule belongs to containing message or component, it must identify members of the group. Members as <fieldRef> or <componentRef> are identified by id attribute (tag), but how do you know which type it is? Theoretically, there could be collision between field and component IDs.

<message>
<fieldRef id='1' presence='required'/>
<rule which='oneOf' members='38, 152, 516'/>
<fieldRef id='38'/>
<fieldRef id='152'/>
<fieldRef id='516'/>
</message>

An alternative would be to make the rule a container of its members, something like ...

<message>
<fieldRef id='1' presence='required'/>
<rule which='oneOf'>
<fieldRef id='38'/>
<fieldRef id='152'/>
<fieldRef id='516'/>
</rule>
</message>

Advantages: supports more than one "oneOf" group. No ambiguity about IDs and IDs are not duplicated. Could even theoretically support mutually exclusive field / component references.
Disadvantage: members of the grouping are in a different XML nesting than other fieldRefs in the message.

[kleihan]

We will need Yuval to clarify. I have understood the requirement to be only for a component. "allOf" a group of fields is the same thing as making them all required IF we change the semantic of "Req'd" for fields inside a component. We need Jim&Lisa to weigh in on such a decision. I would prefer to get rid of all the comments saying "Conditionally required if NoXXX>0" by setting the field to required. It should not matter if the component is repeating or not. Non-repeating means that it is a single instance and you have to use its required fields if you use any of its fields. I believe CommissionData is underspecified in the sense that Commission(12) does not make sense withotu CommType(13). They would both be set to required in the new world.

My example with the commission data is something different. 'oneOf' does not fit that example as both components are optional. However, IF one of them is used, the other one should not be used. The term to use for the rule would rather be "mutuallyExclusive" and both components would have such a rule. The same rule can apply on the valid values level, e.g. ExecInst(18) values H,Q,l and J,K,m and n,o,p.

Now to the oneOf syntax. I assume that "fieldRef id='1' presence='required'" refers to the component OrderQtyData. Why is it a fieldRef and not a componentRef? I agree that member IDs are not unique, hence you need the second syntax.

I struggle with Yuval's "anyOf". It seems to reflect the list of optional elements of a message or component. Walking through the presence settings of all elements seems equivalent.

[YuvalCohen]

@kleihan @donmendelson
I agree that allOf may be redundant if and once we get rid of the "Conditionally required if NoXXX>0" and let the required flag meaning that an entity is required within the component (or group) (And set the first field of each repeating group as required)

Per the Parties, I would like to clarify that in practice, many institutions that uses FIX, have there own 'repository' which is a subset of the FIX repository (and in many times with some amendments like user define fields and components etc.). Ideally, using Orchestra, they are able to "well define" or at least better define the syntax rules. Hence many of these would simply remove the PartyRoleQualifier(2376) from Parties (as it is not in use in most of their business cases)... And yet I am happy that you found a fitting example in FIX.

The feature (oneOf and anyOf) should apply to either a component or a group (technically possible for structure and message).
The members can be any references in them: fieldref, componentref, groupref. And a mix of them. So an example would be

oneOf
{
<fieldRef id="44" /> <!-- YC: Price -->
<componentRef id="NNN1" /> <!-- YC: YieldData component-->
<componentRef id="NNN2" /> <!-- YC: SpreadOrBenchmarkCurveData component-->
}

Note: the above example is not about the syntax (how to define the oneOf) but it is about the requirements.

The anyOf is where at least one of these must exist...
FYI: for my business case, we need only the oneOf. Yet I think that there may be use cases where the anyOf is useful.

Last, per the syntax, I like the 'which' to be part of the component. i.e.:

<component id="NNN0" name="Price Definition" which="oneOf"> 
<fieldRef id="44" /> <!-- YC: Price -->
<componentRef id="NNN1" /> <!-- YC: YieldData component-->
<componentRef id="NNN2" /> <!-- YC: SpreadOrBenchmarkCurveData component-->
</component> <!-- Price Definition -->

Although this is not important... So you can be defined with .

[kleihan]

@YuvalCohen please check whether I understand it properly:
oneOf = EXACTLY one from a list (FIX semantic: mutually exclusive elements, one is required)

Example for oneOf is OrderQtyData but only if the component is required (NewOrderSingle). It is optional in an IOI message. So should oneOf rather be "AT MOST ONE", depending on the Req'd setting of the component? You show the syntax without message context which is much simpler, i.e. the semantic would be "if component X is used then rule oneOf applies". The "if" then depends on the message context. Correct?

anyOf = one or more (but not zero) from a list (FIX semantic: equivalent elements, at least one is required)

Example for anyOf is the TCR (TradeReportID or TradeID or both). You have to decide between message chaining and entity based but can also fill both fields.

[donmendelson]

Some models we can learn from:

XML Schema expresses oneOf with <choice>. We could follow this model with the outer element being a message, group or component, and the choice elements being any mixture of fieldRef, componentRef, groupRef.

Example: an element named "person" which must contain either a "employee" element or a "member" element. From https://www.w3schools.com/xml/el_choice.asp

<xs:element name="person">
  <xs:complexType>
    <xs:choice>
      <xs:element name="employee" type="employee"/>
      <xs:element name="member" type="member"/>
    </xs:choice>
  </xs:complexType>
</xs:element>

Google Protocol Buffers has a oneof keyword. It acts as a discriminated union; it prohibits anyOf semantics.

Example from https://developers.google.com/protocol-buffers/docs/proto#oneof

message SampleMessage {
  oneof test_oneof {
     string name = 4;
     SubMessage sub_message = 9;
  }
}

[YuvalCohen]

@kleihan

Excellent examples!

Per the FIX OrderQtyData example one would expect:

<component name="OrderQtyData" which="oneOf" >
        <fieldRef name="OrderQty" />
        <fieldRef name="CashOrderQty"/>
        <componentRef name="CIVOrderQtyData"/>
</component > <!-- OrderQtyData /  -->

clearly the CIVOrderQtyData needs  to be defined properly as:

<component name="CIVOrderQtyData"/>
        <fieldRef name="OrderPercent" required="true" />
        <fieldRef name="RoundingDirection" required="false"/>  
        <fieldRef name="RoundingModulus" required="false"/>  
</component> <!-- name="CIVOrderQtyData"/ -->


Note that in IOI there should be:
<componentRef name="OrderQtyData" required="false"/>

vs. New Order Single that should have:
<componentRef name="OrderQtyData" required="true"/>

Note that fieldRef and componentRef within a component that are has which="oneOf" must not have required attribute.

I hope it helps.

[donmendelson]

In ISO 20022, this is called a ChoiceComponent. They are quite common.

Example: PartyIdentificationInformation is a choice of OrganizationIdentification or PersonIdentification.

[kleihan]

@donmendelson There is a reason why it is very uncommon in FIX. The FIX messages are the business model (i.e. no separate business model so far) and we would model this in FIX (if we did not yet have a Parties component which we do) with a single party identifier and a second field telling you if the first field is for an organization or a person. But we would not add two identifier fields and a "oneOf" rule to ensure that only one of the two fields is in a given message.