fixtradingcommunity/fix-orchestra

Machine readable rules of engagement

FIX Orchestra Resources

This project contains resources and sample code for FIX Orchestra version 1.1. Technical specifications for FIX Orchestra are in project fix-orchestra-spec.

FIX Orchestra is intended to provide a standard and some reference implementation for machine readable rules of engagement between counterparties. The goal is to reduce the time to get counterparties trading, and improve accuracy of implementations.

Tutorials

See Orchestra tutorials and FAQ.

References

Orchestra specifications specifications for Orchestra in GitHub.

Orchestrations public Orchestra files for service offerings

FIX Standards normative standards documents

Standards Versions

Version 1.0 Technical Standard

Orchestra version 1.0 Draft Standard was promoted to Technical Standard with minor enhancements and corrections on February 17, 2021. Version 1.6.10 of this project conforms to Orchestra version 1.0 Techncial Standard.

Normative Modules

The following modules are normative.

repository

This module contains an XML schema for Orchestra. It is used to convey message structures and their components, as well as FIX application behaviors. Users may express workflow as responses to messages under different scenarios, as well as external state information that may influence behaviors.

In addition to providing the XML schema as a resource, the module builds Java bindings for the schema.

interfaces

This module provides an XML schema for service offerings, protocols and session provisioning. In addition to providing the XML schema as a resource, the module builds Java bindings for the schema. Service elements may link to Orchestra files composed in the repository schema.

dsl-antlr - Score Domain Specific Language (DSL)

An orchestra file may contain conditional expressions to describe conditionally required fields and tell when a certain response to a message applies. Also, the DSL may be used for assignment expressions to set message fields and external state variables.

The Score grammar is provided in the notation of ANTLR4, and the project builds a lexer/parser for the grammar. This project generates Java code, but ANTLR4 is capable of generating several other programming languages from the same grammar.

Informational Modules and Utilities

repository2010

Repository 2010 Edition was the version of the FIX Repository prior to FIX Orchestra. This module provides a parser for its XML schema. It may be used to process existing Repository files and to convert their message structures to Orchestra format.

orchestra2doc

This utility generates documentation for an Orchestra file that can be view in any web browser. The output of the generator may be used locally or from a web server.

repository-util

This module contains RepositoryValidator to validate that a file conforms to the repository schema. Also, RepositoryCompressor extracts slices of a repository file. An XSLT is provided to translate existing Repository 2010 Edition files to the Orchestra schema.

interfaces-util

This module contains InterfacesValidator to validate that a file conforms to the interfaces schema.

Experimental Modules

Experimental modules have been moved to GitHub repository fix-orchestra-experimental

This following modules are experimental. Requirements are still being gathered and discussed by the FIX Orchestra working group. Participation in the working group is encouraged by FIX Trading Community members, and more broadly, feedback is welcome from interested GitHub users.

state-machine

A demonstration of state machine code generation from an Orchestra file.

testgen

This module is a demonstration of acceptance test generation from an Orchestra file using Behavior Driven Design (BDD) concepts.

Code Generation for Encoding

orchestra2sbe

This utility has been moved to repository FIXTradingCommunity/fix-sbe-utilities. It creates a Simple Binary Encoding message schema from an Orchestra file.

fix-orchestra-protobuf

See repository FIXTradingCommunity/fix-orchestra-protobuf for utilities to generate Google Protocol Buffers schemas from an Orchestra file.

FIX Engine Provisioning

Modules specific to QuickFIX have been moved to repository fix-orchestra-quickfix. Some models in this project are intended to be operational while others are proofs of concept.

repository-quickfix

This module generates a QuickFIX data dictionary from an Orchestra file. The format can be consumed by the C++, Java and .NET versions. Additionally, the module generates message classes for QuickFIX/J directly from an Orchestra file. Although the QuickFIX data dictionary format is not as richly featured as Orchestra, it is hoped that this utility will help with Orchestra adoption.

model-quickfix

This module generates code that is conformant to the QuickFIX/J API for validating and populating messages. It is dependent on repository-quickfix.

session-quickfix

A demonstration of session configuration for QuickFIX open-source FIX engine. It consumes an XML file in the interfaces schema.

License

© Copyright 2016-2022 FIX Protocol Limited

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Technical Specifications License

Note that the related Technical Specifications project has a different license than these resources. See fix-orchestra-spec

Data Files

Data files in this project under test/resources are strictly for testing and to serve as examples for format. They are non-normative for FIX standards and may not be up to date.

Build

The project is built with Maven version 3.3 or later.

This project requires Java 11 or later. It should run on any platform for which a JVM is supported. Some modules use Java Platform Module System (JPMS).

Several open-source JDK implementations are available, including Eclipse Temurin and Azul Zulu.

fix-orchestra's Issues

Standardize names of FIX protocols in FixInterfaces.xsd

The interfaces XML schema has methods to specify a protocol stack. The names of FIX protocols such as Tag Value encoding should enumerated to standardize them. However, other protocols should also be accepted. See datatypeStandard_enum in repositorytypes2016.xsd for an example of an open-ended enumeration.

How to generate orchestra file from fix dictionary and fix session config file

I am trying to create orchestra file for a project which has currently fix dictionary and fix session config file. Is there any way to generate automatically fix orchestra file from fix dictionary and fix session configuration file?

Interfaces: Security keys for sessions

Capturing a question asked by Katherine Patrick:
Can the interfaces schema be used with a process to distribute security keys for sessions?

How to define rules

I checked everywhere but I couldn't find any example about the following.
For example I have tag ExpireTime(126) and TimeInForce(59). The tag ExpireTime(126) should be required if TimeInForce(59) is defined. How can I define this kind of rules in orchestra file?

Translate Score DSL expressions to natural language

Although the Score DSL grammar is very familiar to programmers, it should be translated to natural language for non-technical users in visualizations of Orchestra. This applies to docgen and testgen as well as other tools developed in the future that will have a UI.

Most translations can be simple substitution of tokens, e.g. "is less than" in place of the token "<". Ideally, the substitution should be table-driven so expressions can be rendered in multiple languages.

Module repositoryToOWL does not load base ontology from jar file

The semantic application can load the base ontology from a flat file but not from the jar file built for distribution. This is caused by more general problem in Java: files in a jar are not addressable by a URL. They are only accessed by getResourceAsStream(). But the OWLAPI expects a URL for imports, not an InputStream.

A member of the owlapi project offered a suggestion:
owlcs/owlapi#621

Unfortunately, other semantic tools such as Protege and TopBraid Composer will also need it as a flat file. Perhaps the application should just extract the file from the jar for users rather than read it in place.

Update build to current Java version

The current Java version is 10, version 11 is planned for release in Sept. 2018, and subsequent releases are planned at 6 month intervals. The Orchestra project code was originally built with Java 8. Oracle is making no more updates for older versions, although commercial support continues for another year. The main difference going from version 8 to later versions is the introduction of the module system in Java that requires some adjustments to the build procedure.

Extend scenarios to fields and code sets

Currently, the Orchestra XML schema provides for scenarios of messages and components. It has been suggested that they are needed for fields and code sets as well. The classic example is PartyRole in the Parties block. Often there is one set of values allowed on inbound order messages and another, more enriched set of values on outbound execution reports. In that case, PartyRole code set should have two (or more) scenarios, each with a finite set of valid values.

An impact of this change would be that field references would need a scenario to select the correct code set scenario.

We can continue to have the scenario default value of "base" so it need to be specified if there is only one scenario of an element.

Removal of 'conditional' enumeration in the 'presence' attribute

After a recent discussion with Jim Northey and Don Mendelson, it was mutually decided that the 'conditional' enumeration for the 'presence' attribute was unnecessary. The existence of an arbitrary list of rules in, for example, fields in messages, which can be applied across the other sets of value enumerations such as 'required', 'forbidden', etc., performs all of the functionality, and more, that was previously considered for the 'conditional' enumeration.

State machine demonstration

Orchestra provides syntax to define a state machine with enumerated states and transitions. Although docgen generates documentation, including a state machine diagram, from an Orchestra file, we lack a code demonstration of a working state machine. The demo should generate code from an Orchestra file.

Enhance RepositoryCompressor to select by section

The RepositoryCompressor utility currently selects messages by category. Alternatively, it should be able to select by section (PreTrade, Trade, PostTrade).

SBE / Orchestra integration

Generate a message schema for Simple Binary Encoding (SBE) from an Orchestra file.

Some requirements:

• Populate the datatypes section in Orchestra with mappings to SBE types.
• Since SBE lacks components, any components in messages must be expanded to their underlying fields

Also tracked as GTC Management issue GM-188.

Strip out session messages

Transform Repository2010to2016.xsl converts FIX Repository 2010 Edition to Orchestra schema. Unified Repository contains session messages and application messages. However, we have stated that an Orchestra file should only contain one protocol. That is, split into two Orchestra files for FIXT session protocol and application messages.

Either the converter should be enhanced or a separate script should be written to filter messages by section or category. The filter needs to be either inclusive or exclusive.

Scenarios for components

In the Orchestra model, messages currently may have multiple scenarios (use cases) with different layouts, but components do not. But it would be a useful feature to allow multiple flavors of a component. For example, an Instrument block would have a different set of fields depending on asset class. Like messages, the different scenarios of a component would share the same name, supplemented by scenario name, but would be uniquely identified by a different numeric ID.

Prepare Release Candidate 1

• Distinguish between normative and experimental modules by documentation and project organization.
• Update project build for OSSRH (Maven Central) deployment as version with suffix RC1.
• Use group ID io.fixprotocol and rename Java packages with that domain to be consistent with XML namespace.

Separate components and groups

As in the Unified Repository, the current Orchestra XML schema mixes component and repeating group definitions under one parent element, <components>. Phil Oliver proposed that we separate them into two paths: <components>/<component> and <groups>/<group>.

Update example

Update Orchestra example file mit_2016.xml:

• Remove Session messages. We now recommend that an Orchestra file should only contain one protocol; separate session protocol from application messages.
• Populate Concepts section.

Consider XML patch format for repository diff/merge utility

The current implementation creates its own simple format for an XML diff file. In the interest of standardization, consider changing it to this proposed standard: An Extensible Markup Language (XML) Patch Operations Framework Utilizing XML Path Language (XPath) Selectors. One benefit of the standard is that it is suitable for use with HTTP PATCH operation.

DocGenerator: provide for field and code set scenarios

The docgen utility was written against the RC3 schema which had scenarios for messages and components but not for fields and code sets. It needs to be updated for RC4 which does allow scenarios for fields and code sets.

Two enhancements are required:

1. Add scenarios to field and code set lists for navigation.
2. Generate a unique name for each HTML file.

Rule language label

As part of Orchestra, we have defined a Domain Specific Language (DSL) called Score for conditional expressions. However, some users may wish to use a different programming language, e.g. Groovy or Typescript. We can add an attribute to rule elements to specify the programming language. It can default to Score so we don't need to label every rule.

unified2orchestra.xslt incorrectly sets field type when unified repository element has enumDatatype

The script unified2orchestra transforms Unified Repository to Orchestra schema. In Orchestra, it sets a field's type to either a FIX datatype, e.g. UTCTimestamp, or the name of a code set. It detects when there will be a code set by looking for the presence of <enum> child elements of <field> in Unified Repository. However, it fails to handle another case, the presence of attribute enumDatatype that indicates that a field is borrowing enumerations from another field.

To resolve this, the script must look up the field name corresponding to the tag in the enumDatatype attribute.

Scenario names should not allow special characters

Currently, scenario names are defined in the XML schema as datatype xs:string. However, web-base tools for Orchestra, such as documentation generators, may turn element names and their scenarios into URL links. The string type allows almost any characters, including line feeds, carriage returns, and tab characters. However, URLs are more restrictive. RFC 3986 defines sets of reserved characters as follows.

reserved = gen-delims / sub-delims
gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"
sub-delims = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "="

I propose that scenarios should be restricted to the same XML type as elements since it would restrict them to characters that are safe for URLs. That type is called Name_t and is defined as follows:

<xs:simpleType name="Name_t">
<xs:restriction base="xs:string">
<xs:minLength value="1"/>
<xs:maxLength value="255"/>
<xs:pattern value="([A-Z]|[a-z]|)([0-9]|[A-Z]|[a-z]|)*"/>
</xs:restriction>
</xs:simpleType>

XML schema [RC1] lacks integrity contraints

The XML schema allows duplicate entries and broken element cross-references, and indeed some errors have been found in the current Repository. XML Schema provides for "key" to ensure uniqueness and "keyref" to validate cross-references. These XML Schema features are not very flexible, but whatever validations we can implement will help prevent errors.

Eliminating redundant data in Orchestra files

The ancestor of Orchestra files, the Fix Repository, contains redundant information. For example, a field reference:

<fixr:fieldRef id="5" name="AdvTransType" presence="required" added="FIX.2.7"> [...]

And then a related field definition entry:

<fixr:field id="5" name="AdvTransType" type="AdvTransTypeCodeSet" added="FIX.2.7" abbrName="AdvTransTyp"> [...]

The name attribute in the fieldRef element is redundant. It should always match the name in the field definition. This denormalization is unnecessary and wasteful. The id attribute is sufficient to specify the field reference.

The same also applies to components and groups, and there may be other similar examples.

Syntax for header and trailer inclusion

Orchestra could use syntax to specify when headers are applicable. Currently, Repository has StandardHeader and StandardTrailer on every message although they are only used in FIX4/FIXT tag value encoding.

Rich text documentation

It would be desirable to encode documentation elements in an Orchestra file with markup for rendering on a web page or in an Orchestra editor. Markdown is a possibility, but HTML would have wider applicability, and more tools are available for it. The problem is how to embed HTML within an XML document.

Jim Northey suggested polyglot markup which can be parsed as valid HTML or XHTML if certain constraints are honored and appropriate namespaces are declared. A single DOM is produced. See this guideline: Polyglot Markup: A robust profile of the HTML5 vocabulary

The usual use case for polyglot markup is for the document root to be well-formed HTML. In fact, an XML declaration is forbidden and a well-formed XML document is not guaranteed, according to the guideline. Our use case in Orchestra is slightly different. We wish to make embedded HTML parsable without breaking the parsing of surrounding valid XML.

XHTML requires strict syntax, unlike much HTML in the wild. That is, every tag must be closed, and nesting must be consistent. Element and attribute names must be lowercase. To satisfy both XML and HTML requirements, polyglot documents must be encoded as UTF-8.

Java compiler to use for FIX Orchestras code

Which Java compiler is required to compile the .java code files for FXI Orchestra test code ?

Scenarios for fields

Add scenarios for fields to capture different usage for fields. The key identifiers of tag and name would remain the same, and datatype would remain constant. However, annotations, rules, and codesets could be different.

Attribute lengthName is redundant

In issue #33, we removed redundant name attributes when an element is referenced by id (tag). However, lengthName remains in the schema. It is the name of a Length field associated with data field. The attribute is redundant with lengthId since it refers to the same field so it should be removed.

DocGenerator: write output to any filesystem

It would be desirable to be able to write the HTML documentation files to any filesystem, including a .zip archive or a remote system. The only requirement is that a FileSystemProvider is available for the storage scheme.

Concept names may be non-unique

Currently, concept elements are constrained in the XML schema to have unique name attributes. However, there may be more than one encoding that corresponds to the same concept. Therefore, the unique key constraint should be removed.

Example: In FIX 4.2 there were values of ExecType for PartialFill and Fill. These were replace in FIX 4.4 with one value Trade. Using the newer term, the concept is Trade. However, it would have two encodings in FIX 4.2, one with 150=1 and another with 150=2.

Unit test validation

In a fresh build outside the original build environment, a unit test failed in the docgen module for failure to find a test Orchestra file.

Also, all unit tests that generate a Orchestra file should validate output against the XML schema.

Section for default values

In a recent discussion with Jim Northey and Don Mendelson, the topic of default values (e.g. for the 'presence' attribute) came up. It was agreed that, rather than having these values be simply by external convention, that they should be explicitly specified once in a new section in the Orchestra file.

The issue of "default defaults", or globally default values which are determined to be most logical by the FPL community, can be handled by providing them in an initial template for new Orchestra files, which should be made available in any case for de novo development.

Fails to process FIXatdl test files

• Throws NullPointerException if XML root element attribute does not have a prefix for namespace.

Feature request: "which" attribute

In the 2016 repository, I would like to specify in component the possibility to include:
oneOf, anyOf, allOf
the fields (or sub components) that are nested (with similar effect as JSON which attribute).

Specifically, I advise to add an optional attribute called "which" to component, group and structure.
"which" may be set to either: oneOf, anyOf, allOf
In this case, all nested fieldRef must not be set as required.

Please consider.

XML schema [RC1] lacks stable keys for some elements

XML schema implementation. See standard issue FIXTradingCommunity/fix-orchestra-spec#1

docgen: add documentation search

The documentation generator would be easier to use with a way to find similarly named elements of different types, i.e. message, components, fields, codes. This could be accomplished by generating an index of all elements with a search box with suggestions or autocomplete.

unified2orchestra script does not properly handle FIX 4.2

The FIX Unified Repository represents FIX 4.2 repeating groups differently than it does for later versions of FIX. The FIX 4.2 representation does not provide a name or separate id attribute for the group; it only gives the tag of its NumInGroup field. Meanwhile, the FIX 4.4. representation provides all attributes in a wrapper component.

The unified2orchestra script needs to be enhanced to handle both representations. A consequence is that it will need to be provided with a FIX 4.4 or later repository to retrieve the missing attributes in FIX 4.2. It will match on the NumInGroup tag.

More efficient identifier structure

Previously, a system using lengthy globally unique identifiers on many elements in an Orchestra file was considered. Discussion with Jim Northey and Don Mendelson led to the thought that this was excessive, in terms of required processing power and file size, and possibly other issues.

The conclusion drawn is that a globally unique identifier for each new Orchestra file, specified in a top level element, would suffice to differentiate each element, via appropriate concatenation of that id with elements in the file (such as GUID+Field+Field_id)

The identifier might itself be comprised of an organizational unique identifier, potentially assigned by FPL organization, plus a further random, unique GUID to ensure uniqueness per file globally.

DSL enhancement: test for existence of field or group entry

It would sometimes be useful to test for existence of a field or repeating group entry in a DSL expression. The grammar could invoke a built-in function with "exists id" or "!exists id".

Motivating example: Test for existence of an entry in SecMassStatGrp by the SecurityID from an order. If there is no entry for security status for the instrument, then reject the order due to unknown instrument.

XML schema optimizations

Candidates for optimizing the Repository 2016 Edition XML schema:

• Move abbrName to the oidGrp attribute group. Every element that has a name can have an abbreviated name.
• Section attribute on message is redundant since it has category and category already has a section. No other element, aside from category, has section attribute.
• Only field has baseCategory while component and message have category. Unless there is an intended distinction, make them the same.
• Only field has baseCategoryAbbrName . What is it used for? Only 46 fields have it, and it is different than abbrName and category. Remove it?

Tooling for FIX members

Could someone please advise how FIX members can get the FIX Orchestra tooling that allows you to generate Orchestra files from existing log files? I understand that some of the tooling like this is only available to FIX members.

Thanks,
Dominic

NumInGroup encoding

In the current Orchestra XML schema, a repeating group element has an attribute to reference the tag (numeric id) of a NumInGroup field, like this:

<fixr:group id="1007" added="FIX.4.4" name="LegStipulations" numInGroupId="683" category="Common" abbrName="Stip">

Phil Oliver pointed out that fields are now referenced by tag and scenario name. Therefore, to be complete, a NumInGroup should allow a scenario also.

Rather than adding another attribute to group, we can create a child element numInGroup with the same XML type as fieldRef. It would gain a scenario attribute as well as all other attributes available to field references.

Docgen should visualize state machines

Docgen currently visualizes message structures and lists actors and shows workflow. However, it does not visualize state variables or state machines. A user should be able to explore states and transitions.

Module dsl-xtext fails to build in Maven

The dslxtext project builds when GenerateScore.mwe2 is manually invoked from the Eclipse IDE but it fails to build in Maven. Until this issue is resolved, the module is not included in the fix-orchestra parent project.

Example file has fields after StandardTrailer

Example file mit_2016.xml has some message structures with some fields after StandardTrailer (componentRef id='1025').

docGen: the module doesn't have unit test

orchestrations(https://github.com/FIXTradingCommunity/orchestrations/tree/master/NYSE%20Pillar) use the DocGen maven module to generate doc. I'll copy repository(NYSEPillarBinaryPhase2.xml) to docGen module and create a unit test.

Key fields for message correlation

Orchestra could benefit from identification of key fields in a message to link an execution to an order, for example. (It already has syntax to define uniqueness of keys.) Sometimes there are alternative keys, such as ClOrdID and OrderID.

session-quickfix: Session Configuration file is created empty

When the test runs, the session configuration file is created but empty. The reason is that sessionProtocol doesn't have version attribute in SampleInterfaces.xml.

XMLDiff should support ordering of elements

The XML Diff utility currently compares elements and attributes by content only, regardless of order. This must remain the case for attributes since the XML standard says that order of attributes is not deterministic. We can change this for elements, however.

If an XML file is controlled by a schema, then elements are ordered in some cases while unordered in others. A sequence is ordered while "all" and other cases are unordered. However, the Diff utility works on any XML file and is schema unaware. Therefore, it cannot distinguish those cases. A simple approach is to provide program switch to either consider all elements ordered or not.

The Patch operations (RFC 5261) are add, replace, and remove; there is no move operation (although the standard alludes to a possible future extension). Be aware, therefore, that an element move will be rendered as an add and remove pair if ordering is turned on.