qt4cg/qtspecs

QT4 specifications

X(Q)uery and XSL(T) Specifications (qtspecs)

This is a forked source for the XSLT and XQuery Specifications created for the development of draft 4.0 proposals. This source builds the HTML specs and errata. It is maintained by the W3C XSLT-4.0 Community Group established by Michael Kay.

View the latest HTML version of the draft specs via the links below:

Draft 4.0 Spec	Diff against 3.0 Specs
XSLT Transformations (XSLT) Version 4.0	(Diffs)
XPath and XQuery Functions and Operators 4.0	(Diffs)
XML Path Language (XPath) 4.0	(Diffs)
XQuery 4.0: An XML Query Language	(Diffs)

Participate in Spec Development

There are numerous paths to participating in qtspec development.

• Join the XSLT Extensions Community Group--the official W3C Community Group established to develop 4.0 specs. On the group's page you can find mailing list archives and instructions for subscribing to the list. The team is now actively meeting weekly via Zoom. Join the Community Group for more information.
• Open or comment on an issue for the specs. This is the primary method for requesting modifications to the specs. Additions to the spec should be well described in order to aid adoption and implementation. Editorial changes are also accepted via issues.
• Fork the repo, make changes, and make a pull request. Note the contribution guidelines.
• Join the xml.com Slack. qtspecs discussion is centered in the #xpath-ng channel.

qtspecs's Issues

[DM31] Function types do not form a hierarchy

Section §2.8.1 in the XDM specification states "The space of all possible function signatures forms a hierarchy of function types. "

This is clearly incorrect. It's a directed acyclic graph. (It's not even a lattice, because there is no minimum or maximum - there is no function signature that is a supertype of all others, nor is there one that is a subtype of all others). (I think there is one lattice for each possible arity.)

Allow tokens in xsl:mode/@name

A minor request:

I find I often want to re-use the same behaviours across modes, or use related modes across similar content.

In the xsl:template and xsl:apply-templates, we can use a space separated list of mode names: it would be convenient if I could do the same thing in xsl:mode instructions.

array:values (resolved: map:values, map:entries)

EDIT: Revised and aligned with the recently added and proposed map and array functions (#314, #357):

The suggested functions are based on the observations that:

• users who prefer readable function names tend to reject ?* as handy shortcut;
• functions are often more flexible and better composable than FLWOR enhancements (but of course both could be added).

array:values

Summary

Returns all members of an array as a sequence. Equivalent to $array?*, but better composable and easier to read (especially for newcomers), and known from other programming languages.

Signature

array:values($array as array(*)) as item()*

Example

(: Query :)
let $array := [ (), 2, [ 3, 4 ] ]
return array:values($array)

(: Result :)
2
[ 3, 4 ]

map:values

Summary

Returns all values of a map as a sequence. Complementary to map:keys, and equivalent to $map?*, but better composable and easier to read (especially for newcomers), and known from other programming languages.

Signature

map:values($map as map(*)) as item()*

Example

(: Query :)
let $map := map { 'a': (), 'b': 2, 'c': [ 3, 4 ] }
return map:values($map)

(: Result :)
2
[ 3, 4 ]

map:entries

See #357 on composing and decomposing Key-Value Records.

Summary

Returns each entry of a map as a singleton map. Equivalent to map:for-each($map, map:entry#2).

Signature

map:entries($map as map(*)) as map(*)*

Example

(: Query :)
let $map := map { 'a': (), 'b': 2, 'c': [ 3, 4 ] }
for $entry in map:entries($map)
return element { map:keys($entry) } { string-join(map:values($entry)) }

(: Result :)
<a/>
<b>2</b>
<c>34</c>

array:members

See #314.

Specify optional parameters to create bounded variadic functions

The current Editor's Draft for XPath and XQuery define a %variadic("bounded") function type, but does not define a syntax for specifying these.

Grammar

ParamList ::= RequiredParamList ( "," OptionalParamList )?
RequiredParamList ::= Param ("," Param)*
Param ::= "$" EQName TypeDeclaration?
OptionalParamList ::= OptionalParam ("," OptionalParam)*
OptionalParam ::= Param ":=" ExprSingle

Note:

I've followed the structure of positional and keyword arguments here, so the optional parameters are only valid at the end of the function. If it is decided that optional parameters can be declared anywhere in the parameter list, the grammar simplifies to:

ParamList ::= Param ("," Param)*
Param ::= "$" EQName TypeDeclaration? ( ":=" ExprSingle )?

Semantics

[Definition: a parameter is an optional parameter if it has a default value specified using the := ExprSingle syntax.] Optional parameters affect the value of R (the number of parameters that do not have a default value) in the 4.4.1 Static Functions section.

Notes

There are open questions on what to allow in the default value expression. Specifically, how to support things like the context item for functions such as fn:data#0 that use the context item if not specified (e.g. when used at the end of a path expression).

An investigation should be done on the standard functions and vendor built-in functions to see what values they take as defaults.

The item-type(T) syntax is not defined

Section 3.7.2 The judgement subtype-itemtype(A, B) of the XPath 4.0 and XQuery 4.0 specifications mention item-type(N), as does section 5.19 Item Type Declarations of the XQuery 4.0 specification. It is also not in the EBNF grammar -- searching for "item-type" only finds the ItemTypeDecl symbol in the XQuery 4.0 EBNF.

This should be defined in section 3.6 Item Types.

[FO] The parameter types for fn:unique and array:partition are incorrectly specified.

1. In both signatures of fn:unique the $values parameter has the type xs:anyAtomicType** which should be xs:anyAtomicType*.
2. In array:partition the $input parameter is item(*)* which should be item()*.

[XPath] `%variadic("sequence")` does not allow specifying some argument values in the variadic sequence, and in one case even not the variadic sequence itself

The current specification for sequence variadic functions (at: https://qt4cg.org/branch/master/xquery-40/xpath-40-diff.html#id-static-functions) has the following issues:

1. An empty sequence cannot be specified into the sequence of variadic arguments
2. Even the last (sequence) argument that is supposed to contain the variadic arguments "disappears" when no variadic argument is specified on the function call

This results in difficulties to describe, explain and implement such sequence-variadic functions, and in inability to provide any argument value (like the empty sequence) within the last, sequence argument.

Proposed improvement, which doesn't have the above problems:

Use array-variadic functions instead of sequence-variadic:

%variadic("array") indicates that the function is array-variadic. An array-variadic function declares one or more parameters, of which the last must be an array. If the declaration includes N parameters, then a call on the function may supply N-1 or more arguments. However, the effective call always consists of N arguments, the first N-1 of which are the values for the positional parameters of the function, and the Nth argument is an array that contains all provided variadic arguments.

Why an array-variadic function doesn't have the problems 1 and 2 above?

1. An empty sequence can be specified in the last argument, for example: [1, (), 2]. For example:
   myFun($posArg1, $posArg2, ..., $posArgN_minus_1, [1, (), 2])
2. When no variadic arguments are supplied, the last argument of the effective function call is still supplied -- it is the empty array. For example: myFun($posArg1, $posArg2, ..., $posArgN_minus_1, [])

try/catch: New variable for all error information

Within the scope of a catch clause, some variables are implicitly declared. However, it’s more than tedious to forward all this information to another function:

try {
  'oh' + 'dear'
} catch * {
  handle-error($err:code, $err:description, $err:line-number, $err:column-number, ...)
}

What about binding all data to an additional $err:map variable?

map {
  'code': $err:code,
  'value': $err:value,
  'description': $err:description,
  'line-number': $err:line-number,
  'column-number': $err:column-number,
  'additional': $err:additional
}

The code could then be shortened to:

try {
  'oh' + 'yes'
} catch * {
  handle-error($err:map)
}

[XPath] [XQuery] Better names for ThinArrowTarget and FatArrowTarget

The ThinArrowTarget and FatArrowTarget EBNF symbols do not follow the convention in the XPath and XQuery specifications of describing the semantics of the symbols, but instead describe how they are written.

I propose the following names:

1. CurryingArrowTarget instead of FatArrowTarget -- This is because the LHS is being curried into the function, making the function call it applies to take one less parameter. Other possible names would be SimpleArrowTarget or UnaryArrowTarget.
2. MappingArrowTarget instead of ThinArrowTarget -- This is because both variants make use of the simple map operator.

xsl:sequence: @as

I'd like to see@as on xsl:sequence. That way i can write, e.g.

<xsl:function name="dc:slice-count" as="xs:integer">
  <xsl:param name="toast" as="element(toast)" />
  <xsl:for-each select="$toast" as="xs:integer">
    <xsl:sequence select="@cooked-slices + @raw-slices"  as="xs:integer" />
  </xsl:for-each>
</xsl:function>

It would be an error for the for-each to have other than exactly one integer as its result, and the same for the @sequence. In this simple example there's not much scope for that to happen of course,

Maybe on anything with a select attribute?

Parenthetically, a context-item attribute on xsl:sequence would obviate the XSLT1-ish xsl:for-each there, although $toast/(@A, @b) => sum() would work as well and be XSLT 3-ish.

fn:slice, array:slice: Signatures, Examples

EDIT: 1. is obsolete, 2. and 3. are still up-to-date:

1. The current specification for fn:slice has only one signature.

It might be recommendable to also provide signatures with 1 and 2 arguments (especially for users who don’t want to use the new syntax for specifying optional arguments).

2. The last examples look wrong; I would expect the input as results:

The expression fn:slice(("a", "b", "c", "d"), 0) returns ().
The expression array:slice(["a", "b", "c", "d"], 0) returns [].

3. The first argument of array:slice should be renamed from $input to $array.

Relax type incompatibility in order by clause (impl. dep. instead of XPST0004)

In the case where XQuery is used with very large sequences (billions/trillions of items or of tuples) with a parallel evaluation [1], the order by clause in its current state is costly to evaluate, because checking the primitive types for compatibility requires an extra step and materialization (in the case of Spark: an additional action to perform this check).

Relaxing this by making the order between different primitive types implementation-dependent (for the purpose of order by) rather than throwing XPST0004, in case of several incompatible primitive types in the comparison keys, would make parallel implementations more efficient.

[1] http://www.vldb.org/pvldb/vol14/p498-muller.pdf

Second parameter of fn:sum must be neutral element for +

Currently fn:sum specifies the intent of the second parameter in a note:

The second argument allows an appropriate value to be defined to represent the sum of an empty sequence. For example, when summing a sequence of durations it would be appropriate to return a zero-length duration of the appropriate type. This argument is necessary because a system that does dynamic typing cannot distinguish "an empty sequence of integers", for example, from "an empty sequence of durations".

When implementing fn:sum on sequences of billions of items (numerics, or durations, etc), another aspect arises: this second parameter must also be, for this to work and for optimizations to be possible, a neutral element for +.

Indeed, a distributed system like Spark will produce intermediate sums for (possibly empty) subsets, and will naturally use $zero for the sum of an empty subset. Intermediate totals are aggregated in a treewise fashion. For the result to be correct, it must be the case that $zero + $x eq $x for any item in the sequence provided as the first parameter. It is fully aligned with the idea of the note above, but I would suggest to make this requirement a bit stricter.

[xslt] annotation-prefixes

By analogy with Schema annotations, i'd like to see an annotation-prefixes attribute on xsl:stylesheet/transform containing a space-separated list of NCName namespace prefixes that are associated with annotations; the XSLT processor would discard these elements (including children) during compilation. Here's a rough go at some text.

The purpose is to be able to include annotations at any level where elements are allowed - for example, inside an xl:variable or template or function body. Annotations might include XTest unit tests, Schmatron rules, human-readable documentation, CSS styles, or more, and could be used by other operations than the XSLT transformation: for example, by processing the XSLT source itself with XSLT.

It should be possible for the same element to be both an extension element and an annotation, but the behaviour is implementation-dependent in this case (for example, an API might allow an extension to access content or convert the annotation elements to something else on compilation).

XSLT instructions occurring inside annotation elements are ignored along with other content, except for xsl:fallback instructions (and their contents) if the prefix was also declared as an extension prefix and no matching extension was found. Similarly, extension attributes are discarded. The fallback behaviour might be used to support an XSLT-based implementation, for example by reading the XSLT source and processing embedded Schematron tests.

Attributes of annotation elements are not considered to be attribute value templates and content is not considered to be text value templates - that is, { and } are not special. However, an annotation element or attribute backed by an extension could perform such processing if an implementation supported it. The behaviour of shadow annotation attributes and xsl:use-when is implementation defined, but expressions contained in them must be processed as elsewhere.

Support standard and user-defined composite values using item type definitions

The composite values defined in 4.14.4 Composite Atomic Values are currently specified as a table. This means that it is not possible for users to define their own properties for custom types. It is also harder for editors/IDEs, or other tools to implement as there is an element of hard-coding the logic.

These could be implemented as a properties/values record associated with the defined type. The values of the record could then be arity-1 functions that are called with the supplied value when accessed via maps. For example, in XQuery:

declare %composite-values("composite-values") type xs:date external; (: built-in :)
declare type date-composite-values := record(
    year: fn:year-from-date#1,
    (: ... :)
);

and XSLT:

<xsl:item-type name="xs:date" composite-values="date-composite-values"/>
<xsl:item-type name="date-composite-values" as="record(
    year: fn:year-from-date#1,
    (: ... :)
)"/>

So xs:date("1999-10-15")?year would be evaluated as date-composite-values?year(xs:date("1999-10-15")).

[FO] fn:all and fn:some have an xs:integer* return type, but describe an xs:boolean return type

The fn:all function states:

The result of the function is true if and only if the expression every $i in $input satisfies $predicate($i) is true.

but the return type is specified as xs:integer*. -- It should have a return type of xs:boolean.

A similar issue occurs with fn:some.

[FO] Conversion between xs:QName and Q{uri}local format

It would be good to have functions to convert from Q{uri}local format to xs:QName (perhaps with the option of supplying a prefix), and the reverse, generating Q{uri}local from an xs:QName.

I don't think we can extend the xs:QName constructor function to do this, because that has to remain consistent with XSD and (sadly) we can't change the lexical space in XSD.

So I would suggest a single-argument form of fn:QName that accepts either local or Q{}local or Q{uri}local. But this doesn't allow a prefix to be added. An alternative would be fn:EQName#1 accepting local or Q{}local or Q{uri}local, with an optional second argument to supply a prefix.

For the reverse, perhaps fn:EQName-from-QName#1? But the terminology here is a bit loose: in the grammar, EQName covers various formats of which URIQualifiedName is one. But fn:URIQualifiedName-from-QName is a bit of a mouthful.

[XPath] [XQuery] The modified SingleType EBNF symbol is redundant.

The SingleType definition has changed as follows:

-SingleType ::= SimpleTypeName "?"?
+SingleType ::= (SimpleTypeName | LocalUnionType) "?"? 	

However, SimpleTypeName has also changed:

-SimpleTypeName ::= TypeName
+SimpleTypeName ::= TypeName | LocalUnionType 	

Therefore, the change to SingleType is not needed.

[XPath] Introduce the lookup operator for sequences

In XPath 3.1 it is convenient to use the ? lookup operator on arrays and maps.

It is easy and readable to construct expressions, such as:

  [10, 20, 30]?(2, 3, 1, 1, 2)

And this understandably produces the sequence:

20, 30, 10, 10, 20

However, it is not possible to write:

(10, 20, 30)[2, 3, 1, 1, 2]

or

(10, 20, 30)(2, 3, 1, 1, 2)

or

(10, 20, 30)?(2, 3, 1, 1, 2)

This proposal is to allow the use on sequences of the postfix lookup operator ? with the same syntax as it is now used for arrays.

The ? lookup operator will be applied on sequences whose first item isn't an array or a map. The only change would be to allow the type of the left-hand side to be a sequence, in addition to the currently allowed map and array types. At present, applying ? on any such sequence results in error. In case the first item of the LHS sequence is an array or a map, then the current XPath 3.1 semantics is in force, which applies the RHS to each item in the sequence.

The restriction in the above paragraph can be eliminated if we decide to use a different than ? symbol for this operator, for example ^

The goal of this feature is achieving conciseness, readability, understandability and convenience.

For example, now one could easily produce from a sequence a projection / rearrangement with any desired multiplicity and ordering.

Thus, it would be easy to express the function reverse() as simply:

$seq?($len to 1 by -1)

Improve the discoverability and parseability of the mathematical operator symbols

In order to keep the XPath and XQuery languages parseable without the need of a pre-processing step, the operators should be integrated into the grammar, such as in:

[29] UnionExpr ::= IntersectExceptExpr ( ("union" |  "|" | "∪") IntersectExceptExpr )*

To make them more easily discoverable to someone reading the specification, the behaviour of the Unicode aliases should be included the description in the text related to that section. For example:

The union, |, and ∪ operators are equivalent. They take two node sequences as operands and return a sequence containing all the nodes that occur in either of the operands.

NOTE: With the introduction of the LocalUnionType, a pre-processing step would need to map ∪ to | in order to avoid ∪(xs:string, xs:integer) being valid. -- This would then mean that things like the CatchErrorList would then be able to use ∪, as in try {} catch err:FOER0001 ∪ err:FOER0002 {}.

Provide an XML version of the stack trace

While the string version of fn:stack-trace() is useful for debugging and including in log messages, being able to process that (from an XML representation) is also useful.

Use Cases

1. providing extended functionality, like implementing a current-function-name() helper function -- e.g. fn:stack-trace("json")[1]?function-name;
2. customizing the format of the stack trace (e.g. standardizing it across different implementations);
3. using the information in libraries/IDEs/editors that call the queries -- e.g. by returning the XML and processing it in the library/IDE/editor, such as mapping the data to stack frames in the IDE/editor. Note: This is what I'm doing in my IntelliJ plugin with the MarkLogic stack XML to process query exceptions and the stack when debugging a query.

fn:stack-trace

fn:stack-trace($format as enum("text", "xml", "json") := "text") as item()

Like the current specification version of this function (with the same default semantics), but also supports XML and JSON formats. The "text" format returns an instance of xs:string in an implementation-defined format, the "xml" format returns an instance of element(fn:stack-trace), and the "json" format returns an instance of array(fn:stack-frame).

Here, fn:stack-frame is defined as:

declare type fn:stack-frame as record(
    uri: xs:string,
    function-name: xs:QName?,
    line-number: xs:integer?,
    column-number: xs:integer?,
    *
);

The XML version has the same information as elements in the fn: namespace (e.g. fn:uri).

fn:format-stack-trace

fn:format-stack-trace($stack as item(),
                      $format as enum("text", "xml", "json") := "text") as item()

If $stack is an instance of element(fn:stack-trace), it is converted into the desired output format. (If the output format is "xml", no processing is performed.)

If $stack is an instance of array(fn:stack-frame), it is converted into the desired output format. (If the output format is "json", no processing is performed.)

Otherwise, an err:XPTY0004 error is raised.

fn:parse-stack-trace

fn:parse-stack-trace($stack as xs:string,
                     $format as enum("xml", "json")) as item()

This function takes a stack trace in the implementation-defined format and parses it to XML or JSON. The "xml" format returns an instance of element(fn:stack-trace), and the "json" format returns an instance of array(fn:stack-frame).

If $stack is not in the correct format, an error (error code TBD) is raised.

Note: This could be useful when processing log messages or similar output.

[XPath] [XQuery] Support RecordTest self references without occurrence indicators

This would be useful for defining things like binary trees, where the fields are optional but (if supplied) the values are not. So it is more logical to define them as:

declare item-type binary-tree as record(
    left? as ..,
    right? as ..,
    value as item()*
);

Allow item-type to be matched within its definition scope

In https://qt4cg.org/branch/master/xpath-functions-40/Overview-diff.html#func-random-number-generator, the rng item type is defined as:

record(
    number   as xs:double,
    next     as (function() as record(number, next, permute, *)),
    permute  as (function(item()*) as item()*),
    *
)

It would be helpful and more type specific if this could be defined as:

record(
    number   as xs:double,
    next     as (function() as rng),
    permute  as (function(item()*) as item()*),
    *
)

where the next field references the rng type being defined -- this is like how structures in other languages (C/C++, Java, C#) can reference themselves as property types.

This would also provide an alternative for the .. (self reference) specifier.

Note: the .. syntax is still useful in the case of anonymous record types.

Allow record(*) based RecordTests

The other ItemTypes that support specifying information about the type allow type(*) to represent any instance of the type. The new RecordTest ItemType should support this.

Syntax

The:

RecordTest ::= "record"  "("  FieldDeclaration  (","  FieldDeclaration)*  ExtensibleFlag?  ")"

symbol should be changed to:

RecordTest ::= AnyRecordTest | TypedRecordTest
AnyRecordTest ::= "record"  "("  "*"  ")"
TypedRecordTest ::= "record"  "("  FieldDeclaration  (","  FieldDeclaration)*  ExtensibleFlag?  ")"

NOTE: This follows the structure of the other any/typed tests (e.g. MapTest).

Semantics

The record(*) item type test is equivalent to map(*).

[FO] fn:intersperse

A simple proposal, for a change:

A new function would be handsome to insert separators in a sequence. Equivalent functions exist in other languages; in Haskell and Dart, they are called intersperse:

Summary

Inserts the defined separator between the items of a sequence and returns the resulting sequence.

Signature

fn:intersperse($values as item()*, $separator as item()*) as item()*

Example

let $div :=
  <div>
    <span>one</span>
    <span>two</span>
    <span>three</span>
  </div>
return fn:intersperse($div/span, ',')

[XPath] Allowing multiple let clauses in LetExpr and for clauses in ForExpr

Currently, it is possible to have multiple SimpleLetBindings in a SimpleLetClause, but not have multiple SimpleLetClauses in a LetExpr. The same applies for ForExpr.

I propose that this should be possible. In other words, make the following changes to the XPath 4.0 grammar:

[12] ForExpr ::= SimpleForClause SimpleForClause* "return" ExprSingle
[15] LetExpr ::= SimpleLetClause SimpleLetClause* "return" ExprSingle

This way, a user can write the following in XPath 4.0:

let $x := 1.0
let $y := 2.0
return $x + $y

in addition to the following in XPath 3.1 and earlier:

let $x := 1.0, $y := 2.0
return $x + $y

Note that this does not add any new capability to XPath (in particular, for and let clauses cannot be mixed like in XQuery) as a user can define multiple let or for clauses, it just provides them with an alternative way of expressing that like they can do in XQuery (where it is common to use a series of let clauses instead of let bindings for multiple local variables).

Kind regards,
Reece

URILiteral is defined in the EBNF grammar but not used

Comparing the WithExpr grammar, XPath has:

[10] NamespaceDeclaration ::= QName  "="  StringLiteral

whereas XQuery has:

[44] NamespaceDeclaration ::= QName  "="  URILiteral

Therefore, it looks like the intention is to use URILiteral in the XPath EBNF grammar for NamespaceDeclaration as well.

[FO]The `union ( | )`, `itersect`, `except` and `combine (,)` operators are not mentioned in the F & O. Have not the best categorization in the XPath spec.

1. Searching the F & O document for the union ( | ), intersect and except operators finds 0 occurrences. Neither can , be found, and of course, it is unsearchable.

2. In the XPath spec these are described in section "4.7.3 Combining Node Sequences". The word node-set is not mentioned at all and this is misleading, because the union ( | ), intersect and except operators are in fact set-operators, or at least set-generating operators.

Thanks,
Dimitre

Allow function keyword inline functions without parameters

The current draft InlineFunctionExpr adds -> as a shorthand. This shorthand allows optional parameter lists (e.g. -> { true() }), but the function keyword version of this requires a parameter list. For consistency, the function keyword version should also have an optional parameter list.

This means that the syntax for InlineFunctionExpr can be simplified to:

InlineFunctionExpr ::= ("function" | "->")  FunctionSignature?  FunctionBody

---

Update: From recent discussions, the -> operator as both a thin arrow expression and an inline function definition is confusing. As such, a replacement for -> in the inline function context should be identified.

In the context of the variant without a parameter definition (e.g. when used with arrow operators), the question is how should it work. I suggest:

1. it should be a 0 and 1 arity function with the parameter argument defaulting to ();
2. if the parameter is a single value, it should bind to the . (context item) and ~ (context value -- #129);
3. if the parameter is an empty sequence, or multi-valued sequence, it should bind to the ~ (context value -- #129) only.

This way, it will be usable in multiple contexts.

[FO] The math:atan2 notes incorrectly defines its behaviour.

When $y is positive and $x is negative, the notes define this to be equal to π - atan($y div $x) when it should be atan($y div $x) + π.

When $y is negative and $x is positive, the notes don't define the value. It is the same as the first case, when $y is positive and $x is positive, so the first case should be "If $y is positive or negative, and ...".

When $y is negative and $x is negative, the notes don't define the value. The value is equivalent to atan($y div $x) - π.

[XPath]A value in the last row (for "sequence-variadic" functions) of the table "Number of Arguments allowed in a Function Call" is incorrect

In another issue ( #25 ) it was already discussed that the proposed "sequence-variadic" functions have definite problems.

This issue is about the incorrect value in the MaxK column of the "sequence-variadic" (last) row of the table titled "Number of Arguments allowed in a Function Call", which is in the section "4.4.1 Static Functions".

As published, the value of the 7th (last, for the MaxK column) item in this row is 0.

In fact all of the required positional arguments can be entered as keyword arguments in a function call (there is no reason why this cannot/shouldn't be done).

Thus, the value for MaxK (the maximum number of keyword arguments) must be R, (the current value of 0 is wrong).

Proposed corrective action:

1. Replace "sequence-variadic" with "array-variadic", which avoids the problems of "sequence-variadic" (as discussed in #25 )
2. In the row for "array-variadic" of the table, specify R as the value for the column MaxK

[XPath] [XQuery] Keyword arguments don't work with all parameters/keys in static functions.

The KeywordArgument symbol restricts the argument name to an NCName. This has two issues:

1. for non-variadic and bounded-variadic functions, a parameter can be a QName, so may be in a different namespace, or there can be ambiguity if there are multiple parameters with the same local-name in different namespaces;
2. for map-variadic functions, a parameter key can contain spaces, so cannot be expressed as an NCName.

Syntax

KeywordArgument ::= KeywordArgumentName  ":"  ExprSingle
KeywordArgumentName ::= EQName | StringLiteral

NOTE: I'm using the favoured map-based syntax here. If that is not used, then the ":" should be ":=" as it is in the current draft.

Semantics

For non-variadic and bounded-variadic functions, a KeywordArgumentName is matched as follows:

1. An EQName matches against the expanded QName of the parameter;
2. A StringLiteral is cast to an NCName (with an XPTY0004 error if it is not a valid NCName), which is in no namespace (like other variables such as VarName symbols); the resulting expanded QName then matches against the expanded QName of the parameter.

For map-variadic functions, a KeywordArgumentName is matched as follows:

1. An NCName uses the local-name as the key in the constructed map cast to the key type of the map. This follows the XQFO casting rules with the source type of the local-name being xs:NCName and the target type being the map's key type.;
2. A QName or URIQualifiedName results in an XPTY0004 error as it does not form a valid key name;
3. A StringLiteral uses the value of the string as the key in the constructed map.

[XPath] [XQuery] The unknown ArgumentPlaceHolder EBNF symbol is referred to in several places

The references to ArgumentPlaceHolder should have a lower case h to match the EBNF symbol name ArgumentPlaceholder in the grammar.

Kind regards,
Reece

[XQuery] String Value Templates

A string value template (SVT) is a StringLiteral that supports enclosed expression values and entities. It is written as either T"..." or T'...', where the T stands for "template".

Note: An SVT is similar to an attribute value template or text value template in XSLT.

For instance, the following expression:

for $s in ("one", "two", "red", "blue")
return T"{$s} fish"

evaluates to the sequence ("one fish", "two fish", "red fish", "blue fish").

Note: A string value template T"xyz" is equivalent to the expression <svt t="xyz"/>/@t/string().

Grammar

PrimaryExpr ::= ... | StringValueTemplate
StringValueTemplate ::= ('T"' (EscapeQuot | QuotAttrValueContent)* '"')
                      | ("T'" (EscapeApos | AposAttrValueContent)* "'")

Note: The T" and T' are a single token/unit (i.e. no whitespace/comments are allowed between the characters), just like the Q{ in BracedURILiterals.

Generalize lookup operator for function items

The current lookup operator is a specialized expression for maps and arrays. All kinds of data structures can be realized with functions, and maps and arrays are functions as well, so it would be pretty straightforward to extend the lookup operator to arbitrary function items:

Use Cases

Return name elements whose string values contain supplied substrings

declare variable $DOC := <xml>
  <name>Jack Daniels</name>
  <name>Jim Beam</name>
  <name>Johnny Walker</name>
</xml>;

let $names := function($key) {
  $DOC//name[contains(string(), $key)]
}
return $names?('Jack', 'Jim', 'Johnny')

(: result :)
<name>Jack Daniels</name>,
<name>Jim Beam</name>,
<name>Johnny Walker</name>

Return squares of supplied integers

let $square := math:pow(?, 2)
return $square?(1 to 5)

(: result :)
1, 4, 9, 16, 25

Remarks

• XPTY0004 must be raised if the wildcard * is specified as key, and if the input is neither a map nor an array.
• The extension could easily be combined with the extension for sequences (see #50).

[XQuery] The TypeswitchExpr and CaseClause symbols have repeated VarNames

The TypeswitchExpr symbol has changed as follows:

-TypeswitchExpr ::= "typeswitch" "(" Expr ")" CaseClause+ "default" ("$" VarName)? "return" ExprSingle
+TypeswitchExpr ::= "typeswitch" "(" Expr ")" CaseClause+ "default" ("$" VarName)? "$" VarName "return" ExprSingle

The CaseClause symbol has changed as follows:

-CaseClause ::= "case" ("$" VarName "as")? SequenceTypeUnion "return" ExprSingle
+CaseClause ::= "case" ("$" VarName "as")? "$" VarName "as" SequenceTypeUnion "return" ExprSingle

That is, both of these have a repeated "$" VarName ... section from the previous optional section.

Is this an effect of removing the references to the older specifications from the grammar?

readme

The readme needs an update

Link to the 4.0 html spec files rather than the 3.1 spec

Extending element and attribute tests to NameTest unions

In the XPath and XQuery drafts the element and attribute tests have been extended to support any NameTest, meaning they now support all wildcard forms, which is great.

It is possible to write a path expression that takes a union of different paths -- such as html//(ol|ul) -- however it is not possible to define a precise type that accepts that path expression in variables, parameters, or return types, so if a user does specify a type some type information is lost during the static analysis phase.

As such, I propose renaming CatchErrorList to NameTestUnion and making ElementTest and AttributeTest accept a NameTestUnion:

[86] CatchClause ::= "catch" NameTestUnion EnclosedExpr
[87] NameTestUnion ::= NameTest ("|" NameTest)*
[212] AttributeTest ::= "attribute" "(" (NameTestUnion ("," TypeName)?)? ")"
[215] ElementTest ::= "element" "(" (NameTestUnion ("," TypeName "?"?)?)? ")"

Kind regards,
Reece

Extend FLWOR expressions to maps

Edit: Current proposal (#31 (comment)):

for key $key in ...
return …

for value $value in ...
return …

for key $key value $value in ...
return …

---

With the addition of the for member syntax for arrays, it is possible to use a ForExpr/FLWORExpr to enumerate the contents of sequences and arrays, but not maps. In order to be consistent and symmetric across these types, the for member syntax should be extended to support maps by enumerating the key/value entries of the map.

Given a map of type map(K, V) the member RecordTest would be record(key as K, value as V). Given a map of type map(*), the member RecordTest would be record(key, value).

This would allow a user to write expressions like:

for member $entry in $map
return element { $entry?key } { $entry?value }

NOTE: With the addition of the array:values and map:entries functions in issue #29, it is possible to avoid the need of the for member syntax for arrays and maps, but may be worth keeping for people who prefer the wordy style of the XPath/XQuery syntax.

JSON Parsing & Serialization: Numbers

To-dos:

• Add number-formatter to the json method in the serialization spec.

---

Sometimes people put very large numbers in a JSON file, if they are parsed as double, the numbers are corrupted. Or they become confused when 1000000 from the input becomes 1e6 in the output. Finally parsing a double is slower than parsing an integer.

There could be an additional option for parse-json/json-doc number-type with possible values:

• double: Parse all numbers as xs:double
• decimal: Parse all numbers as xs:decimal
• string: Return numbers as xs:string (so 1e6 stays "1e6" and 1000000 stays "1000000")
• auto: Parse numbers containing e as xs:double, numbers containing . as xs:decimal, and numbers containing neither as xs:integer

[XQuery] The 'member' keyword is still present on ForMemberBinding

The latest editor's draft (13 January 2021) moves the member keyword to a new ForMemberClause symbol:

ForMemberClause           ::=          "for" "member" ForMemberBinding ("," ForMemberBinding)*

With this change, the ForMemberBinding syntax has retained the optional member keyword from the previous change to ForBinding:

ForMemberBinding          ::=          "member"? "$" VarName TypeDeclaration? PositionalVar? "in" ExprSingle

This means that for member member ... and for member $a in [], member $b in [] ... are valid with the current grammar.

The ForMemberBinding grammar should be:

ForMemberBinding          ::=          "$" VarName TypeDeclaration? PositionalVar? "in" ExprSingle

[FO] fn:replace no longer has the 3 an 4 argument variants

The signature for fn:replace in FO 4.0 [1] only has the new 5 argument variant, whereas FO 3.1 has 3 and 4 argument variants.

1. https://qt4cg.org/branch/master/xpath-functions-40/Overview-diff.html#func-replace
2. https://www.w3.org/TR/xpath-functions-31/#func-replace

[XPath] [XQuery] Allow argument placeholders on keyword arguments

This would allow a user to name the arguments that are used as placeholders, making the code more readable. For example:

let $pow2 := math:pow(2, y: ?)

Syntax

This proposal would change KeywordArgument from:

KeywordArgument ::= NCName  ":="  ExprSingle

to:

KeywordArgument ::= NCName  ":="  Argument

or (using the proposed : syntax) to:

KeywordArgument ::= NCName  ":"  Argument

Semantics

A function call with N argument placeholders will create an N-arity function. The order of the argument placeholders correspond to the order of the parameters in that new function. Those parameters map to the corresponding parameter in the target (partially applied) function, which can be in a different order, or bind to keys in an options map (in the case of functions like fn:serialize). For example:

math:pow(y: ?, x: ?)

would create a function that calculates y^x instead of x^y as the arguments are reversed.

Support sequence, array, and map destructuring declarations

Given a function that returns a sequence, array, or map of a fixed length or structure, it would be useful to extract those values in a destructuring declaration like can be done in other languages (such as JavaScript, Kotlin, C++, and Python). For example:

let $(sin, cos) := sincos(math:pi()) (: sequence :)
let $[x, y, z] := camera-angle() (: array :)
let ${r, i} := complex(1, 2) (: map :)

These would be equivalent to:

let $ret := sincos(math:pi()), $sin := $ret[1], $cos := $ret[2] (: sequence :)
let $ret := camera-angle(), $x := $ret?(1), $y := $ret?(2), $z := $ret?(3) (: array :)
let $ret := complex(1 ,2), $r := $ret?r, $i := $ret?i (: map :)

It should be possible to define the type of a component and/or the whole construct:

let $(sin as xs:float, cos) as xs:float* := sincos(math:pi()) (: sequence :)

For maps, it would also be useful to rename the components, such as:

let ${re := r, im as xs:double := i} := complex(1, 2) (: map :)

It should also be possible to capture any left-over items in the sequence/array/map, for example:

let $(headings, rows) := load-csv("test.csv")

A destructuring declaration should be usable anywhere a variable binding can be defined.

It should not be an error to use the same variable name twice. This supports conventions such as using _ for unused values. For example:

let $[_, y, _] := camera-angle()

[FO] fn:filter with a function returning empty sequence

filter has signature:

fn:filter($seq as item()*, $f as function(item()) as xs:boolean) as item()*

It could easily be changed to:

fn:filter($seq as item()*, $f as function(item()) as xs:boolean?) as item()*

with () meaning false().

Then it could be used with a map to pick elements from the seq. E.g.

filter(1 to 10, map {3: true(), 4: true()}) 

~> (3, 4)

filter(("a", "b", "b", "c", "d"), map {"b": true(), "d": true()}) 

~> ("b", "b", "d")

Same for array:filter

[FO] fn:namespace-uri-for-prefix no longer supports passing a prefix by string

The type signature of the $prefix variable has changed from xs:string? to union(xs:NCName, enum(''))?. This means that passing a prefix like "fn" will no longer work as it is not an xs:NCName and is not a zero-length string (enum('')).

Note: The only other affected function is the new fn:in-scope-namespaces method. It would be useful in some cases to be able to pass the value as an xs:string (e.g. "fn") without having to cast the value.

fn:function-annotations (Allow support for user-defined annotations)

Requirements/Use Cases

1. It should be possible for a library or application written in XQuery to define, access, and use custom annotations without relying on vendor extensions.
2. It should be possible for a processor or editor/IDE to check and verify user-defined annotation usage (can an annotation be used multiple times; what argument values/types are valid) to provide better validation for custom annotations.
3. It should be possible to access the details of the annotations on a function (name, argument values) so tools like xqDoc can read and format annotations without using vendor extensions.

Annotation Declarations

1. An annotation declaration is a function declared using the %annotation annotation.
2. Annotation parameters must be a SequenceType that has an ItemType which is either a) a subtype of union(xs:string, xs:double, xs:float, xs:decimal)*, or b) item() (to denote any permitted literal value). Note: This is because annotation values are restricted to literal values. Note: It is therefore possible to use an EnumerationType to define a set of allowed string values (e.g. "yes" and "no").
3. If a function annotation name matches an annotation declaration in the statically-known functions using the function name resolver, it is verified using the same rules for static function calls; if there are no matching annotation declarations, then no error is issued, although an implementation may choose to issue a warning.

Annotation Tests

1. A new AnnotationTest (annotation(...)) SequenceType is provided with the same structure and semantics as a FunctionTest.
2. AnnotationTest parameters must be a SequenceType that has an ItemType which is either a) a subtype of union(xs:string, xs:double, xs:float, xs:decimal)*, or b) item() (to denote any permitted literal value). Note: This is because annotation values are restricted to literal values. Note: It is therefore possible to use an EnumerationType to define a set of allowed string values (e.g. "yes" and "no").

fn:annotations

fn:annotations($f as function(*)) as annotation(*)*

Returns all the annotations on the function.

The annotation signature will match the signature of the annotation, so %a(1, 2) will have the annotation(xs:integer, xs:integer) annotation type.

If a function has multiple annotations with the same signature (e.g. %values(1) %count(2)) then two annotations will be returned with the same signature, referring to the different annotations.

fn:annotation

fn:annotation($f as function(*), $name as xs:QName) as annotation(*)*

Returns all the annotations on the function with the name $name.

The annotation signature will match the signature of the annotation, so %a(1, 2) will have the annotation(xs:integer, xs:integer) annotation type.

If a function has multiple annotations with the same name and signature (e.g. %values(1) %values(2)) then two annotations will be returned with the same signature, referring to the different annotations.

fn:annotation-name

fn:annotation-name($annotation as annotation(*)) as xs:QName

Returns the name of the annotation.

fn:annotation-arguments

fn:annotation-arguments($annotation as annotation(*)) as array(*)

Returns the arguments passed to the associated annotation.

New reserved function names

In section A.3 (Reserved Function Names) of the XPath 4.0 and XQuery 4.0 draft specifications, the tuple and union keywords have been added.

I don't think these (or more accurately, the keywords for the new item type constructs) are needed, as they cannot currently be used in places where functions can. Only KindTests can appear as path expressions, and the only other reserved names should be for other types of expression (if, switch, typeswitch, etc.).

From a similar reasoning, the following keywords should be able to be removed from the list as they are also only ItemTypes/SequenceTypes:

1. array
2. empty-sequence
3. map

Kind regards,
Reece

Proposal to introduce the set datatype in XPath 4

It is high time that we come up with a set type in XPath. We actually have to deal all the time with sets (not just node-sets, but sets of any-type values), and it is painful to read in the spec how two maps are compared for equality when explaing fn:deep-equal():

"If $i1 and $i2 are both ·maps·, the result is true if and only if all the following conditions apply:

Both maps have the same number of entries.

For every entry in the first map, there is an entry in the second map that:

has the ·same key· (note that the collation is not used when comparing keys), and

has the same associated value (compared using the fn:deep-equal function, under the collation supplied in the original call to fn:deep-equal)."

When if we had the set type the above would simply say:

"If $i1 and $i2 are both ·maps·, the result is true if and only if the sets of their keys are equal, and the corresponding values for each key in the two maps are deep-equal."

I propose that starting with XPath 4.0 we introduce the set type and define set equality, the union ( | ), intersection (intersect) and set difference (except) not only for node-sets but for sets of any-typed values.

Then we can have a function: to-set($collection as item()*) as set, which would produce a set (of the distinct values) of any collection-typed argument supplied to it: sequence(its distinct values) , map (a set of its entries), array (a set of its members).

This makes fn:distinct-values() almost unnecessary.

We will no longer have to explain in a "Remarks" section that the result of a function is "unordered" or that its order is "implementation-defined" -- just by making this function return a set.

How can almost all major programming languages (not even speaking of SQL), such as C#, Python and Java have a set data type / interface, but even in XPath version 4 we still have to describe it in a free language narrative?

Thanks,
Dimitre

Highlight EBNF grammar differences in the diff versions of the specs

The diff versions of the specs highlight what has changed in the wording of the specifications, which is useful, but don't highlight the differences in the EBNF grammar. This makes it harder for implements and tool vendors to work out what has changed in the grammar to then know what changes need to be made to the lexer and parser in order to support the new constructs.