Data Aggregation CS03 Release Candidate (#71)

* ODATA-1442 - Update Aggregation ABNF with changes from previous issues (#35) * ODATA-1384: Node value property path in hierarchical transformations (#72) * ODATA-1468: Paths in aggregate in analogy with groupby (#75) * ODATA-1391 - Enable usage of $this and $root in filter transformation (#34) * ODATA-1533: Generalize 4th parameter of ancestors/descendants (#80) * countdistinct and custom aggregation methods can aggregate non-primitive values (#81) * `from` multiple dimensions (#82) * ODATA-1532: OData-Aggr WD05 sections 3.10.2.2 and 3.22 (#79) * Walkthrough OData-Aggr 3.2 (#85) * Incomplete searchExpr in $apply=search (#84) * Allow aggregation with paths ending in a type-cast (#91) * $count after single-valued primitive segment (#92) * Abolish transformnested (#95) * Allow lambda variables in aggregate function (#93) * "@odata.context": "$metadata#EntitySet(@Core.AnyStructure)" (#94) * Optional parameters for rolluprecursive (#86) * Allow `commonExpr` in aggregate expressions of type 2 (#98) * Remove optional parameter from `ancestors` and `descendants` (#100)
oasis-tcs · Oct 18, 2023 · 24234aa · 24234aa
1 parent fc73caa
commit 24234aa
Show file tree

Hide file tree

Showing 5 changed files with 1,643 additions and 697 deletions.
diff --git a/.github/workflows/nodejs.yml b/.github/workflows/nodejs.yml
@@ -15,7 +15,7 @@ jobs:
 
     strategy:
       matrix:
-        node-version: [18.x,20.x]
+        node-version: [20.x]
 
     steps:
       # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it

diff --git a/abnf/odata-aggregation-abnf.txt b/abnf/odata-aggregation-abnf.txt
@@ -1,57 +1,56 @@
 ;------------------------------------------------------------------------------
 ; OData Aggregation ABNF Construction Rules Version 4.0
 ;------------------------------------------------------------------------------
-; 04 November 2015
-;
-; Latest version: https://github.com/oasis-tcs/odata-abnf/blob/main/abnf/odata-aggregation-abnf.txt
+; 19 September 2023
 ;------------------------------------------------------------------------------
 ;
 ; Technical Committee:
 ;   OASIS Open Data Protocol (OData) TC
 ;   https://www.oasis-open.org/committees/odata
 ;
 ; Chairs:
-;   - Barbara Hartel (barbara.hartel@sap.com), SAP AG
-;   - Ram Jeyaraman (Ram.Jeyaraman@microsoft.com), Microsoft
+;   - Ralf Handl (ralf.handl@sap.com), SAP SE
+;   - Michael Pizzo (mikep@microsoft.com), Microsoft
 ;
 ; Editors:
-;   - Ralf Handl ([email protected]), SAP AG
+;   - Ralf Handl ([email protected]), SAP SE
 ;   - Hubert Heijkers ([email protected]), IBM
-;   - Gerald Krause ([email protected]), SAP AG
+;   - Gerald Krause ([email protected]), SAP SE
 ;   - Michael Pizzo ([email protected]), Microsoft
-;   - Martin Zurmuehl ([email protected]), SAP AG
+;   - Martin Zurmuehl ([email protected]), SAP SE
+;   - Heiko Theissen ([email protected]), SAP SE
 ;
 ; Additional artifacts: 
 ;   This grammar is one component of a Work Product which consists of:
 ;   - OData Extension for Data Aggregation Version 4.0
-;   - OData Aggregation ABNF Construction Rules Version 4.0
-;   - OData Aggregation ABNF Test Cases
 ;   - OData Aggregation Vocabulary
+;   - OData Aggregation ABNF Construction Rules Version 4.0 (this document)
+;   - OData Aggregation ABNF Test Cases Version 4.0
 ;
 ; Related work:
 ;   This specification is related to:
-;   - OData Version 4.0 Part 1: Protocol
-;   - OData Version 4.0 Part 2: URL Conventions
-;   - OData Version 4.0 Part 3: CSDL
-;   - OData ABNF Construction Rules Version 4.0
-;   - OData Core Vocabulary
-;   - OData Measures Vocabulary
-;   - OData JSON Format Version 4.0
+;   - OData Version 4.01 Part 1: Protocol
+;   - OData Version 4.01 Part 2: URL Conventions
+;   - OData ABNF Construction Rules Version 4.01
+;   - OData ABNF Test Cases Version 4.01
+;   - OData Common Schema Definition Language (CSDL) JSON Representation Version 4.01
+;   - OData Common Schema Definition Language (CSDL) XML Representation Version 4.01
+;   - OData JSON Format Version 4.01
 ;   This specification replaces or supersedes:
 ;   - None
 ;
 ; Declared XML namespaces:
 ;   - None
 ;
 ; Abstract:
-;   This specification adds basic grouping and aggregation functionality (e.g.
+;   This specification adds basic grouping and aggregation functionality (such as
 ;   sum, min, and max) to the Open Data Protocol (OData) without changing any
 ;   of the base principles of OData.
 ;
 ; Overview:
 ;   This grammar uses the ABNF defined in RFC5234 and RFC7405. 
 ;
-;   It extends the OData ABNF Construction Rules Version 4.0
+;   It extends the OData ABNF Construction Rules Version 4.01
 ;
 ; Contents:
 ;   1. New alternatives for OData ABNF Construction Rules
@@ -70,8 +69,11 @@ expandOption =/ apply
 
 boolMethodCallExpr =/ isdefinedExpr
 
-primitiveProperty =/ expressionAlias / customAggregate
+primitiveProperty =/ customAggregate
+
+firstMemberExpr =/ currCollectionExpr
 
+collectionPathExpr =/ %s"/aggregate" OPEN BWS aggregateFunctionExpr BWS CLOSE
 
 ;------------------------------------------------------------------------------
 ; 2. System Query Option $apply
@@ -80,96 +82,180 @@ primitiveProperty =/ expressionAlias / customAggregate
 apply      = ( "$apply" / "apply" ) EQ applyExpr
 applyExpr  = applyTrafo *( "/" applyTrafo )
 applyTrafo = aggregateTrafo
-           / bottomcountTrafo
-           / bottompercentTrafo
-           / bottomsumTrafo
            / computeTrafo
            / concatTrafo
-           / expandTrafo
-           / filterTrafo
            / groupbyTrafo
-           / identityTrafo
-           / searchTrafo
-           / topcountTrafo
-           / toppercentTrafo
-           / topsumTrafo
-           / customFunction
-
-aggregateTrafo  = %s"aggregate" OPEN BWS aggregateItem *( BWS COMMA BWS aggregateItem ) BWS CLOSE
-aggregateItem   = %s"$count" asAlias 
-                / aggregateExpr
-aggregateExpr   = commonExpr aggregateWith [ aggregateFrom ] asAlias
-                / pathPrefix primitiveProperty aggregateWith [ aggregateFrom ] asAlias
-                / pathPrefix customAggregate [ customFrom asAlias ]
-                / pathPrefix pathSegment OPEN aggregateExpr CLOSE 
-aggregateWith   = RWS %s"with" RWS aggregateMethod
-aggregateFrom   = RWS %s"from" RWS groupingProperty aggregateWith [ aggregateFrom ]
-customFrom      = RWS %s"from" RWS groupingProperty [ aggregateWith ] [ customFrom ]
-aggregateMethod = %s"sum"
-                / %s"min"
-                / %s"max"
-                / %s"average"
-                / %s"countdistinct"
-                / namespace "." odataIdentifier
+           / joinTrafo
+           / nestTrafo
+           / outerjoinTrafo
+           / preservingTrafo
+preservingTrafo = bottomcountTrafo
+                / bottompercentTrafo
+                / bottomsumTrafo
+                / filterTrafo
+                / identityTrafo
+                / orderbyTrafo
+                / searchTrafo
+                / skipTrafo
+                / topTrafo
+                / topcountTrafo
+                / toppercentTrafo
+                / topsumTrafo
+                / ancestorsTrafo
+                / descendantsTrafo
+                / traverseTrafo
+                / customFunction ; custom functions could be preserving, hence are allowed in preservingTrafos
+preservingTrafos = preservingTrafo *( "/" preservingTrafo )
+
+aggregateTrafo   = %s"aggregate" OPEN BWS aggregateExpr *( BWS COMMA BWS aggregateExpr ) BWS CLOSE
+aggregateExpr    = ( aggrPathPrefix / aggrCastPath ) nonprimAggWith [ aggregateFrom ] asAlias
+                 / aggregatableExpW [ aggregateFrom ] asAlias
+                 / aggregateCount [ aggregateFrom ] asAlias
+                 / aggregateCustom [ [ customFrom ] asAlias ]
+aggregatableExpr = commonExpr ; resulting in an aggregatable value
+aggregatableExpW = aggregatableExpr aggregateWith
+                 / [ aggrCastPath "/" ] aggrPrimPath aggregateWith
+aggrPathPrefix   = [ aggrCastPath "/" ] aggrPropPath
+aggregateWith    = RWS %s"with" RWS aggregateMethod
+nonprimAggWith   = RWS %s"with" RWS nonprimAggMethod
+aggregateFrom    = RWS %s"from" RWS groupingProperties aggregateWith [ aggregateFrom ]
+customFrom       = RWS %s"from" RWS groupingProperties [ aggregateWith ] [ customFrom ]
+aggregateMethod  = %s"sum"
+                 / %s"min"
+                 / %s"max"
+                 / %s"average"
+                 / nonprimAggMethod
+nonprimAggMethod = %s"countdistinct"
+                 / namespace "." odataIdentifier ; custom aggregation methods may work on non-primitive values
+aggregateCount   = %s"$count"
+                 / [ aggrCastPath "/" ] aggrPrimPath count
+                 / ( aggrPathPrefix / aggrCastPath ) count
+
+aggregateCustom = [ ( aggrPathPrefix / aggrCastPath ) "/" ] customAggregate
 
 asAlias         = RWS %s"as" RWS expressionAlias
 expressionAlias = odataIdentifier
 
 customAggregate = odataIdentifier
 
-groupingProperty = pathPrefix
-                   ( entityNavigationProperty [ "/" qualifiedEntityTypeName ]
-                   / primitiveProperty 
-                   / complexProperty
-                   )
-pathPrefix       = [ ( qualifiedEntityTypeName / qualifiedComplexTypeName ) "/" ] *( pathSegment "/" )
-pathSegment      = ( complexProperty / complexColProperty ) [ "/" qualifiedComplexTypeName ]
-                 / navigationProperty [ "/" qualifiedEntityTypeName  ]
-
-computeTrafo = %s"compute" OPEN BWS computeExpr *( BWS COMMA BWS computeExpr ) BWS CLOSE
-computeExpr  = commonExpr asAlias             
+; Three flavors of data aggregation paths are defined now:
+; - one for use in aggregate, whose segments can be single- or collection-valued (rules with prefix aggr)
+; - one for use in groupby, whose segments must be single-valued (rules with prefix sngl)
+; - one for use in addnested, without entity-valued segments (rule with prefix nest)
+; Term casts are not allowed in data aggregation paths.
+aggrPropStep = ( complexProperty / complexColProperty / entityNavigationProperty / entityColNavigationProperty )
+               [ "/" aggrCastPath ]
+aggrPropPath = aggrPropStep [ "/" aggrPropPath ]
+aggrPrimPath = aggrPropStep "/" aggrPrimPath
+             / primitiveProperty / primitiveColProperty / streamProperty
+aggrCastPath = optionallyQualifiedComplexTypeName / optionallyQualifiedEntityTypeName
+
+nestPropPath = ( complexProperty / complexColProperty ) [ [ "/" optionallyQualifiedComplexTypeName ] "/" nestPropPath ]
+
+snglPropPath = ( complexProperty / entityNavigationProperty ) [ [ "/" aggrCastPath ] "/" snglPropPath ]
+snglPrimPath = ( complexProperty / entityNavigationProperty )   [ "/" aggrCastPath ] "/" snglPrimPath
+             / primitiveProperty / streamProperty
+
+groupingProperty   = [ aggrCastPath "/" ] ( snglPrimPath / snglPropPath )
+groupingProperties = groupingProperty *( BWS COMMA BWS groupingProperty )
+
+; Expressions evaluable on a collection
+collectionExpr     = commonExpr ; but where every firstMemberExpr must be a currCollectionExpr
+currCollectionExpr = %s"$these" collectionPathExpr
+
+computeTrafo    = %s"compute" OPEN BWS computeExpr *( BWS COMMA BWS computeExpr ) BWS CLOSE
+computeExpr     = commonExpr asAlias             
 
-bottomcountTrafo   = %s"bottomcount"   OPEN BWS commonExpr BWS COMMA BWS commonExpr BWS CLOSE
-bottompercentTrafo = %s"bottompercent" OPEN BWS commonExpr BWS COMMA BWS commonExpr BWS CLOSE
-bottomsumTrafo     = %s"bottomsum"     OPEN BWS commonExpr BWS COMMA BWS commonExpr BWS CLOSE
+bottomcountTrafo   = %s"bottomcount"   OPEN BWS collectionExpr BWS COMMA BWS commonExpr BWS CLOSE
+bottompercentTrafo = %s"bottompercent" OPEN BWS collectionExpr BWS COMMA BWS commonExpr BWS CLOSE
+bottomsumTrafo     = %s"bottomsum"     OPEN BWS collectionExpr BWS COMMA BWS commonExpr BWS CLOSE
+
+concatTrafo     = %s"concat" OPEN BWS applyExpr 1*( BWS COMMA BWS applyExpr ) BWS CLOSE
+
+nestTrafo       = %s"nest" OPEN BWS nestApplyExpr BWS CLOSE
+                / %s"addnested" OPEN BWS nestPath BWS COMMA BWS nestApplyExpr BWS CLOSE
+nestPath        = [ aggrCastPath "/" ]
+                  ( [ nestPropPath "/" ] navigationProperty [ "/" optionallyQualifiedEntityTypeName ]
+                  / nestPropPath
+                  )
+nestApplyExpr   = applyExpr asAlias *( BWS COMMA BWS applyExpr asAlias )
+
+joinTrafo       = %s"join" OPEN BWS joinProperty asAlias [ BWS COMMA BWS applyExpr ] BWS CLOSE
+outerjoinTrafo  = %s"outerjoin" OPEN BWS joinProperty asAlias [ BWS COMMA BWS applyExpr ] BWS CLOSE
+joinProperty    = ( complexColProperty 
+                  / complexAnnotationInQuery ; must be collection-valued 
+                  / entityColNavigationProperty [ "/" optionallyQualifiedEntityTypeName ]
+                  / entityAnnotationInQuery  ; must be collection-valued
+                  )
+
+ancestorsTrafo  = %s"ancestors" OPEN BWS
+                  recHierReference BWS
+                  COMMA BWS preservingTrafos BWS
+                  [ COMMA BWS 1*DIGIT BWS ]
+                  [ COMMA BWS %s"keep start" BWS ]
+                  CLOSE
+
+descendantsTrafo = %s"descendants" OPEN BWS
+                  recHierReference BWS
+                  COMMA BWS preservingTrafos BWS
+                  [ COMMA BWS 1*DIGIT BWS ]
+                  [ COMMA BWS %s"keep start" BWS ]
+                  CLOSE
+
+traverseTrafo   = %s"traverse" OPEN BWS
+                  recHierReference BWS
+                  COMMA BWS ( %s"preorder" / %s"postorder" ) BWS
+                  [ COMMA BWS preservingTrafos BWS ]
+                  [ COMMA BWS orderbyItem *( BWS COMMA BWS orderbyItem ) BWS ]
+                  CLOSE
+
+recHierReference = rootExpr ; must have type Collection(Edm.EntityType)
+                   BWS COMMA BWS recHierQualifier
+                   BWS COMMA BWS recHierPropertyPath
+recHierQualifier = odataIdentifier
+recHierPropertyPath = [ aggrCastPath "/" ] aggrPrimPath
+
+filterTrafo     = %s"filter" OPEN BWS boolCommonExpr BWS CLOSE
+
+searchTrafo     = %s"search" OPEN BWS ( searchExpr / searchExpr-incomplete ) BWS CLOSE
 
-concatTrafo = %s"concat" OPEN BWS applyExpr 1*( BWS COMMA BWS applyExpr ) BWS CLOSE
+groupbyTrafo    = %s"groupby" OPEN BWS groupbyList [ BWS COMMA BWS applyExpr ] BWS CLOSE
+groupbyList     = OPEN BWS groupbyElement *( BWS COMMA BWS groupbyElement ) BWS CLOSE
+groupbyElement  = groupingProperty / rollupLevels / rollupRecursive
 
-expandTrafo = %s"expand" OPEN BWS expandNavPath BWS COMMA BWS
-              ( expandTrafo *( BWS COMMA BWS expandTrafo )
-              / filterTrafo *( BWS COMMA BWS expandTrafo )
-              ) BWS CLOSE
-expandNavPath = [ ( qualifiedEntityTypeName / qualifiedComplexTypeName ) "/" ] 
-                *( ( complexProperty / complexColProperty ) "/" [ qualifiedComplexTypeName "/" ] )
-                navigationProperty [ "/" qualifiedEntityTypeName ]
+rollupLevels      = %s"rollup" OPEN BWS ( rollupUnnamedHier / rollupNamedHier ) BWS CLOSE
+rollupRecursive   = %s"rolluprecursive" OPEN BWS
+                    recHierReference BWS
+                    [ COMMA BWS preservingTrafos BWS ]
+                    CLOSE
 
-filterTrafo = %s"filter" OPEN BWS boolCommonExpr BWS CLOSE
+rollupUnnamedHier = groupingProperty 1*( BWS COMMA BWS groupingProperty )
+rollupNamedHier   = odataIdentifier ; qualifier of Aggregation.LeveledHierarchy annotation
 
-searchTrafo = %s"search" OPEN BWS searchExpr BWS CLOSE
+identityTrafo   = %s"identity"
 
-groupbyTrafo   = %s"groupby" OPEN BWS groupbyList [ BWS COMMA BWS applyExpr ] BWS CLOSE
-groupbyList    = OPEN BWS groupbyElement *( BWS COMMA BWS groupbyElement ) BWS CLOSE
-groupbyElement = groupingProperty / rollupSpec
-rollupSpec     = %s"rollup" OPEN BWS 
-                  ( %s"$all" / groupingProperty )
-                  1*( BWS COMMA BWS groupingProperty )
-                  BWS CLOSE
+topcountTrafo   = %s"topcount"   OPEN BWS collectionExpr BWS COMMA BWS commonExpr BWS CLOSE
+toppercentTrafo = %s"toppercent" OPEN BWS collectionExpr BWS COMMA BWS commonExpr BWS CLOSE
+topsumTrafo     = %s"topsum"     OPEN BWS collectionExpr BWS COMMA BWS commonExpr BWS CLOSE
 
-identityTrafo = %s"identity"
+topTrafo        = %s"top" OPEN BWS 1*DIGIT BWS CLOSE
+skipTrafo       = %s"skip" OPEN BWS 1*DIGIT BWS CLOSE
 
-topcountTrafo   = %s"topcount"   OPEN BWS commonExpr BWS COMMA BWS commonExpr BWS CLOSE
-toppercentTrafo = %s"toppercent" OPEN BWS commonExpr BWS COMMA BWS commonExpr BWS CLOSE
-topsumTrafo     = %s"topsum"     OPEN BWS commonExpr BWS COMMA BWS commonExpr BWS CLOSE
+orderbyTrafo    = %s"orderby" OPEN orderbyItem *( BWS COMMA BWS orderbyItem ) CLOSE
 
-customFunction = namespace "." ( entityColFunction / complexColFunction / primitiveColFunction ) functionExprParameters
+customFunction  = namespace "." ( entityColFunction / complexColFunction / primitiveColFunction ) functionExprParameters
 
 
 ;------------------------------------------------------------------------------
-; 3. Extensions to $filter
+; 3. New functions
 ;------------------------------------------------------------------------------
 
 isdefinedExpr = %s"isdefined" OPEN BWS ( firstMemberExpr ) BWS CLOSE
 
+aggregateFunctionExpr = aggregatableExpW [ aggregateFrom ]
+                      / aggrPathPrefix nonprimAggWith [ aggregateFrom ]
+                      / aggregateCount [ aggregateFrom ]
+                      / aggregateCustom [ customFrom ]
 
 ;------------------------------------------------------------------------------
 ; End of odata-aggregation-abnf