Lionel Smith-Gordon - ART Phase 3 (finos#3127)

* 3 - ART Phase 3

Initial commit (includes P2 changes)

* Correction to Asset Payout

and bring forward P2 changes

* Refactoring of Index and FRO

FloatingRateOption collapsed into new Index structure

* FX updates for P3

FX updates for P3

* PriceQuantity refactoring

PriceQuantity refactoring

* Rosetta syntax fixes

* Fix compilation issues

* Make ingestion run

* Update FpML 5.13 rates mappings

* Inflation swaps and ancillaryParty fix

* Map bond options

* Update mappings

* Update mappings for equity swaps

* Repo and bond - bond execution - initial commit

* Initial merge to v2

* Fix compilation issues

* Fix mapper packages

* Fix interest-rate-derivatives

* Fix inflation-swaps

* Fix bond-options

* Fix equity-swaps

* Fix equity-options

* Fix equity-swaps basket duplicate mappings

* Reference fixes

* Temp testing change

* Update DSL to version 9.15.1 and bundle to 11.16.2

* DSL update

* Update bundle to version 11.17.1

* Clean up expectations

* Update bundle version

* Fix FIS mappings

* Fix pom

* Update interest-rate-derivatives with reference fix

* Update inflation-swaps with reference fix

* Update bond-options with reference fix

* Update equity-swaps with reference fix

* Update equity-options with reference fix

* Tidy up

* Map correlation swaps

* Map variance-swaps

* Mapping fix

* Add comments to highlight model changes by HH

* Map dividend swaps

* Map volatility swaps

* Map commodities

* Map credit

* Map credit

* Map credit

* Map credit

* Fix assetClass

* Fix assetClass

* Fix assetClass

* Fix assetClass for FRAs

* Fix assetClass

* Tidy up

* Map FX (not finished)

* Update repo and bonds

* Update repo and bonds - bond execution

* Add address annotation

* Add address/location annotations and update interest-rate-derivatives expectations

* Update inflation-swaps expectations

* Update bond options expectations

* Update equity-swaps expectations

* FX

* Update equity-options expectations

* Update correlation-swaps expectations

* Update equity swap and equity options

* Update variance swaps

* Update dividend swaps

* Update volatility swaps

* Update commodity

* Update credit

* Update expectations

* Update fx

* Tidy up

* Use CreditIndex

* Tidy up

* Add metadata address

* Update FpML 5.13 processes

* Tidy up test class

* Comment out some conditions

* FloatingRateIndex identifier

* Fix zc perUnitOf

* Fix settlementType

* Comment out condition PriceQuantity.SingleCurrencyQuantity

* Fix assetIdentifier for CD indexes

* Fix assetIdentifier indexName scheme

* Comment out condition PriceQuantityNonCurrencyQuantities

* Remove duplicate mappings

* Update basketConsitituent type

* Fix DividendPayoutRatio references

* Bundle 11.18.3

* pricePerOption

* pricePerOption units

* tidy up

* Fix FX option and equity option qualification functions

* ART P3 with mapping

ART P3 with mapping

* Fix capacity unit mapping

* Refactored OptionUnderlier to Underlier

Refactored OptionUnderlier to Underlier

* Completed Underlier refactoring

Completed Underlier refactoring

* Fix FX option quantity

* Updated PerformancePayout

Updated PerformancePayout

* Update mappers

* Update expectations

* Update mappings and expectations for performancePayout underlier

* Map optionType

* Correlation swap price units

* Remove unused mappers

* Clean up mapping stats

* Fix typo

* Fix correlation price units

* Remove debug logging

* Fix FX variance and volatility

* FX variance and volatility - asset class

* FX variance and volatility - qualification

* Set optionType for swaptions

* Set optionType for swaptions

* Update expectations for processes

* Remove unused metadata location annotations

* Fix repo and bond samples

* OptionPayout.optionType mapping - remove swaption mapping, relax cardinality, and add condition

* Update documentation snippets

* Update documentation snippets

* Fix Examples compilation issue

* Update FpML 5.15 incomplete processes

* Update FpML 5.13 incomplete products

* Update FpML 5.13 processes

* Fix Qualify_EquityOption_PriceReturnBasicPerformance_SingleName qualification

* Update expectations

* FpML 5.12 update expectations

* FpML 5.12 update expectations

* Update sec lending

* Update repo and bond

* Add testcase for block-execution-trade-state

* Collateral Position is a Product

SecLending qualification on a TransferableProduct in CollateralPortfolio

* Update FpML 5.10 expectations

* Fix bond forward mappings

* Fix bond forward mappings

* Fix merge issues

* Update sec lending

* Update bond and repo

* Update secLending

* Update secLending

* Update secLending

* Update secLending allocation and reallocation

* Update FpML 5.10 processes

* Update FpML 5.10 incomplete products

* Update FpML 5.10 incomplete processes

* Update ingestion expectations

* Refactor the financial product section

* Replace "flow" --> "transfer" as already used term

* Update ingestion expectations

* Update cdm-java expectations

* Update FIS expectations

* Update ORE expectations

* Comment out margin-schedule-func.rosetta

* Update CME submission

* Update sec lending

* More fixes for sec lending

* Update repo and bond

* Update repo and bond / sec lending

* Update expectations

* Fix template example

* Margin Asset Refactor

Refactored Margin functionality in line with ART

* Remove template functionality

* Clean up examples project

* Fix example

* Fix usage of only exists on choice types

* Fix validation error

* formatting

* ARTP3-RC

Feedback

* Updates based on Leo's feedback

Modelling updates

* Further review updates

Closing all items on PR3127

* ARTP3-RC

Ready for Hugo

* Fix qualification and update FpML 5.13 expectations

* Update expectations

* Fix CME mapping

* Fix CME mapping

* Fix broken link

* Update RELEASE.md

* Update bundle with C# fix

* Fix merge issues

---------

Co-authored-by: Lionel Smith-Gordon <[email protected]>
Co-authored-by: Simon Cockx <[email protected]>
Co-authored-by: lolabeis <[email protected]>
Co-authored-by: Lionel SG <[email protected]>

RELEASE.md

@@ -1,12 +1,91 @@
-# _Infrastructure - Dependency Update_
+# _Product Model - Asset Refactoring: Product, SettlementPayout, Underliers_
+
+WORK IN PROGRESS
+
+_Background_
+
+The Asset Refactoring initiative (see https://github.com/finos/common-domain-model/issues/2805) is seeking to improve the Product Model to address some long-standing issues and to ensure the continued extensibility to additional  financial products and markets.  A proposal is being developed - through a cross-industry Task Force - to implement this remodelling in the CDM.  
+
+This release includes the third tranche of changes (of three planned tranches in CDM 6) to implement the refactored model. It introduces some significant refactoring of the Product structure in to the model.
 
 _What is being released?_
 
-This release updates the DSL dependency.
+Product Refactoring:
+- This release completes the refactoring of the major financial product data types, that is `Asset`, `Observable`, and `Index`, and combines their use in a new structure for financial products.
+- The new data type `NonTransferableProduct` has replaced the former `ContractualProduct` and is the main "product" data type used on a trade; it appears as the attribute `product` on the data type `TradableProduct`.
+- On a `Trade`, all financial products should be composed into the `EconomicTerms` of a `NonTransferableProduct`.
+- *Separate PR but included here for completeness* The data type `payout` is now a choice construct with the consequence that references to specific instances of a specific payout should refer to the capitalised data type name rather than an attribute, for example `economicTerms -> payout -> performancePayout` becomes `economicTerms -> payout -> PerformancePayout`. This has large impact in terms of the number of changes in this PR.
+
+Underliers:
+- An `Underlier` represents the financial product that will be physically or cash settled.
+- Whereas all underliers were previously defined to use data type `product`, this has now been improved so that they can also be an `Observable` when the case warrants it.
+- `Underlier` is modelled as a choice data type, that is, it can either be an `Observable` or a `Product`.
+- A `Product` is also a choice type, either a `TransferableProduct` (a type of financial product which can be held or transferred, represented as an Asset with the addition of specific EconomicTerms), or a     NonTransferableProduct (a product that can be traded, as part of a TradableProduct, but cannot be transferred to others).
+
+Product and Trade Hierarchy:
+- The `Trade` data type now extends `TradableProduct`; this means that the latter is "hidden" in many uses in the CDM, eg in the graphical view, and one level within the hierarchy is removed when generating JSON.  This change has resulted to updates to 100s of occurences to path accesses within the model (particularly in the Event and Product functions).  For example, the previous access path `tradeState -> trade -> tradableProduct -> tradeLot` has become `tradeState -> trade -> tradeLot`. 
+- The financial product hierarchy has also changed so that the previous path `trade -> tradableProduct -> product -> contractualProduct -> economicTerms -> payout -> InterestRatePayout` has become `trade -> product -> economicTerms -> payout -> InterestRatePayout`.
+
+Product details:
+- `Security`
+  - Removed the attributes `economicTerms` and `ProductTaxonomy` because these are not appropriate for an `Asset` data type which is standardised. if `economicTerms` are required, these can be added by wrapping `security` into a `TransferableProduct`. The taxonomy of assets is explicitly defined in the type.
+- `Index`
+  - `Index` is now an `Observable` and the replaces the existing data type `rateOption`.
+  - The path `observable -> rateOption` has become `observable -> Index -> FloatingRateIndex`.
+  - Data type `CreditIndex` now extends `IndexBase`.
+
+ProductQualification:
+- Additional functions have been created to ease with the qualification process: `UnderlierQualification`. `ObservableQualification`.
+- Extensive refactoring has been made to the qualification functions to reflect the refactoring, albeit with no changes to the actual logic.
+- The qualification for options has been tidied up to be more readable and maintainable in the new model, without change to the functionality.
+- The introduction of `SettlementPayout` has been included in the qualification logic.
+- The qualification of Foreign Exchange transactions has been updated.
+- The securities financing qualification functions have been enhanced to differentiate repos and lending; the new functions are: `Qualify_RepurchaseAgreement`, `Qualify_BuySellBack`, and `Qualify_SecurityLending`.
+- A new enumerator data type has been added to support repo qualification: `RepoTypeEnum`.  The corresponding attribute `repoType` was added to `AssetPayout`.
 
-Version updates include:
-- DSL 9.19.0: support for `switch` operation on `choice` types. For further details see DSL release notes: https://github.com/finos/rune-dsl/releases/tag/9.19.0
+Payouts:
+- A new payout type has been created:  `SettlementPayout`
+- The existing `ForwardPayout` has been collapsed into `SettlementPayout`; the latter should be used whereever the former was previously.
+- *Separate PR but included here for completeness* The `Payout` data type has been refactored to be a `choice` and payouts now have multiple cardinality on `EconomicTerms`.  The conditions that validated the business logic on payouts has been moved to `EconomicTerms`.
+
+Event Model:
+- `Create_Exercise` has additional logic to support an option underlier that coan be an `Asset`, a `TransferableProduct` or a `NonTransferableProduct`.
+- `Create_Execution` now acts upon a more narrowly defined `NonTransferableProduct` rather than a generic product.
+- Additional functions have been created to support events using the new product model:  `Create_NonTransferableProduct`, `Create_TransferableProductFromAsset`, `Create_TransferableProductFromIndex`, `CheckTransferableProduct`, `CheckTradeNotTransferableProduct`, `CreateTradableProduct`.
+- The function `NewEquitySwapProduct` now creates a `NonTransferableProduct` not a generic product.
+- Event processing has been refactored to handle the new modeling of `TradableProduct`.
+- The unused data types `Affirmation` and `Confirmation` have been removed.
+
+Observable:
+- The attribute `Observable` has been removed from `ObservationTerms` where it created duplication.
+
+Collateral:
+- `AssetIdentifier` replaces `ProductIdentifier` on `AssetCriteria`.
+- The function `CheckEligibilityForProduct` now uses `TransferableProduct` not `Product`.
+
+Namespace re-alignments:
+- The following data objects have been moved to a more appropriate namespace:
+  - `enum PutCallEnum` to cdm.base.staticdata.asset
+  - `choice Index` to cdm.observable.asset
+  - `type IndexBase` to cdm.observable.asset
+  - `type PriceQuantity` to cdm.observable.asset
+- The function `InterestRateObservableCondition` has been moved to the cdm.observable.asset function namespace.
+
+Deprecated data types which have been removed:
+- `IndexReferenceInformation`: replaced with `Index`.
+
+Documentation updates:
+- Significant improvements have been made to the pages on Product Model, Event Model and Process Model.
+- As well as incorporating the direct changes that result from the refactoring, the pages have been restructured to improve the hierarchy, sections have been resequenced where this improves understanding, and "tips" have been added to highlight important definitions.
+- The use case sections on Collateral, Securities Lending and Repos have been updated to reflect the revised modeling.
 
 _Review directions_
 
-The changes can be reviewed in PR: [#3152](https://github.com/finos/common-domain-model/pull/3152)
+The changes can be reviewed in PR: [#3127](https://github.com/finos/common-domain-model/pull/3127)
+
+_Backward-incompatible changes_
+
+This release contains changes that are not backward-compatible:
+- TBA
+
+A full description of the backward-incompatible changes, and how persisted objects should be remapped, will be included in the release notes for the production release of CDM 6.

cdm-python/test/semantics/test_validation.py

@@ -11,7 +11,7 @@
 from cdm.product.template.TradableProduct import TradableProduct
 from cdm.product.template.Product import Product
 from cdm.product.template.TradeLot import TradeLot
-from cdm.product.common.settlement.PriceQuantity import PriceQuantity
+from cdm.observable.asset.PriceQuantity import PriceQuantity
 from cdm.base.staticdata.party.Party import Party
 from cdm.base.staticdata.party.PartyIdentifier import PartyIdentifier
 from cdm.base.staticdata.party.Counterparty import Counterparty