Transforming DisplayStyles with huge amounts of element references has a big impact on memory and performance

[ViliusRuskys]

The test iModel I tested it on had 210 DisplayStyle3D Elements. Each of those had more than 250k element references in jsonProps.excludedElements.
Cloning each DisplayStyle takes close to 1minute of processing time (later on it takes more than that because of a different issue).

Because transformer processes definition elements first, all those references are not found in the target iModel at the moment the DisplayStyle is processed, so remapping them is impossible and the DisplayStyle is inserted as a "partial element". Currently transformer is very inefficient in storing the partially committed entity references and in this specific scenario it amounts to a large performance hit and requires a lot of memory.

[MichaelBelousov]

Saw this by email...

Another possible approach which the experimental branch I made a long time ago did, was insert an "empty" element for each element that was missing references (which the current branch does now) but update at the end instead of when the reference becomes valid.

Then you change the $\approx O(n^2)$ memory space complexity with $O(n)$ since you only need to store which elements to update, not when to update them (stored as the references it's waiting for).

This is probably both faster and simpler for multiple reasons.

[MichaelBelousov]

I actually recorded this proposal already in the last bullet point here:
#83 (comment)