Feat/datalist runtime override

[chrismclarke]

PR Checklist

• PR title descriptive (can be used in release notes)

Description

Add support for overriding data_lists at runtime with override_target and override_condition fields, in the same way as templates

Main Changes

• Add backend code to support data_list overrides
• Create backend spec test for data_list overrides
• Create content test sheet for data_list overrides (or update existing if similar exist)
• Add frontend code to support data_list overrides
• Add frontend spec tests
• Copy local debug sheets to shared drive
• PR Demo
• (Blocked) Builds on fix: sync_local workflow #2041 - wait for merge

Additional changes

• Add new methods for dynamic content evaluation to be used outside of templating system
• Add new methods for identifying dynamic content at runtime (outside of templating system)
• Tidy logging code to distinguish file-based and console-based logging
• Improve logging test integration
• Fix QC check for duplicate flows names within a given flow type
• Remove deprecated script code

Review Notes

I've updated example_overrides sheet in debug repo to include an example_list_override template with 2 corresponding data lists (shown in demo below)

This will require a sync as parser methods have been updated, yarn workflow sync

Author Notes

Override functionality is still implemented within individual flow types at the back-end (with template and data_list both calling shared method to implement). This means that the functionality is not available for flows that have been generated via data_pipe or generator flows, nor flow types such as globals or tours . We may wish to consider making a low-priority to address this in the future if we want similar override system on all flow types (may not be required if most authoring to be focused around sheets and lists)

There is also a limitation currently, that the only dynamic expressions supported in conditions are @field notation. This is due to the fact all other syntaxes (@global, @local etc.) are coupled tightly with the templating system logic and would require larger refactor to make available. For now just the field logic has been duplicated

Dev Notes

In order to allow dynamic data evaluation outside of templating, a new appDataEvaluator service has been created. This currently only supports @field syntax and is mostly a duplicate of the template field service, but should form a base to extract additional methods in the future and eventually move all dynamic context handling out of the templating system. Service spec tests have also been created, and can be run directly via yarn ng test --include="src\app\shared\services\data\app-data-variable.service.spec.ts"

Additionally new methods have been added to identify a list of all dynamic variables used within a flow/sheet. Currently this is only used as a 2-part extract variables and evaluate method in the appDataEvaluator, however could be used in the future to extract more data during parsing (replace/combine with #1223) or to keep a list of realtime variable changes (alongside #1714).

Curiously in the example sheet the buttons don't align in the same way - likely due to some confusion from the ng:deep left selector (seems buttons by default align center but inner text aligns left and the 2 sets of rules can conflict in inconsistent ways - see output in demo video). Recommend to review

Git Issues

This addresses the first part of #1266

Screenshots/Videos

Example sheet - items that can be updated by setting data list dynamically (based on override condition)

Debug.App.webm

Example inputs and outputs for new method and tests to extract list of context variables

Example output logs generated from duplicate flows

[jfmcquade]

Looks good. I've made an issue for the strange button behaviour here: #2078

Great to have the extra tests, although I'm having issues running them:

When I run yarn ng test --include="src/app/shared/services/data/app-data-variable.service.spec.ts" (after running yarn install) I get the following error:

If I run just ng test then I get the following errors relating to some of the ODK components (I can't remember if we expect ng test to run successfully in general)

I don't have time to debug now, but am happy to approve

[chrismclarke]

Thanks for the review @jfmcquade , can confirm I'm also now seeing the error so can debug (guessing one of the incoming PRs changed some of the service mocking).

Would be nice to get the full ng test working (even if only partial coverage) so that we can include the tests against PRs, but probably a job for another day

[chrismclarke]

Ng testrunner should now be passing @jfmcquade
I've made a follow-up PR branch #2080 to start work on fixing tests more generally (low priority, although feel free to contribute to it if wanted)

[esmeetewinkel]

Functional test passed. I noticed two typos in the code, which I've suggested corrections for.

I'm understanding that this PR only closes the first part of the issue, and the second part will be a follow up PR, so I've edited the PR description so that the issue remains open.

[chrismclarke]

Thanks both, I've added a couple small additions to the existing overrides documentation and will merge once tests passing
86ef2fd

@esmeetewinkel - I think we can probably go ahead and close the issue as the second point of overriding at compile time should be covered by multiple data sources proposal in #2081
I've also included this case in the updated documentation