Research interaction between data, filter panel and modules

[kumamiao]

As teal and teal.devel have been separated to packages: teal, teal.code, teal.data, teal.logger, teal.slice, teal.transform & teal.widgets, it is worthwhile to consolidate the interaction between different packages.

An example case: teal.slice should only be slicing the data and not do any data manipulation. Once users filter out some factor levels, should there be dropping of the empty levels? Should the level dropping happen on the app level or each module level?

What done looks like: finish user interviews on user preferences, implement changes either on teal level, or teal modules level.

---

This item is a research issue on whole team framework. Issue parts are:

• meeting with app users to gather informations about how they are using our apps with focus on keys and data extract/data merge.
• Set the requirements for our packages and see how much we can simplify.
• Meet with all team members to gather opinions about teal.code, teal.data, teal.slice and teal.transform.
• Conduct sessions with core dev members to draw (design) packages classes with proposed changes.
• Reiterate to find optimal solution to all ideas and requirement.
• Present conclusions and plan (make issues) future changes.

Design doesn't have to be specific to the level of all methods and fields in the classes. Conclusion should contain relationships between packages/classes and general assumptions.

---

Issues

• nicer way to call a teal module outside of teal teal#624
• data_merge - remove dependency on FilteredData and chunks. teal.transform#61
• Change constructor of the FilteredData to not depend on TealData teal.slice#33
• Removing get_rcode dependency on chunks and FilteredData is a part of this Simplify Chunks - implement quosure #45
• Consider migration to dm teal.data#8
• https://github.com/insightsengineering/coredev-tasks/issues/256
• Design data extract and data merge #36 (milestone)
• https://github.com/insightsengineering/coredev-tasks/issues/363

[gogonzo]

Users feedback:

1. Global filter panel is very important for most of the studies.
2. In the filter-panel child datasets without duplicated columns comparing to parent is a good feature.
3. Experienced user avoids to use data_extract/data_merge and writes own module dedicated to specific problem where merge is more controllable. Our data_merge is unpredictable.
4. It's not obvious that any merge is happenning (even if the variables are taken from multiple datasets).
5. Merge between the same dataset is causing keys issue which is undebuggable.
6. Requirement to specify the same filters for the same dataset each time.
7. dataset_connector has no parent argument
8. FilterPanel takes too much space when setting many filters. It's hard to navigate.
9. Unable to just run the teal_module on defaults
10. From @danielinteractive - it's not expected that the teal_module(s) should be universal to the level that It could work with any data (for example tmg with MAE). Module developer is able to create a module which is able to work with certain data type. See for example tmh. Proposition of interactive transformation panel is interesting but it's not recommended for big MAE, where extracting and conversion to data.frame can be computationally expensive and hard to maintain.

Team conclusions:

• There is still ongoing discussion about Design data extract and data merge #36 which also determines shape of the teal_module(s). What we currently know is that:
  • data_extract_spec should be split on "select" and "filter" (interactive data transformation). This addresses (5) and (6)
  • merge is necessary when we allow "select" to be from multiple datasets. Merge in fact is inevitable, the question is who is responsible for merge and wether it should be automatic or with some interactivity options.
  • Another question is how to allow app developer specify own merge or data transformation - to address (3)
• Fix UI of the filter panel to be more user-friendly - to address (8) Move the add new filter next to the list of filters teal.slice#13
• teal::get_rcode shouldn't depend on chunks and FilteredData - it requires some work on chunks first Simplify Chunks - implement quosure #45
• teal_module shouldn't required FilteredData to allow nicer way to call a teal module outside of teal teal#624
• teal.transform should not depend on other classes data_merge - remove dependency on FilteredData and chunks. teal.transform#61
• teal.slice shouldn't depend on teal.data Change constructor of the FilteredData to not depend on TealData teal.slice#33
• make connectors be reusable https://github.com/insightsengineering/coredev-tasks/issues/256

[nikolas-burkoff]

Also note: https://github.com/insightsengineering/coredev-tasks/issues/256 - added into list above

[danielinteractive]

Thanks @gogonzo for adding my feedback above, one addition I would like to add here: for really high dimensional MAE we don't expect automatic ingestion in tmc, tmg, etc.
However for low dimensional MAE, meaning CDISC datasets being combined as list columns in the colData of an MAE, it would be potentially pretty useful to help app developers unwrapping these elegantly and then using them in the standard teal modules.
I recommend to reach out to Armen Karapetyan and team since they are working on closely related packages.

[gogonzo]

I'm happy to close this item in favor of:

1. API simplification #2
2. Design data extract and data merge #36

We reached the conclusion of having teal components being more universal and being able to be called by non-teal shiny apps. For this item (1) we already created necessary issues. After (1) is completed we plan to refactor data_merge/data_extract in another increment as doing both in the same is to much for the team capacity. Postponing data_merge gives us also a time to gid more into chevron , to find the way to enable teal for dm.