teal transformations API: operations approach

[m7pr]

This thread is related to #1216, however I started a new discussion as the other one is cluttered and I wanted to present a new approach in this thread.

• Intro
• API
• UI
• Operations Manager
• Ability to create custom operations through the app
  • Creation of operation
  • Creation of filter
• What will be removed
• What will need to be changed and implemented
• Pros and cons
• What to think about

Table of contents generated with markdown-toc

Intro

To distinguish between all other ideas I will call this approach operations approach and will use operations term on the actions called transformations in the other approaches. The goal is to create an API, with an idea for UI that will allow teal applications to utilize custom data transformations within modules. Ideally those transformations should be applied globally or locally (module specific), depending on the user's (app developer) needs. The order of the transformations should also be customizable by the app developer and ideally by app user. API should be straightforward and easy to grasp. Those are the goals I have in mind while presenting the below idea. I also keep in mind that I would like to introduce as least changes as possible (so if we can modify one package vs all of the packages, I'd opt for modifying one package) and would like to reuse as much components as we already have.

---

API

The idea that I present does not affect modules, but introduces new parameter in teal::init called operations. Operations takes a list of operations as an input. The order of operations matter. Below you can see possibility to

• set operations to be globally shared by all modules (filter1)
• ability to exclude some modules from globally shared operations (operation2)
• ability to have the same operation to be shared within multiple modules (operation2)
• ability to have the same operation that is reused in multiple modules but is not shared (operation3)
• ability to have operations that are module specific (operation3)

By shared I mean that changes on the view of one module are also visible for other modules that share the operation.

teal::init(
    data = data,
    modules = modules(
        module(label = "Module 1"),
        module(label = "Module 2"),
        module(label = "Module 3")
    ),
    operations = list(
        teal_slice( # applied to all modules
          filter1, 
          modules = "all"
        ), 
        teal_operation( # applied to Module 1 and Module 2 and shared between those modules
          operation1, 
          modules = c("Module 1", "Module 2")
        ), 
        teal_opeartion( # applied to all modules besides Modules 2 and is shared between modules (without Module 2)
          operation2, 
          exclude_modules = "Module 2"
        ), 
        teal_operation( # only applied to Module 1 - not shared
          operation3, 
          modules = "Module 1"
        ), 
        teal_operation( # the same operation as for Module 1 but not shared with Module 1 - not shared
          operation3, 
          modules = "Module 2"
        )
    )
)    

---

UI

We will reuse the space that is currently used by Filter Panel. We will have collapsed chips for every operation that was specified through operations list. Chips will be colored respectively to its context for the module

• red chip - this operations is used in this module and shared in at least one other module
• no color chip - this operation is only used in this module
• grey chip - this operation is not used in this module

Each chip (that represents one operation) next to the color, to make it more obvious, will have a label: shared (red chip), module specific (no color chip) and not used (grey chip) so it is clear what is the connection of this operation to this module, and what impact the changes to this operation have for other modules.

• shared (red chip) operations
  • Will have a button, that on a click, displays the labels of modules with which this is shared.
  • Will have a button to deactivate this operation in this module.
• no color chip operations
  • Will have a button to enable to share this operation with other modules.
  • Will have a button to deactivate this operation in this module.
• grey chip operations
  • Will have a button, that on a click, displays the labels of modules with which this is shared.
  • Will have a button to activate this operation in this module.

There is also an option to hide/show operations not-used in this module. This is hidden by default.
With such specification as below, we are able to control every operation from the UI view of each module.

---

Operations Manager

The same as currently existing filter manager - it allows to activate or deactivate operations on specific modules. It also has a button to deactivate or activate on all modules. It also enables to copy the operation, that lands as a last operation. Maybe with drag and drop we can then change the order of operations.

---

Ability to create custom operations through the app

On the RHS of the app, there will be a huge + sigh that will allow to

• add new operations
• add new filters

From every module UI view, you will be able to add new operation or filter. It will land as a last operation on the list.

Creation of operation

During the creation you will be able to specify if this is shared (and with which modules) or not shared. New operations will be just a window where you can put an R code that should work on objects called the same way as datanames used in this module. So if 2 datanames are used in this module (e.g. ADSL and ADTTE), the operation will allow to modify those two

ADSL$colum1 <- ADSL$colum1 + 2
ADTTE <- ADTTE[, -1]

We should inform a user that (s)he is only able to use functions and packages available through session info.
Warnings/errors should be displayed.
We should limit the ability to run install.packages() and library().

Creation of filter

It will just start the new collapsable filter panel. From that moment you can create the filter the same way you normally did before.

---

What will be removed

• teal_slices is not needed as we only use teal_slice now

What will need to be changed and implemented

• teal::init gets a new parameter operations
• teal_operation is implemented in teal.transform
• if we need some more customization, operations parameter might take an object as an input (called teal_operations) instead of a list - this teal_operations can take parameters for greater

---

Pros and cons

Advantages

• changes only in teal and not in modules
• possibility to set any order of preferences for the operations
• not much changes on the UI side for the current state - we only reuse the space of current filter panel
• visibility on the UI side on which operations are shared and used on module's view
• possibility to activate and deactivate operations for module from module's view
• possibility to activate and deactivate operations for modules from special manager modal

Disadvantages

• relies on modules' labels, but currently the filtering also relies on it

---

What to think about

• possibility to reorder operations with drag and drop - should the ordering happen only within module or should be applied to all modules

[gogonzo]

Thanks for this @m7pr , I have few neutral comments:

1. Proposed is a an interpretation of "6. Mixed filters and transforms" with teal_operation instead of teal_transform and modules argument which is an third alternative after mapping and module(transforms).
2. If any order of operations is possible then each filter call will be evaluated individually. For example:

x <- dplyr::filter(x, var1 == "a")
x <- dplyr::mutate(x, varx == rnorm(n())) 
x <- dplyr::filter(x, var2 == 1)

# instead of (now)
x <- dplyr::filter(x, var1 == "a" & var2 == 1)

3. Regardless of the solution we need to think how to propagate validation errors, which in this solution have a highest possibility (multiple operations filter/mutate alternately)
4. teal_slices could be used to group filters and produce a single call to avoid (2) (if 2 is really the problem)

[m7pr]

1. Yeah, I just directly specified how you pass a connection between transformer/operation and a module, and also elaborated a bit more about the UI. But it looks very similar.
2. I don't think it's a problem.
3. You propagate errors on a transformer/operations UI in which the error appeared first, and all next transformers/operations UI can have the same error message displayed.

[gogonzo]

Same applies here #1216 (comment)

We can't have shared custom operation module. @m7pr could you add more details to teal_operation, what teal_operation(operation1) means?

[gogonzo]

how would you connect code with shiny input so the operation is interactive?

[m7pr]

textInput with a button to save/execute code

[gogonzo]

How would you define informations to match input with the code element? What I mean is

• if operation is operation1 <- "iris <- mutate(iris, new = "some value")"
• and if textInput produces value

How you'd linked them together? Can you provide any pseudo code of an api which could make this modification interactive?

[m7pr]

This is fairly simple:

Imagine a module, in which an operation1 will be performed, contains a below teal_data()

tdata <- teal_data(x = 5, y = 6, IRIS = iris)

For an interactive session, if a user wants to add an operation, there is a huge + button that you click and opens a modal with an information

Currently the objects in this module are called $datanames(tdata), use the window below to overwrite them or create new objects.

And you have a textInput and a create button. Whathever gets passed to textInput during a click on create will be put into tdata <- eval_code(tdata, code = input$textInput).

It is the tdata that will be overwritten/updated with the custom code.

[m7pr]

We had a talk where it occurred I misunderstood interactive transformation (that is dependent on the shiny input) with the interactive possibility to add a new operations/trafnsormer.

[gogonzo]

Closing along with #1216