Form Gramplet: per Form, per Column actions[v52]

[stevenyoungs]

Extend the Form gramplet so that the Form creator can specify per column actions which the user can invoke as required. An 'action' is a snippet of Python code defined in the form XML definition. These actions are intended to automate steps you can already do manually.
An example action is provided that extends the UK1841 census form to allow an 'Occupation' event to be created, with appropriate date and source citation, from the Occupation column.
It is hoped that the framework is sufficiently generic to allow more complex actions to be implemented in future. Indeed you already can add actions to other columns by simply editing the form XML file.
The XML definition of an action consists of three bits of data

1. "title" attribute is used to build the popup menu
2. "enable_if" determines if the action should be added to the popup menu
3. "command" the Python code that implements the action

WARNING: this is very much a proof of concept.

• There is little error checking
• Currently you must save the form and reopen it before the exmple action will work correctly (saving the form for the first time creates the citation and census event)
• The example action does not check to see if the event already exists - so you can keep adding duplicate events to a person!

The extended popup menu

The event created by this action

To Do

• enable_if needs to be eval()'d from the EditFrom class, not EntryGrid as currently.
• implement actions in Person and Family sections of a form. Currently only Multi sections are supported
• auto-save the form before an action is run so you don't have to close and re-open a form
• check the text is localisable.
• allow an column's action to be run for all rows in the form section
• make example action robust: don't create duplicate events.
• consider running an action for all rows in a form
• consider running all actions
• consider automatically running an action when a form is first saved
  and much much more

I'm no Gramps, Gtk or Python expert so I'm sure the code can be improved - all ideas and comments welcome. In particular

• I'm not happy with the way I have action_callbacks in each class.
• Have I missed any steps when creating the Occupation event (see XML file)?
• The command implementation is tightly bound to the implementation of EditForm. Maximum flexibility now, but a problem for the future?
• Is EditForm the best place to exec() the command from?

Ideas for actions

• Add name citations
• Create Birth events from a census form Age column. Method varies by form. e.g. UK1841 rounds down ages for those over 15 so the Birth event date would be, for example, "between 1821 and 1826"
• Create a shared residence event from a census form "household"
• Create or extend a family from a birth certificate form

[stevenyoungs]

Following discussion on the gramps-devel list I've completly reworked the implementation.

Actions are now separated from transcription. A new "Actions" button has been added for the FormGramplet window. When clicked, this displays a list of available actions for the selected form. This list is dynamic, only displaying actions which could make sense.

The user selects the actions they wish to apply (multi-selection is permitted) and when OK is clicked, the actions are run.

The interesting code is

• FormActions._populate_model() which calls a series of class members to generate the list of actions to show in the list
• FormActions.run() which runs the selected actions (as a single transaction)

The code to decide which actions are possible, and the implementation of the action can be found in UK1841.py

All actions derive from a common base class, ActionBase.

I've provided two prototype actions.

• Adds a primary name citation.
• Add an occupation event.

Longer term I think more code can be moved from UK1841.py into ActionBase.py. ActionBase then becomes a library of helper functions to keep the amount of form specific code as light as possible.

This is still a work in progess submitted to get feedback on the implementation approach.

[stevenyoungs]

I've added initial support for creating birth events
This helps to illustrate that there is some common code in the populate_model functions (see UK1841.py) that can be extracted . It also demonstrates that for some forms the logic to decide what event to create gets quite involved. You could argue that this work could be deferred until the user runs the action, but I believe it gives a better UX if the user can see the detail of the action in advance. It would be cumbersome for a user to manually find and check the event(s) an action creates afterwards, one by one

Code now compatible with Python 3.3

[Nick-Hall]

This is a good start.

As far as the UI is concerned, I think that adding checkboxes for selection would be clearer. You could also add some extra information such as the place of birth and date of the occupation event.

The design also needs some more thought. I see the interpretation of the raw data as being part of the transcription phase. This would include calculation of the date of birth from the age as well as things like transliteration/translation where relevant. In previous discussions, I have suggested extra hidden attributes for this purpose (e.g %DOB for date of birth).

I suggest that you also start to tidy up the code:

• The actions should not contain UI code. Just return a list to be added to the model.
• Obtain a dictionary of attributes for each person once that can then be passed to an action.

[stevenyoungs]

• The actions should not contain UI code. Just return a list to be added to the model.

This is now done. Thanks.

I've added a new action: residence event, relevant to your second bullet point

• Obtain a dictionary of attributes for each person once that can then be passed to an action.

This action is slightly different because there is a single action, no matter how many rows on the form. It also does not require any attributes from the people on the form.
In general I'd prefer to keep FormActions ignorant of the specific details of a form. There may or not be people or families on a form from which to cache attributes etc. Do you think performance will be a problem if I don't cache the attributes?

Do you have any advice on what level of granularity is best for db transactions? See for example ResidenceEvent.command where I have N+1 tranactions for the N rows on the form.

I've changed to having individual transactions in preparation for allowing the user to, optionally, display the object edit window before the action completes. I can see two reasons for wanting this:

1. the form data cannot be understood by the action code. e.g. an Age of "2m" (or "2 months" or ...). So the action may mandate that the object edit window be shown

2. the user wishes to manually add some additional information e.g. notes. The user could tick a tick box to say "show the object edit window".

Each action would need to decide which steps to show the edit window for. However this does go against your first bullet point of no UI in the actions code. Thoughts?

Actions also need to be smarter

• add a citation if the event already exists.
• don't list the action if the event and citation already exist

Check boxes - yes agreed, before merging. They are more discoverable for a user. Need convenience routines e.g. select several actions and tick in one go. For now, multi-selection is good enough for me to test with!

re hidden attributes:

• how would these hidden attributes get set?
• what about backwards compatibility?
• what about not understood data (e.g. my "2 months" example above)?

[stevenyoungs]

UI now uses checkboxes to determine which actions to run.
Checkboxes beside the action categories turn all actions within the category on or off.

[stevenyoungs]

Some more experimental work to investigate if it is possible / sensible to allow the user to edit the detail of an action when it is run. Each action can indicate if a user cannot, can or must edit the detail of the action. This is then shown in the GUI.
When an action is run, I envisage passing it the edit_detail flag. The action can then use present relevant information to the user for adjustment / confirmation. For example by using the standard EditEvent dialog. This is still to do.

In the example below I've deliberately configured the actions, for testing purposes, so that all 3 states of editing are present.

All comments welcome.

[romjerome]

It sounds good.
Thanks!

Maybe just an issue around python version as we need to use at least python3.5 for
'importlib.util.module_from_spec(spec)'?

87440: ERROR: grampsapp.py: line 157: Unhandled exception
Traceback (most recent call last):
File ".gramps/gramps51/plugins/Form/formgramplet.py", line 187, in __form_actions
actions = FormActions(self.gui.dbstate, self.gui.uistate, [], citation)
File ".gramps/gramps51/plugins/Form/formactions.py", line 120, in init
self.actions_module = importlib.util.module_from_spec(spec)
AttributeError: 'module' object has no attribute 'module_from_spec'

https://docs.python.org/3/library/importlib.html#importlib.util.module_from_spec

[stevenyoungs]

Maybe just an issue around python version as we need to use at least python3.5 for
'importlib.util.module_from_spec(spec)'?

Ooops. That wasn't deliberate - thanks for letting me know.

The gramps 5.2 roadmap shows a dependency update to python 3.5.x. (though note that pyton 3.5 is end of life 2020-09-13).

This PR is targetting 5.2 so I don't think any change is required.

[stevenyoungs]

The edit check boxes are now functional and use the standard Editor windows, displaying them sequentially, as required. The user can make any desired edits and these get reflected in the change applied to the database.
The main FormAction window is now modeless, rather than modal, in keeping with other windows in gramps. This changed also reduced the amount of boiler plate code. Thanks @prculley for the idea.

Overall, this PR is now at the stage where I feel it is ready for review however I will not officially change its status due to the notes below.

Note:

• python 3.5 or later is required
• To test the edit functionality, you will need to patch editevent.py, pass callback into EditPrimary in init, otherwise, the event will not be saved, because the callback is never called. Separate PR to follow.

[Nick-Hall]

@stevenyoungs Thanks. I should get a chance to look at this over the weekend.

[stevenyoungs]

@stevenyoungs Thanks. I should get a chance to look at this over the weekend.

Thanks. I'm tempted to split out the (example) actions, in UK1841.py, into a separate PR, if the core change gets to an accepted level. Let me know if you think that's the right way to go. For now it made sense to keep things together so I, and reviewers, had concrete examples to consider and test.

[prculley]

First of all, I am only looking at code, I have not yet tried it out. One area I am concerned about is your method of sequentially opening the Editors. At first glance, it looks like you are depending on each editor doing the callback to go onto the next. I saw another PR for the main Gramps code to make this more consistent.

One concern is what happens if the user doesn't use 'Save', but rather closes the Editor? I think the callback doesn't get called in that case. So What happens to the Form?

One possibility that occurs to me that would make this PR independent of the main Gramps code changes, and fix the potential problem above, would be to use a Gtk callback when the Editor Window is destroyed. I think this would always work, as that must happen when the Editor is closed by 'Cancel', the corner 'x' or Save'. Something like

editor = action(dbstate, uistate, track, edit_detail)
editor.window.connect("destroy', lambda...)

I don't know if this will work, in part because I wonder if stacking up a bunch of new window creations will work in a destroy callback. If it doesn't, I would then try setting a flag in self in the destroy callback, and using Glib.idle_add to restart your code when the flag gets set.

[Nick-Hall]

Sequentially calling editors is a horrible design. Users hate this type of interface.

We must think of something better.

[stevenyoungs]

Sequentially calling editors is a horrible design. Users hate this type of interface.

I can't really argue with that :-(
What I would really have like to do, and I think would be better UX, is

1. build the new and revised (updated) objects in memory (not yet in the db)
   this would be the get_actions part of the code
2. display the possible actions as now
   allow user to select the actions they want to run
3. allow the user to edit the actions new and revised object(s) as they wish (still outside the db)
   for example, via an 'Edit action' button on the FormActions window.
   reuse the existing EditX windows, but without saving to the db
4. for enabled actions, add / commit the revised (potentially edited) objects to the db
   no user interaction required.
   optionally close the FormActions window on completion

but I saw challenges in achieving this which I wasn't sure I could overcome.

• what to do if the revised object, external to FormActions, changes after step 1, but before step 4
  may help if the FormActions window was modal, but I'm trying to avoid that.
  concurrent users would mean even a modal window would not help
• what to do if two or more actions try to update the same object?
  if the revised objects are constructed, sequentially, in step 1, then the revisions from earlier actions would get overwritten by later actions (because they both, independently, built their revised version of the object)
• how to handle Edit windows updating other, linked, objects
  should these changes really hit the db until the action is run in step 4?

A fallback could be to let the user only run one action at a time. You could still, depending on the action, get sequential edit windows appearing.

Consider the 'simple' action of adding a residence event for N people on the form. The user potentially needs / wants to edit

• 1 x event object (the residence event)
• N x event_refs objects
• N x Person objects (potentially)
  You could obviously simplify down to N X Person edits, at the expense of the user having to manually find the correct event_ref, if they wished to edit it, or the event itself.

Ways Forward
I can see the following ways forward:

1. remove the 'edit' part from this PR
   submit a separate PR, containing a revised 'edit' solution, if and when ready
2. continue to include 'edit' functionality in this PR

I'm tending toward route 1; in general I prefer multiple, smaller, PRs over one larger one. It would give more space/time to develop a better UX for the Edit side. The downside is it would restrict possible actions to those where there was certainty.

Advice on any and all of the above sought and welcome.

[stevenyoungs]

First of all, I am only looking at code, I have not yet tried it out. One area I am concerned about is your method of sequentially opening the Editors. At first glance, it looks like you are depending on each editor doing the callback to go onto the next. I saw another PR for the main Gramps code to make this more consistent.

That's correct.

One concern is what happens if the user doesn't use 'Save', but rather closes the Editor? I think the callback doesn't get called in that case. So What happens to the Form?

Correct. I took any closure of the Editor window, other than via the 'Save' button, to mean cancel the actions that have not yet been run.

One possibility that occurs to me that would make this PR independent of the main Gramps code changes, and fix the potential problem above, would be to use a Gtk callback when the Editor Window is destroyed. I think this would always work, as that must happen when the Editor is closed by 'Cancel', the corner 'x' or Save'. Something like

editor = action(dbstate, uistate, track, edit_detail)
editor.window.connect("destroy', lambda...)

I did experiment with this approach. I had two issues:

1. I could not tell if the user clicked 'Save' or the window was destroyed for some other reason
2. It felt wrong to assume knowledge of editor.window from an external method

Is there a way to overcome 1 and is 2 acceptable?

[stevenyoungs]

• what to do if two or more actions try to update the same object?
  if the revised objects are constructed, sequentially, in step 1, then the revisions from earlier actions would get overwritten by later actions (because they both, independently, built their revised version of the object)

Thnking out loud, I wonder if I could store the difference between the original objects __dict__ and the revised objects __dict__? Something like

edits = updated_object.__dict__.difference(original_object.__dict__)

and then if the action is run, apply the stored difference to the current db version of the object?
It might need to be a bit smarter, but hopefully the basic idea is clear.

Not sure it would work for adding object references though, since the handle of the object the action intends to add a ref for may not have been created yet.

[prculley]

The usual way to compare these objects is to do original_object.serialize() == updated_object.serialize() which evaluates to True if not changed (cancel edit), and False if modified (saved). This works for nearly everything (I think there is a bug for Notes with markup, but that is another issue).

[pqhf5kd]

@stevenyoungs are you still working on this feature?

[stevenyoungs]

@stevenyoungs are you still working on this feature?

Yes in the sense that I would still like to get an acceptable working solution to the basic problem and I continue to think about ways to acheive that. A solution will save me a lot of time with my own research.

No in the sense that I've not had time to revisit since February.

There are still some design level challenges that need to be overcome. I'm not 100% sure in my own mind what the "correct" design is. All ideas and thoughts welcome.

[Nick-Hall]

There are still some design level challenges that need to be overcome. I'm not 100% sure in my own mind what the "correct" design is. All ideas and thoughts welcome.

I'll put some more thought into this shortly. I'd like to make a little more progress with the new combined view first and the place enhancements in core may take some of my time.