Plugin Configuration system V2

[jgclark]

We've had a conversations in several Discord channels and briefly by video conference about this; I thought it would help to start a discussion here to bring it into one place that's easier to reference. cc @EduardMe @mikeerickson @weyert @nmn @dwertheimer but of course open to anyone else too.

Why do we need a V2?

V1 was never designed; it just grew out of @nmn spotting a way to do config without any specific support from the NP app itself. We're now spending support time fixing fragility of this single, shared, _configuration resource, and that JSON is too complex for most users to handle.

Requirements for V2 (using will/should/could prioritisation)

• NotePlan will have UI support that abstracts away the JSON from users
• plugin authors will specify the settings in the plugin.json file
• to ease debugging and trouble-shooting, settings will be persisted to disk in a JSON file in the plugin's folder
• settings values should also be available to plugin authors through the DataStore.preference and DataStore.setPreference() APIs. To avoid name collisions, the setting name will be further distinguished by its specified plugin.id
• settings UI will be triggered from a button in the plugin preferences panel
• settings UI should be shown, with defaults in place, when a plugin is enabled
• settings UI could include decorative items (eg. sub-headings and horizontal lines, and URL links) also specified in the plugin.json file
• settings will be persistent across upgrades to the plugin
• settings will support at least string, numeric, boolean and array of strings types
• settings should support single-selection from a list, and multiple-selection from a list
• settings will support default values
• settings could be versioned using a simple one-up numbering system. If so, when a plugin is updated this number will be checked, and if it is greater than the current value, then the settings UI will be shown to the user.

[jgclark]

Here's an example of the extension to the structure of the plugin.json file I'm proposing, showing examples of most of what I think is needed. Note that it includes both settings and 'decoration' instructions for the UI:

{ // I suggest we treat this as JSON5 not strict JSON
  "plugin.id": "jgclark.DailyJournal",
  // rest of existing file ...
  "plugin.settingsVersion": 1,
  "plugin.settings": [   // assumes the order of what follows is to be kept for the UI
    {
      "type": "heading",  // i.e. not a setting but a heading to show. Might be possible to have this as type "info", but I'm separating this out for flexibility in the UI construction.
      "markdownText": "## Settings for Journalling"
    },
    {
      "type": "string",
      "name": "reviewSectionHeading",
      "text": "Heading to insert before review comments, if not already present",
      "default": "Journal",
      "required": true,
    },
    {
      "type": "[string]",
      "name": "moods",
      "text": "Array of moods to choose between",
      "default": ["🤩 Great","🙂 Good","😇 Blessed","🥱 Tired","😫 Stressed","😤 Frustrated","😔 Low","🥵 Sick","Other"],
      "required": true,
    },
    {
      "type": "info",
      "markdownText": "A section of _markdown_ text, including [external links](https://github.com)."
    },
    {
      "type": "string",
      "name": "reviewQuestions",
      "text": "",
      "default": "@sleep(<number>)\\n@work(<number>)\\nMood:: <mood>\\nExercise:: <string>\\nGratitude:: <string>\\nGod was:: <string>\\nAlive:: <string>\\nNot Great:: <string>\\nWife:: <string>\\nRemember:: <string>",
      "required": true,
    },
    {
      "type": "separator", // implies a horizontal separator line goes here
    },
    {
      "name": "sortMethod",
      "type": "string",
      "text": "How should the output be sorted?",
      "singleSelectList": ["alpha", "date", "length"],
      "default": "alpha",
      "required": false,
    },
  ],
}

[jgclark]

My suggestion, implied from the requirements, is that the specification of settings (example above) live in the extended plugin.json file, and that the actual currently-active settings are persisted in settings.json, which would then simply be:

{
  "plugin.settingsVersion": 1,
  "plugin.settings": {
    "reviewSectionHeading": "Journal",
    "moods": ["🤩 Great","🙂 Good","😇 Blessed","🥱 Tired","😫 Stressed","😤 Frustrated","😔 Low","🥵 Sick","Other"],
    "reviewQuestions": "@sleep(<number>)\\n@work(<number>)\\nMood:: <mood>\\nExercise:: <string>\\nGratitude:: <string>\\nGod was:: <string>\\nAlive:: <string>\\nNot Great:: <string>\\nWife:: <string>\\nRemember:: <string>",
  },
}

[mikeerickson]

@jgclark This is a terrific start!!!

I have a couple questions / comments. I welcome your feedback and we can surely adjust (especially base on item 3)

Note: The "plugin.settings" key is the ONLY item(s) which are displayed in the event a "name" item is removed in the future, thus only plugin defined settings items are displayed.

---

1. I would suggest change the "plugin.settingsVesion" from numeric to string (in the event we have something outside of a sequenced version (i.e., "1.1" "1.2" etc) However, this could be accurate based on your desired usage

settings could be versioned using a simple one-up numbering system. If so, when a plugin is updated this number will be checked, and if it is greater than the current value, then the settings UI will be shown to the user.

2. I would propose we define "type" options (i.e., 'array', 'boolean', 'radio', 'text') [see below]

3. Can you provide a bit more context on how the "reviewQuestions" would be used (I am sure it is for one of your plugins, but I am not familiar with all your offerings). Based on your response, we may need to adjust configuration a bit further to support this answer.

4. I would propose each "pluginSetting" item would follow a consistent set of keys

* fields are required setting 
{
  name:      * unique name (i.e. sortMethod)
  type:      * data type (heading, info, checkbox, radio, text, select, multiple)
  text:      * associated text (optional)  
               [this would replace `marktdownText` field, just use "# Sample"]
  required?:   true [omitting this field would be same as `false`]
  choices?:    array of choices (if type === array)
  default?:    default value
}

Example w/ heading [there can only be ONE heading, it will always be at the top]

{
  "name": "heading1"
  "type": "heading"
  "text": "# Heading Text"
}

Example w/ info [remainder of static text rows will be info]

{
  "name": "info"
  "type": "info"
  "text": "Heading Text"
}

Example w/ select

{
  "name": "moods",
  "type": "select",
  "text": "Select Mood",
  "choices": ["🤩 Great","🙂 Good","😇 Blessed","🥱 Tired","😫 Stressed","😤 Frustrated","😔 Low","🥵 Sick","Other"],
  "default": "😫 Stressed",
  "required": true,
}

Example w/ multiple

{
  "name": "moods",
  "type": "multiple",
  "text": "Select Moods",
  "choices": ["🤩 Great","🙂 Good","😇 Blessed","🥱 Tired","😫 Stressed","😤 Frustrated","😔 Low","🥵 Sick","Other"],
  "default": ["🥱 Tired", "😫 Stressed"],
  "required": true,
}

Example w/ checkbox

{
 "name": "includeFooter",
 "type": "checkbox",
 "text": "Include Signature",
 "default": true,
 "required": true,
}

Example w/ radio

{
  "name": "color"
  "type": "radio"
  "text": "what is your favorite color"
  "required": true
  "choices": ["red", "green", "blue", "orange", "purple"]
  "default": ""
}

Example w/ text

{
  "name": "firstName"
  "type": "text"
  "text": "what is your first name"
}

[jgclark]

Thanks, @mikeerickson. Comments on yours ...

Note: The "plugin.settings" key is the ONLY item(s) which are displayed in the event a "name" item is removed in the future, thus only plugin defined settings items are displayed.

Sorry, I don't quite understand what you're saying ... can you re-word?

settings could be versioned using a simple one-up numbering system. If so, when a plugin is updated this number will be checked, and if it is greater than the current value, then the settings UI will be shown to the user.

1. I would suggest change the "plugin.settingsVesion" from numeric to string (in the event we have something outside of a sequenced version (i.e., "1.1" "1.2" etc) However, this could be accurate based on your desired usage

Could be something else, as long as it's unambiguous both to devs and NP which is the newer version.
I can't see the need for anything approaching semver here: it feels a change is a change is a change.

1. I would propose we define "type" options (i.e., 'array', 'boolean', 'radio', 'text') [see below]

This feels like it mixes data type with display method which I was trying to avoid. Let me enumerate in more detail:

• type: boolean → simple on/off control
• type: string → simple text area
• type: numeric → simple float input. [or split into float & integer if we think we need that]
• type: [string] → then needs additionally to specify whether this is pre-defined single-select, pre-defined multi-select or user-specified unbounded list
• type: [numeric] -- seems unlikely that this would be needed?

Plus type: 'info' or 'decoration' which can then be heading/text/line.

2. Can you provide a bit more context on how the "reviewQuestions" would be used (I am sure it is for one of your plugins, but I am not familiar with all your offerings). Based on your response, we may need to adjust configuration a bit further to support this answer.

This is just a long string type in my existing DailyJournal plugin, used to here to show we need to cope with newlines.

3. I would propose each "pluginSetting" item would follow a consistent set of keys

* fields are required setting 
{
  `name`:      * unique name (i.e. sortMethod)

Yes, unique. I since wondered whether this should be key to help imply this is part of key:value, not an English 'name'.

type: * data type (heading, info, checkbox, radio, text, select, multiple)

As above, I don't think some of those are data types. But yes, a required field.

text: * associated text (optional)
[this would replace marktdownText field, just use "# Sample"]

Yes, I think we're concluding all 'text' fields can be markdown.

required?: true [omitting this field would be same as false]
choices?: array of choices (if type === array)
default?: default value
}

Yes, this is good.
Then we just need a way of distinguishing my pre-defined single-select, pre-defined multi-select or user-specified unbounded list from above. So perhaps:

• choicesList?: array of choices (if type === array)
• choicesSelection: "single" | "multiple" | "any" (i.e. 1 of fixed list; any from fixed list; anything you like)

Just one last thing: in complex lists of settings we might more than one markdown heading. (I think I've seen this in Obsidian.). So perhaps just an 'info' type is needed, as long as it can render '---' as a horizontal line.

[mikeerickson]

Further, I am curious as to what the timeline would be on implementing these new settings. As I am working on np.Templating project, I am in need of a few configuration items and am guessing I will need to use _configuration in the short term.

[jgclark]

I'm curious too, but hopefully doing this specification to help @EduardMe get the UI sooner. I'm holding off creating new plugin functionality to avoid adding to the load on the current _configuration system.

[jgclark]

Elsewhere @weyert suggested checking out VSCode's configuration schema. Sorry I didn't think of that earlier. It took a while but I tracked it down to https://code.visualstudio.com/api/references/contribution-points#contributes.configuration.
Unsurprisingly it is more extensive -- more than we need, I think -- but pretty similar. This is an example for how you specify number, string or boolean:

{
  "gitMagic.views.pageItemLimit": {
    "type": "number",
    "default": 20,
    "markdownDescription": "Specifies the number of items to show in each page when paginating a view list. Use 0 to specify no limit"
  }
}

Entries of type number, string, boolean can be edited directly in the settings UI. ... For boolean entries, the description (or markdownDescription) will be used as the label for the checkbox.
Some object and array type settings will be rendered in the settings UI. Simple arrays of number and string will be rendered as editable lists. Objects that have properties of type string and/or boolean will be rendered as editable grids of keys and values. The object type setting must set "additionalProperties": false to get this support.
If an object or array type setting can also contain other types like nested objects, arrays, or null, then the value won't be rendered in the settings UI and can only be modified by editing the JSON directly.

Here's their slightly different approach to my choicesSelection idea above:

enum: If you provide an array of items under the enum property, the settings UI will render a dropdown menu. You can also provide an enumDescriptions property, which provides descriptive text rendered at the bottom of the dropdown.

[dwertheimer]

Yes VSCode made the same transition nicely!

…

On Fri, Oct 1 2021 at 3:21 PM, Jonathan Clark ***@***.***> wrote: Elsewhere @weyert <https://github.com/weyert> suggested checking out VSCode's configuration schema. Sorry I didn't think of that earlier. It took a while but I tracked it down to https://code.visualstudio.com/api/references/contribution-points#contributes.configuration . Unsurprisingly it is more extensive -- more than we need, I think -- but pretty similar. This is an example for how you specify number, string or boolean: { "gitMagic.views.pageItemLimit": { "type": "number", "default": 20, "markdownDescription": "Specifies the number of items to show in each page when paginating a view list. Use 0 to specify no limit" } } Entries of type number, string, boolean can be edited directly in the settings UI. ... For boolean entries, the description (or markdownDescription) will be used as the label for the checkbox. Some object and array type settings will be rendered in the settings UI. Simple arrays of number and string will be rendered as editable lists. Objects that have properties of type string and/or boolean will be rendered as editable grids of keys and values. The object type setting must set "additionalProperties": false to get this support. If an object or array type setting can also contain other types like nested objects, arrays, or null, then the value won't be rendered in the settings UI and can only be modified by editing the JSON directly. Here's their slightly different approach to my choicesSelection idea above: enum: If you provide an array of items under the enum property, the settings UI will render a dropdown menu. You can also provide an enumDescriptions property, which provides descriptive text rendered at the bottom of the dropdown. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#93 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACEI6VEEAHZVQANMYUOASIDUEYX65ANCNFSM5FC74YXA> .

[mikeerickson]

Agreed, hence the reason I suggested. @weyert also suggested reviewing Shopify, but have not had time to review.

[dwertheimer]

I've spent a lot of time with Shopify and its Liquid templating. It's fantastic, but not the right model for what we're doing here. VSCode feels like the right approach.

…

On Fri, Oct 01, 2021 at 6:53 PM, Mike Erickson ***@***.***> wrote: Agreed, hence the reason I suggested. @weyert <https://github.com/weyert> also suggested reviewing Shopify, but have not had time to review. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#93 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACEI6VDSYTPPDYWJNZS7SWTUEZQYJANCNFSM5FC74YXA> .

[mikeerickson]

Ya, I was thinking of the VSCode interface from the jump. It seems to be a logical implementation for what we need.

[EduardMe]

Thanks for sharing this list. The example JSON helps a lot! I’ll finish some features for the next release and then get to work. The new features are mostly iterations on what’s already there, so I’m hoping this won’t take so long. The next beta is also almost ready. Looks like plugin-wise this is the next high-priority improvement for me from your description.

[mikeerickson]

And nice OS style alert, confirm and prompt please 😊