App shortcuts / actions

[hemanth]

On native apps, we have an option to provide app shortcuts on long press on the app icon, for example:

I feel the same would be very useful for PWAs and it would be easy to mention them in the mainfest?

//cc @slightlyoff

[hemanth]

Maybe something like below makes sense?

contextMenu: [{
   url: '..',
   icon: '..',
   description: '...'
}]

[marcoscaceres]

probably just "label": ... can someone dig up how this registration is done in Android itself (via Java or Xml or whatever)

[marcoscaceres]

Also, icon: '..', should be an "image object", as per the spec.

[marcoscaceres]

Btw, what do the little double bars do? Is that to drag things into different order?

[hemanth]

Double bars will help us to add that action to home screen.

shortcut.xml:

<shortcuts xmlns:android="http://schemas.android.com/apk/res/android">
  <shortcut
    android:shortcutId="compose"
    android:enabled="true"
    android:icon="@drawable/compose_icon"
    android:shortcutShortLabel="@string/compose_shortcut_short_label1"
    android:shortcutLongLabel="@string/compose_shortcut_long_label1"
    android:shortcutDisabledMessage="@string/compose_disabled_message1">
    <intent
      android:action="android.intent.action.VIEW"
      android:targetPackage="com.example.myapplication"
      android:targetClass="com.example.myapplication.ComposeActivity" />
    <!-- If your shortcut is associated with multiple intents, include them
         here. The last intent in the list is what the user sees when they
         launch this shortcut. -->
    <categories android:name="android.shortcut.conversation" />
  </shortcut>
  <!-- Specify more shortcuts here. -->
</shortcuts>

[kenchris]

Proposal from @mgiuca inspired by earlier proposal from me. I think this is pretty nice and the direction we should go:

https://docs.google.com/document/d/1WzpCnpc1N7WjDJnFmj90-Z5SALI3cSPtNrYuH1EVufg/edit

[sheshbabu]

Similar feature in iOS - https://developer.apple.com/ios/human-interface-guidelines/extensions/home-screen-actions/

[bschoepke]

I work on the team that maintains "jump lists" on Windows. Microsoft used the "msapplication-task" meta tag in the Windows 7 days:
https://technet.microsoft.com/en-us/windows/gg491725(v=vs.60)

The corresponding Windows 10 UWP API that windows web apps can use is:
https://docs.microsoft.com/en-us/uwp/api/windows.ui.startscreen.jumplist

Windows has the ability for apps to populate both static and "dynamic" deep link lists with custom category names, which is useful when you have the screen real estate. Google Chrome makes great use of the dynamic list in its taskbar jump list. This overlaps with @mgiuca's proposal.

Let's be careful to define these things in a UX-agnostic way. These are definitely useful in any launcher UX outside of context menus; you can imagine multiple different ways to display these on a home screen. However, you want more than just an icon and display name; you want essentially everything the app manifest itself has--multiple icon sizes, background color, etc. So for standardization, maybe it's helpful to think of "shortcuts" as separate apps, kind of like how Windows 10 "secondary tiles" work:

https://docs.microsoft.com/en-us/windows/uwp/design/shell/tiles-and-notifications/secondary-tiles

iOS 11 takes this even further by allowing widgets within the context menus below the deep links list, so apps can provide custom UI for actions. For example, radio buttons for the Reminders app that let you quickly complete a reminder. I would argue that is out of scope for a v1.0 of this spec though.

[mgiuca]

Hi Ben, thanks for engaging!

OK, I'm interested in exposing as much metadata in shortcuts as possible (with the user agent able to use or ignore whatever it wants). What if the shortcut object (inside the manifest) had a complete copy of the manifest structure inside it, so you can put in title, theme_color, icons, everything else. We'd probably want a whitelist of members that make sense (don't really want the shortcut having its own service_worker), but most of the cosmetic stuff probably makes sense in there.

[bschoepke]

What if shortcuts were just links to other manifests?

From a launcher UX implementer's perspective, I want to enable pinning of shortcuts to various launcher surfaces. Existing examples are the Android home screen as in @hemanth 's screenshot showing both "Take a video" and "Take a selfie" pinned and "secondary tiles" in Windows 10's Start menu.

[mgiuca]

Hi, (oops this is a week old, sorry)

I agree we want that feature, but I'm not sure how forcing developers to split their shortcuts into separate manifests helps. It just makes life harder for developers.

I think the thing you're getting at is that if the shortcuts are links to other manifests, then if the user pins one of these shortcuts, the implementation would store a link to that manifest. I don't think that's really what you'd do --- you'd extract the metadata from the manifest, convert it to the host system's format, and store it in that format. Therefore you don't really gain anything by having the shortcuts be separate manifest files.

(Also, we generally don't want to trade developer convenience for user agent implementor's convenience.)

[bschoepke]

I was thinking more along the lines of if shortcuts require basically all the same stuff as the start_url, then they might as well just be manifests.

From a user's perspective, what should happen when I add amazon.com/gp/registry/wishlist to my homescreen? I kind of expect an amazon wish list "app" to be pinned rather than an amazon.com app, and I expect long-pressing the wish list app to show links to my various wish lists. Amazon could presumably make that happen by providing a separate manifest with a start_url pointing to the wish list page, in which case the root amazon.com manifest might as well just link to the wish list's manifest.

I'll admit I'm not too familiar with the overall PWA spec so this might be against its general philosophy--is the intention for there to be one manifest per domain?

[aarongustafson]

I totally missed this conversation when it happened, but this has been on my mind for a while now too. I’m in agreement with @mgiuca that we should keep it simple. For a large percentage of apps out there, static will be enough. a second version can add to that (perhaps even allowing for custom layouts like player controls for a picture-in-picture video player or audio stream).

Not knowing this conversation was taking place, I’d been working on an explainer as well: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/master/Shortcuts/explainer.md

It has a lot of similarities to what is being proposed currently. I’ll look for deviations I think are worth discussing tomorrow when I’m back @ my desk.

[aarongustafson]

@bschoepke wrote:

I'll admit I'm not too familiar with the overall PWA spec so this might be against its general philosophy--is the intention for there to be one manifest per domain?

Not per domain, but generally per scope. Overlapping manifest scopes might cause issues. That said, your Amazon example is a good one and points out the distinction between a traditional URL shortcut (which points to a specific page) and a PWA (which has a default start_url). Honestly, I think there is likely to be some unavoidable (and unfortunate) user confusion here. This is one reason it makes sense to have a browser icon overlaying the app icon for shortcuts and no browser icon for PWAs.

[raymeskhoury]

It has a lot of similarities to what is being proposed currently. I’ll look for deviations I think are worth discussing tomorrow when I’m back @ my desk.

Thanks @aarongustafson! For reference @mgiuca's latest (but still somewhat old) proposal is at https://docs.google.com/document/d/1WzpCnpc1N7WjDJnFmj90-Z5SALI3cSPtNrYuH1EVufg/edit

[aarongustafson]

@raymeskhoury I made some comments on the doc.

If @mgiuca is cool with it, I’d be happy to synthesize everything thus far into a new explainer doc. We (Edge) are eager to take this work on for Chromium, so I think it makes sense to get at least the v1 rolling. Thoughts?

[kenchris]

I am happy with it! Happy to see something happen on this front

[aarongustafson]

@kenchris Cool. Once I get a 👍🏻 from @mgiuca as well, I’ll give it a go. I can get cracking on it tomorrow.

[mgiuca]

I'm still on leave so I'll let @raymeskhoury speak for me.

I had a quick look at the proposal and it looks fairly similar to what I had. I don't have any objections to Aaron taking and working on this but I'm out of the loop. Check with Raymes / Chrome team to see if they have any plans to work on this.

Great to see progress being made and the fact we independently came to basically the same design means we're probably on the right track.

[aarongustafson]

@mgiuca Enjoy your leave!!!

[aarongustafson]

Hey all! I took a stab at merging the separate explainers into a single one. I think I captured everything, though I intentionally kept the "Considerations for v2" at a high level. I figured we could tuck into that one once we are pretty comfortable with this.

Please let me know if I missed anything (or mischaracterized anything). I did my best to bring it all into one voice, borrowing text from each of the explainers. Happy to make more revisions next week. For ease of editing, I made the changes to my copy: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/master/Shortcuts/explainer.md

[tomayac]

Three quick comments/suggestions:

• Instead of just right-click (desktop-focused), explicitly mention long-tap (mobile-focused) in the intro.
• For the shortcut icons, align with src instead of uri.
• The dark mode link should point to Add a color_scheme member to ImageResource #758 (the current link 404s).

Apart from that really like the Explainer.

[kenchris]

I agree, actually in web specs today we never use the term URI (always URL) and "src" is aligned with many HTML elements, but we do use "start_url" in manifest, so "url" makes more sense... We don't use it as the source for js, images etc but as a URL to go to

[g-ortuno]

Could you mention where the navigation will happen when the user clicks the shortcut in the Responding to shortcut activation section? On Android it'll replace the existing context if there is one or create a new one if there isn't. And on desktop it will always open a new window.

I think always opening a new window on desktop might not be a great experience for sites like Spotify but I don't think we can do anything until we get the proposed launch event.

[raymeskhoury]

The approach makes sense to me and seems to mostly match mgiuca's :)

A few notes:
-+1 to updating uri->url
-+1 to explaining how the URL will be opened if a launch event handler isn't registered (especially since that spec is still in flux, it would be good to detach from it)
-mgiuca had a short_name field but I'm not sure what the rationale was for this

[aarongustafson]

• The dark mode link should point to Add a color_scheme member to ImageResource #758 (the current link 404s).

@tomayac Someone caught that and pointed it to the explainer. Do you think it should point to the issue instead?

Thank you for the other comments. I am working on some language around the activation now.

[aarongustafson]

Could you mention where the navigation will happen when the user clicks the shortcut in the Responding to shortcut activation section? On Android it'll replace the existing context if there is one or create a new one if there isn't. And on desktop it will always open a new window.

I think always opening a new window on desktop might not be a great experience for sites like Spotify but I don't think we can do anything until we get the proposed launch event.

Agreed. In Windows, there is something called an activation event, which is how Twitter handles it within their PWA. This is very similar to the proposed launch event, but there are a few notable differences I plan to bring up in that discussion.

[aarongustafson]

@g-ortuno @raymeskhoury Let me know how the activation section looks to you.

[aarongustafson]

@marcoscaceres If we can come to a general agreement on this one, do you have thoughts on whether this is something that should be integrated into the Manifest spec or whether we should pursue it as an extension to the spec?

[raymeskhoury]

Thanks!

If one or more windows are currently open within the context of the Manifest, the shortcut action should be directed into the only (or most recently used context). If no contexts are open, a new one should be created to respond to the shortcut action.

I think we should follow the behavior of existing features that have the same pattern (e.g. registerProtocolHandler and Web Share Target). In these cases I think currently a new window is opened for each request (but we should verify that :). It can be dangerous to perform the navigation in an existing context because the user may have unsaved changes there. As you mentioned, the launch event will give full control over windowing.

In Windows, there is something called an activation event, which is how Twitter handles it within their PWA.

Hmm, what do you mean by "which is how Twitter handles it within their PWA"? Are you saying the twitter PWA handles Windows-specific activation events?

This is very similar to the proposed launch event, but there are a few notable differences I plan to bring up in that discussion.

Please do! Feel free to file bugs on the repo. We're currently in the process of doing some work on that spec :)

[aarongustafson]

I think we should follow the behavior of existing features that have the same pattern (e.g. registerProtocolHandler and Web Share Target).

I’ll revisit those explainers.

Are you saying the twitter PWA handles Windows-specific activation events?

Yes. Since PWAs on Windows are (currently) delivered by the Store, they are in a UWP wrapper. This gives them access to the WinRT feature set and all of the internals that make them run. The activation event is used for Share Target, Jump Lists, and more. I have an abbreviated example of how it is used with Share Target in this demo. Here’s a slightly modified excerpt from the code:

function activationHandler( e ) {
  console.log( 'Activation Event', e );
  if ( e.kind == ActivationKind.shareTarget ) {
    console.log( 'Handling the share' );
    shareHandler(e);
  }
}

if ( 'ActivationEvent' in window ) {
  activationHandler(window.ActivationEvent);
}

I intend to bring this approach up on the launch event discussion, but have not had a chance to do that yet (trying to get shortcuts wrapped first).

[aarongustafson]

I think we should follow the behavior of existing features that have the same pattern (e.g. registerProtocolHandler and Web Share Target).

Done.

[tomayac]

@tomayac Someone caught that and pointed it to the explainer. Do you think it should point to the issue instead?

Nah, pointing to the explainer is even better :-D

[aarongustafson]

I’ve read through @slightlyoff’s Declarative Web Actions concept and it overlaps a bit with what we’re discussing here. I have some thoughts, but I wanted to let ya’ll check it out before I give my thoughts.

Main question: Do you think we should seek to merge these ideas?

[raymeskhoury]

My only concern is that this API currently seems very well defined and fairly uncontroversial. I suspect that Declarative Web Actions is going to require significantly more discussion to get agreement over but I may be wrong.

On the other hand, things like menus and keyboard shortcuts would seem to fit in very nicely in this API and it would be a shame tack them on later, so perhaps it is worth considering something more general.

[aarongustafson]

@marcoscaceres Any thoughts/feelings on whether this makes more sense to merge into the Manifest spec proper or consider it an extension?

[marcoscaceres]

Merging into the spec proper just requires implementation commitment from two implementers. If this is something you'd like to pursue on the Microsoft side, then it's definitely a candidate for inclusion. I'll need to figure out if it's something we could implement in Firefox on at least Android.

[aarongustafson]

@marcoscaceres Our intent to implement in Chromium is forthcoming. We’re already working on plumbing the implementation.

[marcoscaceres]

Awesome, ok. Let’s move this over.

[raymeskhoury]

Please post a link to the moved explainer when it's ready :)

[aarongustafson]

Awesome, ok. Let’s move this over.

Thanks @marcoscaceres. I’m a newbie with this, so what do you need me to do? Do I move the Explainer to WICG first and start a thread there? Submit a PR with updates to the Manifest spec?

[aarongustafson]

Do I move the Explainer to WICG first and start a thread there?

I am not a member of the WICG GitHub group, so I may need that access first.

[marcoscaceres]

No need to go through the WICG. Are you a member of Web Platform WG? if not yet, you need to ping Mike Champion to add you to the working group (Microsoft is already part of the working group).

Then we just do everything via a pull request to the spec. Happy to help guide you over email if are unsure of anything ([email protected]) - or also happy to have a call if you’d like.

[aarongustafson]

As I started integrating this into the manifest spec, I began to think that maybe ShortcutItem might make more sense than ShortcutInfo as a name for the object type enumerated within the shortcuts member. I feel like it more readily aligns with similar constructs like ImageResource.

Thoughts?

[aarongustafson]

Ok, I think I have the merge complete: #768

Feedback appreciated.

[aarongustafson]

Plan to provide an update on this @ TPAC today.

[aarongustafson]

Closed by #768