Unable to publish snap charms due to missing resources

[stub42]

The snap-layer is being used to write several new charms. To enable deployments in network restricted environments, the snaps may be uploaded as Juju resources. This is an optional step, only needed when deploying in a network restricted environment. These are optional resources, that normally should not exist, but 'charm release' insists that the resources are provided.

Please turn off this restriction. If the snaps are uploaded as resources at publish time, then snap automatic updates will not happen and one of the major benefits of using snaps is lost.

The snap layer works by first attempting to download the resource. If this fails, as is expected to happen in most deployments, the snap is installed from the snap store. If the resource does exist, the resource is installed as a snap and the store is never contacted.

Looking into this problem, I see several other people trying to deal with this unnecessary restriction. Mostly it is for resources that may not be redistributed. In these cases, the operator needs to obtain the resource and upload it at deploy time. If the resource was uploaded to the snap store, people could get sued and DMCA take down notices sent. I haven't done an extensive survey, but this seems to be the most common use case for resources. The usual way of handling the problem seems to be to upload a 0 byte file, which is a hack rather than a decent UI choice. Especially if I'm correct and this is the most normal usage of the feature.

[mitechie]

I don't think it's as simple as that. The charms are meant to be usable out of the box in some fashion. They should guide the user and fail cleanly. The requirement to have at least a token resource is so that the user cannot be left with a dead end and no idea where to go from there. There's also the charm testing situation. If we cannot work the charm w/o a local supplied resource then we cannot provide help to charm authors through automated testing and feedback when things don't work such as the testing we do on different clouds, checking if the charms work in containers, etc.

I've replied to some of this today in the thread on the mailing list on it.
https://lists.ubuntu.com/archives/juju/2016-December/008268.html

I do see the point with snaps. A resource is only pulled if there is a resource-get in the hook. I'd suggest that there either be a config on the charm or a attempt/time-out failure to get the snap from the store that falls back to getting the resource.

Maybe there's another way. I think we can spend some more cycles trying to put together the best experience. At the moment, I really feel that removing the block will get us into a bit of a swamp with many charms that no one can use or test because of missing expected resources.

[stub42]

I think the problem is that most uses I've seen of resources is for them to be optional, and the charm store is treating them all as required. For example, the docker charms I've seen operate similarly to the snap layer charms in that they will use a resource if supplied, and if not default to pulling images from a public docker repo.

To be clear, the charms using the snap layer work great. They are completely usable out of the box, and adding an extra configuration option is completely unnecessary. It would just be an extra burden on operators and something else to get wrong at deployment time. The charms correctly handle the resource not existing. If the resource is not available and connectivity fails to the snap store, the unit goes into a blocked state with a suitable status message. I can not attempt the store install first, as checking for the resource first provides more useful options to operators (eg. pinning a snap, or deploying a local build or modification of a snap). The only problem is publishing the charms to the charm store.

My only work around is to require people using the snap layer to create 0 byte files at publication time and pass them through on the command line. This is a poor UI that I don't want to inflict on my users unless I am forced too.

I'm not sure how it will be a problem testing charms with missing expected resources. The charms with missing expected resources are already the majority from what I can tell. They are just using the 0 byte file work around. We don't block release of charms with missing hooks or libraries or any of the other many components that might be missing and cause tests to fail, so it seems unfair to block the common use of optional resources because someone using the less common usage (required) forgot to pass them on the command line when they published. If you do want to catch this less common problem, I think best handled by the developer specifying that the resource is required in metadata.yaml.

[gabriel-samfira]

An example of a resource that cannot be uploaded to the store is Microsoft Sharepoint. Here is the EULA for that particular software:

https://www.microsoft.com/web/webpi/eula/speulas_en.html

Please see "Scope of license". The interesting bit is:

You may not:

publish the software for others to copy;

Which is essentially what happens when you upload it to the store. Operators can upload it to their own controllers though.

[kwmonroe]

Another +1 for optional resources comes from IBM. Their use case centers around the way they update their software. Many of their charms support the concept of fixpacks -- that is, you deploy a charm with an installer resource that installs some GA piece of software. At some point in the future, IBM issues a fixpack to update that version. You attach the fixpack resource and the charm turns the knobs to do the upgrade.

15 of the 20 IBM charms in the review queue today are using 0-byte resources to work around the store's policy of "all resources are mandatory". It's fragile at best:

http://bazaar.launchpad.net/~ibmcharmers/charms/xenial/layer-ibm-was-base/trunk/view/head:/reactive/ibm-was-base.sh#L227

[kwmonroe]

How about adjusting the resources schema to take a required: key. Something like:

resources:
  my_installer:
    type: file
    filename: my_installer.zip
    description: My charm's installer archive
    required: true

If the charm author forgets required, it would default to true, so the following is equivalent to the above:

resources:
  my_installer:
    type: file
    filename: my_installer.zip
    description: My charm's installer archive

And if I, as a charm author, know that a resource is optional for my charm, I can specify it as:

resources:
  my_fixpack:
    type: file
    filename: my_fixpack.zip
    description: My charm's updater (optional).
    required: false

[lazypower]

@mitechie I think there's a consensus here that the majority of heavy/early resource-using charms really want optional resources. The fact we're forced into working around this problem to achieve the desired result, gives me the feeling we've made some assumptions that aren't necessarily correct for our prime-users.

[stub42]

c/majority/all.

I haven't heard of a single charm making use of the feature as intended. Every charm using resources is using this work around. Even the ones the resources feature was designed for.

[johnsca]

I've had use-cases where I'd prefer to use resources as the primary source but can't because of the issue with large resources. Once that is fixed, I think we'll see much more adoption of the feature.

Still, optional resources are definitely a valid use-case and requiring a 0-byte file to work around that is worse for both the users and charm authors than being able to explicitly marking a resource as optional.

[mhilton]

I'm closing this bug as it's the charmstore itself that would need changing not the client. Please see juju/charmstore#810 for further details.