Document OverConstrainedError for PTZ in explainer

[beaufortfrancois]

As discussed at w3c/mediacapture-main#697, the resolution to the fingerprinting/capabilities query in getUserMedia() in general seems to be to keep things as they are. It means that pan/tilt/zoom properties will be three more properties among the many that getUserMedia() already exposes.

In other words, getUserMedia() will reject with an OverConstrainedError if camera does not support PTZ and PTZ constraints are required without user prompt.

I believe we should document this in the upcoming privacy section of the PTZ explainer.

@riju @reillyeon @eehakkin @andypaicu @guidou

[youennf]

I did not follow PTZ discussions, so I might lack some background here.
Adding more mandatory constraints is adding fingerprinting surface so I would hope we do not go there.

In general, I think we want to move away from mandatory constraints.
Can we define these new constraints so that they are only treated as ideal?

[youennf]

This issue is probably worth some 'privacy' label

[beaufortfrancois]

Can we define these new constraints so that they are only treated as ideal?

@youennf Do we have examples of constraints that are only treated as ideal?

[youennf]

Adding the necessary hooks in media capture main is possible.
Before we start discussing it there, I would like to validate that restricting this constraint to 'ideal' is good enough for PTZ cameras. Is that correct?

[youennf]

At least ideal constraints for getUserMedia, applyConstraints might be another discussion

[beaufortfrancois]

Restricting pan, tilt, and zoom constraints in getUserMedia to be "ideal" would work.
In this case, the PTZ capability request would be optional, not mandatory.

// [NEW] Does not fail with OverConstrainedError if no PTZ camera. 
await navigator.mediaDevices.getUserMedia({ video: { pan: true } });

// [NEW] Both would be equivalent. 
await navigator.mediaDevices.getUserMedia({ video: { pan: true } });
await navigator.mediaDevices.getUserMedia({ video: { advanced: [{ pan: true }] } });

It seems like a bigger change than PTZ according to w3c/mediacapture-main#697

[beaufortfrancois]

For info, #231 documents the current OverConstrainedError in the PTZ explainer.

[beaufortfrancois]

@youennf What would be the next steps?

[youennf]

It seems there is consensus to make the PTZ constraints be ideal only.
The easiest path is to neuter any PTZ required constraint.
getUserMedia step 6.3.3 is cleaning up the constraints given to getUserMedia before proceeding with selecting the settings.

We could add a step in getUserMedia algorithm that would further remove some required constraints, for instance any required constraint whose name is not in a well defined list. This list could of course be extended by specifications defining new constraints. PTZ constraints would not be in that list.

[beaufortfrancois]

Thank you @youennf! Much appreciated.

Once mediacapture-main changes land, I'll update mediacapture-image spec to mention pan, tilt, and zoom are not in this list.

FYI @guidou

[youennf]

Discussing at the webrtc editor meeting, we might need to further discuss this at next webrtc interim.
@jan-ivar brought up that if the PTZ specific prompt happens before PTZ constraints are checked to select devices, required constraints might be fine. The image capture spec is not super clear about this.

[jan-ivar]

// [NEW] Does not fail with OverConstrainedError if no PTZ camera. 
await navigator.mediaDevices.getUserMedia({ video: { pan: true } });

Note the above would never fail with OverConstrainedError because constraints are non-required by default. To fail requires:

await navigator.mediaDevices.getUserMedia({video: {pan: {exact: 0}}});

The latter would indeed be bad for privacy if it didn't prompt first (which the spec currently says to do the way I read it.)

But prompting users for permission only to fail afterwards seems like bad UX, so #246 seems like the right direction.

[youennf]

Also, I am not sure it is a great idea to allow to set exact PTZ values as part of getUserMedia.
If a page does: await navigator.mediaDevices.getUserMedia({video: {pan: {exact: 100}}});
User will have a prompt, user will grant the prompt and just after the prompt, the camera will move to the new position.
If the User Agent showed a preview of what the camera is capturing, it would be the preview before applying the constraints, not after. This might end up be a bad surprise for the user.

All examples in the spec and explainer are getUserMedia with boolean as pan/tilt/zoom constraint values.
Should we tighten the model so that only {pan: true} is actually considered?

Also, is there a case where we would want { pan: true, zoom: false }?
Can we simplify this and allow a single constraint for getUserMedia: { panTiltZoom: true }?

[jan-ivar]

Also, I am not sure it is a great idea to allow to set exact PTZ values as part of getUserMedia.

(That's not a feature unique to exact values, but assuming you mean specific values) Probably not, which I believe is why we added the convenient true alias for {} and use that in all examples.

But I see no reason to outlaw it either, since the following accomplishes the same:

const [track] = (await navigator.mediaDevices.getUserMedia({video: {pan: true}})).getVideoTracks();
await track.applyConstraints({video: {pan: 100}});

In any case, the user was arguably warned when they allowed the site to "Use and move the camera".

[youennf]

Supporting this pattern adds some complexity to the specification and to implementations.
If it is a bad pattern, why adding this complexity?

There is also the edge case of getUserMedia failing to applyConstraints while user granted device: the camera moved even though the web page never received a MediaStreamTrack. This seems slightly annoying.

Also, the following example would most probably succeed, which somehow shows how strange this API is:

const [track] = (await navigator.mediaDevices.getUserMedia({video: {tilt: true}})).getVideoTracks();
await track.applyConstraints({video: {pan: 100}});

[jan-ivar]

Supporting this pattern adds some complexity to the specification and to implementations.
If it is a bad pattern, why adding this complexity?

On the contrary, I think special casing this is what would add complexity. Right now it's the same algorithm.

Good or bad, I think it's important that behavior is predictable. I also don't think there's necessarily anything bad with setting pan, tilt and zoom immediately. That's an app choice, much like a car adjusting seat & steeringwheel when a driver gets in.

There is also the edge case of getUserMedia failing to applyConstraints while user granted device: the camera moved even though the web page never received a MediaStreamTrack. This seems slightly annoying.

That would be an implemention bug I think, since applying constraints is step 6.6 in the getUserMedia algorithm, and is done "In a single operation" once there's nothing left that can fail.

[youennf]

Good or bad, I think it's important that behavior is predictable.

Well, it is very predictable if the web page cannot express that behavior, except as part of applyConstraints :)
I could see User Agents wanting to reset PTZ to the initial position, for instance in case the PTZ position was previously set by another origin.

Also, navigator.mediaDevices.getUserMedia({ video : { width : 481 } }) is still not very interoperable between browsers.
navigator.mediaDevices.getUserMedia({video: { pan: 10, tilt: true }})) might follow the same path.

Things are a bit simpler with applyConstraints since it starts with a device initial state that is exposed to the web page.
The initial state of the camera in getUserMedia algorithm is somehow User Agent specific.

"In a single operation"

step 6.6 is potentially calling the applyConstraints constraints twice for audio and video tracks, so "In a single operation" is not always guaranteed. step 6.6 is also handling the case of an error. If it is guaranteed to never happen, we should remove it from the spec.

[jan-ivar]

What would be unpredictable would be web devs needing to refer to a list over which constraints can be set from getUserMedia or not. It also seems rather pointless since they can just call applyConstraints right after for the same effect.

Things are a bit simpler with applyConstraints since it starts with a device initial state that is exposed to the web page.

Does it? Isn't that only if people make the effort to call getSettings explicitly and take those into account when specifying their constraints somehow? E.g.:

await track.applyConstraints(buildMyConstraintsFromSettings(track.getSettings()));

I don't think that's been common, until this spec at least where you're kinda forced to do that initially since the units of pan, tilt and zoom are unknown. However that doesn't preclude restoring settings from last visit, which should be good for the same deviceId at least.

step 6.6 is potentially calling the applyConstraints constraints twice for audio and video tracks, so "In a single operation" is not always guaranteed. ... [because] ... step 6.6 is also handling the case of an error.

It also says about these steps: "For each media type kind ... run the following sub steps, preferably at the same time"

So it seems possible to implement this where all constraints failures from step 6.6. are dealt with before pan, tilt and zoom move the camera, or by simply doing audio first.

[beaufortfrancois]

Unless there's a strong argument for using a brand new boolean member panTiltZoom in getUserMedia that would be used only for detecting camera PTZ and grant this ability, I'd go with the true syntactic sugar.

[youennf]

Unless there's a strong argument for using a brand new boolean member panTiltZoom in getUserMedia

Some arguments have been described above, some more below.

The current algorithm makes it so that getUserMedia({ pan : true } is different from getUserMedia({ pan : 1 }).
Ditto for getUserMedia({ pan : false } vs. getUserMedia({ pan : 0 }). This is not intuitive at all.

PTZ constraints are currently numerical constraints. This entails selecting devices based on a distance.
If a setup has several PTZ cameras, say PTZ1 and PTZ2, the selection of the device will be based on this distance.
Say a page will do getUserMedia({ pan : 100}), the page may sometimes get PTZ1 or PTZ2 depending on PTZ1 and PTZ2 positions. Even though page does not get access to PTZ2, the fact PTZ1 is selected might give information on PTZ2 position.

The current model is adding complexity to web browser implementations.
The current behavior could be potentially exploited (though prompts will probably help) showing a weak design.
I haven't seen any use case motivating this complexity, all examples in the spec and primer are solely using boolean constraints.
This suggests PTZ numerical constraints for getUserMedia is not a great idea.

[beaufortfrancois]

I haven't seen any use case motivating this complexity, all examples in the spec and primer are solely using boolean constraints.

This is not true ;)
See the explainer example below on how to reset pan.

Note that this example though may be better with a device id.

const videoStream = await navigator.mediaDevices.getUserMedia({
  // Website asks to reset known camera pan.
  video: { pan: 1, deviceId: { exact: "myCameraDeviceId" } }
});

However, as you said, this could be done with gUM and applyConstraints.