threejs example not working on iOS?

[mikaelwozniak]

safari/chrome on iPhone iOS 14.5.

the lips are moving but there is no sound.

(also, note the threejs version has to be specified in the import, for example below:)

<script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r125/three.js"></script>

Is it related to webAudio context? Any workaround? Thanks in advance.

[mikaelwozniak]

WARNING WARNING: Multiple instances of Three.js being imported.
WARNING The audio context is not running. Speech will not be able to be played until it is resumed. Use the "TextToSpeechFeature.resumeAudio" method to try to resume it after a user gesture.

funny enough, the audioListener.context.state is running, not suspended, it should be fine...

[ranilian]

Ditto - this also happens on BabylonJS. It fails to playback any host audio on iOS nor on Safari on Mac.

[lbracegirdle]

I have the exact same issue isolated to Safari on Mac and iOS devices. I found it was related to Safari's AutoPlay policy settings. You can't ask the user to set Allow All AutoPlay of course, but it did help me find a partial solution in older posts like this.

In my three.js project, just using simple javascript, I created a a function attached to the click/tap event handler on a button the user clicks/taps before requesting audio from the host. Future requests for the AWS host to speak, were allowed.

function playMessage() {
   const audio = new Audio("start.wav"); //this was just a silent audio file of 2 second duration.
   const source = audioContext.createMediaElementSource(audio);
   source.connect(audioContext.destination);
   audio.play();
// Then the AWS host's speech plays on iOS and Mac Safari browser
    const { name, host } = getCurrentHost(speakers); //speakers is a reference to the AWS host character
    host.TextToSpeechFeature['play'](message); //message is a string - the text to send to the AWS host to speak.
}

My hopes were dashed after testing further. I noticed if the user delays between clicking and requesting more speech from the host, the audio starts to crackle and become distorted, as if Safari has detected you are pulling a fast one on it!

I post my partial findings in case this helps find a proper solution. This is a great open source project that is hampered with the Safari autoplay policy limitation I think. I hope the dev team on this can solve it better than I can.

[crowers]

Hey @lbracegirdle @mikaelwozniak @ranilian @hyandell Did you find a solution for this? we can't get the audio to play on iOS in Chrome or Safari. We see in the Chrome console using chrome://inspect "the audio context is not running". We added a mute / play button to activate the audio context and replay the message, but still silence. Spent loads of time looking for an answer, so any suggestions would be much appreciated.

[ranilian]

Hey @crowers,

I did manage to fix the issue by changing the underlying method that audio is loaded which bypasses the iOS restriction. Note this is just my approach and is not an official answer. It may or may not work for you.
My fix is to change the streaming flag to false here:

amazon-sumerian-hosts/src/Babylon.js/awspack/TextToSpeechFeature.js

Line 84 in 9984e36

{streaming: true, skipCodecCheck: true}

.

[GeneUNCG]

Hey @crowers,

I did manage to fix the issue by changing the underlying method that audio is loaded which bypasses the iOS restriction. Note this is just my approach and is not an official answer. It may or may not work for you. My fix is to change the streaming flag to false here:

amazon-sumerian-hosts/src/Babylon.js/awspack/TextToSpeechFeature.js

Line 84 in 9984e36

{streaming: true, skipCodecCheck: true}

.

Just wanted to update this issue: This quoted fix was working for over a year, but no longer works with the latest iOS. If anyone can find another way to auto play on iOS for Babylon.js, that would be extremely useful.

I created a start button that plays audio on click, but once it finishes and new audio is called once the first speech is completed, that new audio is mute.