Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bake out side of Unity #81

Open
Ray5wang opened this issue Dec 9, 2024 · 1 comment
Open

Bake out side of Unity #81

Ray5wang opened this issue Dec 9, 2024 · 1 comment

Comments

@Ray5wang
Copy link

Ray5wang commented Dec 9, 2024

Is these any way to bake wav file to data with a exteral tool. In my project, audio is played by wwise, so wav files is not in unity. I wanna ask is there any way to bake data out of the unity. I would be gradful for you suggetsions

@hecomi
Copy link
Owner

hecomi commented Dec 31, 2024

I apologize, as my knowledge of Wwise is limited, and I might not be able to provide precise advice. The data format is simple (it only includes phoneme information for each frame), and the goal is to bake this data. However, baking requires running the uLipSync code logic. Specifically, the baking process is performed using code like the following:

public void Bake()
{
data.bakedProfile = data.profile;
data.bakedAudioClip = data.audioClip;
data.frames.Clear();
var clip = data.audioClip;
int samplePerFrame = clip.frequency / 60 * clip.channels;
var buffer = new float[clip.samples * clip.channels];
var tempBuffer = new float[samplePerFrame];
data.duration = clip.length;
var go = new GameObject("uLipSync Baking...");
var ls = go.AddComponent<uLipSync>();
ls.OnBakeStart(data.profile);
clip.GetData(buffer, 0);
for (int offset = 0; offset < buffer.Length - samplePerFrame; offset += samplePerFrame)
{
Array.Copy(buffer, offset, tempBuffer, 0, samplePerFrame);
ls.OnBakeUpdate(tempBuffer, clip.channels);
var frame = new BakedFrame();
frame.volume = ls.result.rawVolume;
frame.phonemes = new List<BakedPhonemeRatio>();
foreach (var kv in ls.result.phonemeRatios)
{
var pr = new BakedPhonemeRatio();
pr.phoneme = kv.Key;
pr.ratio = kv.Value;
frame.phonemes.Add(pr);
}
data.frames.Add(frame);
var progress = (float)offset / clip.samples;
}
ls.OnBakeEnd();
DestroyImmediate(go);
EditorUtility.SetDirty(data);
AssetDatabase.SaveAssets();
}
}

Therefore, it seems that BakedData can be generated either by running the uLipSync code logic externally or by feeding the audio buffer obtained through Wwise into this logic. If it is possible to retrieve the audio buffer from Wwise via some API, replacing AudioClip.GetData() with the appropriate method should allow it to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants