What I want is to be able to feed in a bunch of videos and generate an animatable (from talking) 3D face from that data. I suppose you in theory only need 3 images (front and sides). But mapping pixels to motion is interesting (facial expressions).
There wouldn't be depth data so it would be inferred from shadows
My case is not directly nefarious, for example an old popular YouTuber that streamed in the early 2000s taking their content and making a model of them for personal use like a 3D chat bot but with that person's quirks
Edit: when I say "nefarious" I mean you can use that tech to impersonate someone (eg. political reason) but for my case it's more the creeper type cloning someone for personal use eg. Replika
Tangent, the holo vtubers industry is interesting since they build up these characters with some unique persona/theme and then people follow that specific model, they could make themselves into an AI easily since it's a rigged 3D asset but of course it would be boring compared to the real thing
Comments
What I want is to be able to feed in a bunch of videos and generate an animatable (from talking) 3D face from that data. I suppose you in theory only need 3 images (front and sides). But mapping pixels to motion is interesting (facial expressions).
There wouldn't be depth data so it would be inferred from shadows
Replicate has Hunyuan video training now. https://replicate.com/blog/fine-tune-video
Also, Kling 1.6 Elements works pretty okay if you use the same person/face for each element.
Kling also has lip sync.
Or this lip sync with replicate: https://replicate.com/bytedance/latentsync
Or there is HeyGen or D-ID or Synthesia, or tavus.io for full interactive digital twins.
Thanks
Why do you want to do that?
My case is not directly nefarious, for example an old popular YouTuber that streamed in the early 2000s taking their content and making a model of them for personal use like a 3D chat bot but with that person's quirks
Edit: when I say "nefarious" I mean you can use that tech to impersonate someone (eg. political reason) but for my case it's more the creeper type cloning someone for personal use eg. Replika
Tangent, the holo vtubers industry is interesting since they build up these characters with some unique persona/theme and then people follow that specific model, they could make themselves into an AI easily since it's a rigged 3D asset but of course it would be boring compared to the real thing
The most popular vtuber on Twitch is an AI tho
You talking NeuroSama? I haven't kept up with it in a bit
I'm not sure if that's truly AI since the Turtle drives her
Edit: if the source was open I'd believe it
It has always been a LLM. There is no human typing at insane speed to the TTS.
I'm referring to live interception of messages which I guess has to be done to be compliant with Twitch's terms -- there is a human there
edit: but yeah the fact that so many people interact with her shows generated content can keep people occupied