Avatars can interact with several ways:
Lip-sync
The avatars generated from AvatarLabs support real-time lip-sync based on audio provided to the SDK.
To make your avatar start talking, you need to create an AudioContext
and connect it to the SDK with connectAudioContext
.
function App ( ) {
const {
avatarDisplay,
connectAudioElement,
connectAudioContext,
connectAudioNode
} = useAvatar ( {
} ) ;
const audioContextRef = useRef< AudioContext > (new AudioContext());
return (
{ avatarDisplay}
< button
onClick= { ( ) => {
connectAudioContext ( audioContextRef. current )
} }
>
Connect AudioContext
</ button>
) ;
}
Connect Audio
After connecting AudioContext, you need to create a HTMLAudioElement or AudioNode and connect it to the SDK in order to sync the audio with lip-sync system.
HTMLAudioElement AudioNode After connection, when you play the audio from the HTMLAudioElement, the avatar will start lip-sync automatically.
const { avatarDisplay, context, connectAudioElement } = useAvatar ( {
} ) ;
const audioPlayer = useRef ( new Audio ( ) )
return (
{ avatarDisplay}
< button
onClick= { ( ) => {
connectAudioElement ( audioPlayer. current , audioContextRef. current ) ;
} }
>
Connect
</ button>
< button
onClick= { ( ) => {
audioPlayer. current . src = "audio.mp3" ;
audioPlayer. current . play ( ) ;
} }
>
Play
</ button>
)
You can visit here for a complete demo shocasing how
our lip-sync system works!
Expression
Avatars created from our creation suite support different expressions.
How many expression the avatar can make depends on which model you are using
to generate the avatar.
const { avatarDisplay, availableEmotions, setEmotion } = useAvatar ( {
} ) ;
return (
< button
onClick= { ( ) => {
setEmotion ( availableEmotions[ 0 ] ) ;
} }
>
Set emotion
</ button>
) ;