Avatars can interact with several ways:

Lip-sync

The avatars generated from AvatarLabs support real-time lip-sync based on audio provided to the SDK.

To make your avatar start talking, you need to create an AudioContext and connect it to the SDK with connectAudioContext.

function App(){
    const {
        avatarDisplay,
        connectAudioElement,
        connectAudioContext,
        connectAudioNode
    } = useAvatar({
        //...
    });

    const audioContextRef = useRef<AudioContext>(new AudioContext());

    return (
        {avatarDisplay}
        <button
            onClick={() => {
              connectAudioContext(audioContextRef.current)
            }}
        >
            Connect AudioContext
        </button>
    );
}

Connect Audio

After connecting AudioContext, you need to create a HTMLAudioElement or AudioNode and connect it to the SDK in order to sync the audio with lip-sync system.

  • HTMLAudioElement

  • AudioNode

After connection, when you play the audio from the HTMLAudioElement, the avatar will start lip-sync automatically.

const { avatarDisplay, context, connectAudioElement } = useAvatar({
  //...
});

const audioPlayer = useRef(new Audio())

//...

return (
    {avatarDisplay}
    <button
        onClick={() => {
          connectAudioElement(audioPlayer.current, audioContextRef.current);
        }}
    >
        Connect 
    </button>

    <button
        onClick={() => {
          audioPlayer.current.src = "audio.mp3";
          audioPlayer.current.play();
        }}
    >
        Play
    </button>
)

You can visit here for a complete demo shocasing how our lip-sync system works!

Expression

Avatars created from our creation suite support different expressions.

How many expression the avatar can make depends on which model you are using to generate the avatar.

const { avatarDisplay, availableEmotions, setEmotion } = useAvatar({
  //...
});
return (
  <button
    onClick={() => {
      setEmotion(availableEmotions[0]);
    }}
  >
    Set emotion
  </button>
);