Avatars can interact with several ways:

​
Lip-sync

The avatars generated from AvatarLabs support real-time lip-sync based on audio provided to the SDK.

To make your avatar start talking, you need to call playAudioFromSource() or playAudioFromNode() based on the audio type.

function App(){
    const {
        avatarDisplay,
        avatarContext,
        playAudioFromSource,
        playAudioFromNode
    } = useAvatar({
        //...
    });

    const audioContextRef = useRef<AudioContext>(new AudioContext());

    return (
        {avatarDisplay}
        <button
            onClick={() => {
                const arrayBuffer = await fetch("audio.mp3")
                    .then(async (res) => {
                    const file = res.arrayBuffer();
                    return file;
                    })
                    .catch((e) => console.error(e));

                // if your audio is in file format,
                // you need to convert it into array buffer first.

                const audioBuffer =
                    await audioContextRef.current.decodeAudioData(arrayBuffer);
                const bs = audioContextRef.current.createBufferSource();
                bs.buffer = audioBuffer;
                playAudioFromNode(bs, audioContextRef.current);
            }}
        >
            playAudioFromNode
        </button>
        <button
            onClick={() => {
                playAudioFromSource('audio.mp3', audioContextRef.current);
                // both relative path and urls can be used with this method.
            }}
        >
            playAudioFromSource
        </button>
    );
}

You can visit here for a complete demo shocasing how our lip-sync system works!

​
Connect Audio Element

If you want to adjust audio settings (For example, to adjust volume or adding eventlistener), you can create a new HTMLAudioElement and connect it to our sdk through the function connectAudioElement():

const { avatarDisplay, context, connectAudioElement } = useAvatar({
  //...
});

const audioPlayer = useRef(new Audio())

//...

return (
    {avatarDisplay}
    <button
        onClick={() => {
          connectAudioElement(audioPlayer.current, audioContextRef.current);
        }}
    >
        Connect
    </button>

    // then when you play the audioPlayer, the avatar will start lip-sync automatically.

    <button
        onClick={() => {
          audioPlayer.current.src = "audio.mp3";
          audioPlayer.current.play();
        }}
    >
        Play
    </button>
)

You cannot connect audio element after using playAudioFromNode() or playAudioFromSource() to play audio.

​
Expression

Avatars created from our creation suite support different expressions.

How many expression the avatar can make depends on which model you are using to generate the avatar.

const { avatarDisplay, availableEmotions, setEmotion } = useAvatar({
  //...
});
return (
  <button
    onClick={() => {
      setEmotion(availableEmotions[0]);
    }}
  >
    Set emotion
  </button>
);