CG Agent system for dialogue system and avatar communication

In our lab, we conduct research and development of CG agents (CG avatars) that can be used for both machine dialogue (human-machine) and avatar-mediated conversations (human-human), some of which are available as open data.

Below, we introduce the CG avatars and systems developed by our lab. These are CG-CAs (CG Cybernetic Agents) created with support from the Moonshot Research and Development Program and the Avatar Symbiotic Society Project.

CG-CAs are high-quality 3D models designed for seamless use, whether operated by machines/AI or humans, without creating discomfort. They feature a strong presence and lifelike qualities, with numerous emotion shapes for expressive dialogue, high-precision facial tracking (aka PerfectSync) capabilities, and fully rigged skeletons for full-body movement. These avatars have been utilized in various related projects and demonstration experiments, and some of them are freely available.

Open models are available on GitHub/mmdagent-ex (PMX format) and VRoid Hub (VRM format, non DL).

Gene

Gene

“Gene” ( /dʒénə/ ) is an cartoon-style CG agent with a gender-neutral appearance. This simple CG model, featuring an androgynous look and heroic attire, is designed to appeal to a wide range of users. Its neutral design makes it suitable as an avatar for both male and female users, allowing for broad application in various societal contexts, including public spaces.

Gene’s 3D model is available under CC-BY 4.0 license.

  • PMX format (GitHub): for MMDAgent-EX and other MMD tools
  • VRM format (VRoid Hub): for VRoid-enabled tools

Note that the copyright holder still has trademark and design rights of this model. You are permitted to use its trademark and design for:

  • Academic purpose (publications and releases), and
  • Personal non-commercial usage (posts to SNS or at an event).

For other commercial use, please contact us.

Uka

Uka

Uka is a CG avatar designed primarily for casual and empathetic dialogue tasks. Featuring a neutral design with male-like appearance, Uka is crafted to embody the image of an attentive and engaging listener, capable of thoughtfully hearing out the user. The avatar sports traditional Japanese attire and uses a calm, earth-tone color palette for a serene and approachable expression.

The ears and tail are additional body parts intended to enhance non-verbal emotional expressions, though they can be removed via morph adjustments if desired.

Uka’s 3D model is available under CC-BY 4.0 license.

  • PMX format (GitHub): for MMDAgent-EX and other MMD tools
  • VRM format (VRoid Hub): for VRoid-enabled tools

Nirva

Nirva

Nirva is a CG avatar designed by a famous and promising illustrator Mai Yoneyama. This high-line model, based on an illustration, envisions a futuristic AI device interface and was developed for formal purposes such as presentations at international events and academic conferences.

Nirva is fully designed with numerous emotional expressions and physics-based effects, offering rich and intricate expressiveness. This gives the avatar a strong presence and a sense of vitality during dialogues. Features such as sound indicators on the clothing, animated irises, and gradient changes in the outfit can be linked to events, providing a highly interactive experience.

You can see Nirva in action as an avatar at a demonstration event in the NHK Kansai News broadcast from September 26, 2024.

Magi

Magi

Magi is a character-focused, gender-neutral CG avatar designed for use in digital spaces and the metaverse. Its design of non-human form such as horns and heterochromatic eyes, as is often seen in the internet culture, avoids association with specific real-world cultures or stereotypes, thus making it more accessible and acceptable as a real-stereotype-free virtual avatar for people from diverse cultural backgrounds.

To enhance emotional expression in anime-style models, Magi incorporates symbols often seen in manga (known as manpu), enabling 2D-CG-specific emotional representations that are visually impactful on screen. Combined with emotion-expression AI and emotional speech synthesis, Magi aims to deliver a richer humanoid interface experience.

Rubica

Rubica

“Rubica” is a high-definition 3D CG human-like agent designed to work fine for both an automated spoken dialogue system and a remote avatar system. With its truely neutral appearance that can be interpreted as either male or female and the rich facial and body expressions for conversation, it gives users a real sense of a digital human when used as an automated spoken dialogue system, while the operator’s presence is felt when used as a remote avatar controlled by a human operator. This portable 3D-CG avatar system makes it possible to apply spoken dialogue and remote avatar interaction technology to a wide range of social situations.

MMDAgent-EX

MMDAgent-EX is a platform software for voice dialogue and avatar communication. This software, which runs on Windows, macOS, and Linux, features a custom rendering engine that displays any MikuMikuDance compatible model in high resolution, including the CG avatars mentioned above. It is a tightly integrated system for multimodal processing of voice, language, and motion. MMDAgent-EX can be used in various fields and applications, including research and development of dialogue systems and agent interaction, remote avatar operation, dialogue data collection, and digital signage. This system is open-source software and is available on GitHub (with some features excluded). For more details, please visit the MMDAgent-EX website.

Valles

Valles is a remote control application that connects to MMDAgent-EX to display CG agents and enables remote avatar communication. In addition to supporting facial and body tracking signals, it uses speech-to-motion technology to automatically generate movements from voice, providing an immersive experience as if the avatar is right in front of you. With WebSocket server relays, it allows connections from anywhere in the world, enabling low-latency operation. Valles also features automatic updates, monitoring, and alert functions for digital signage applications, as well as support for image and music playback, enabling rich remote communication.

Remdis

Remdis is a platform for developing text, voice, and multimodal dialogue systems. By combining asynchronous processing, streaming generation of LLMs, turn-taking through Voice Activity Projection (VAP), and MMDAgent-EX, it enables the development of interactive multimodal dialogue systems.

Information

Other tips