Do you know what is Multimodal Conversation Design? What is a Use Case?

Multimodal design is the science of creating a user interface that includes multiple touchpoints using different modalities.

Let’s walk through a simple example of how multimodal design can improve a user’s experience for something like making a recipe. While it may seem simple on the surface, it involves multiple visual, audio, and touch interactions, sometimes simultaneously.

  1. User verbally asks their smart display for a recipe.
  2. Smart display verbalizes that it can help and provides visual recipe results.
  3. User manually scrolls through options on the smart display, taps for more details, and reads the recipe.
  4. User verbally updates the voice assistant’s shopping list for recipe ingredients.
  5. Smart display verbally and visually confirms the shopping list update.
  6. While driving to the store, the user remembers something else they need, uses an in-car smart assistant to verbally make an addition to their shopping list.
  7. At the grocery store, a user doesn’t want to disturb others, so they use the mobile app’s GUI to read their shopping list, tapping to check-off items as they go.
  8. Back at home, the smart display remembers the selected recipe and both verbally and visually explains the recipe steps.
  9. With their hands busy and messy, the user can verbally ask the smart display to repeat a step, set a timer, play music, etc

Think of the roadblocks this same user would experience through a rigidly audio or visual-only interface. Where might frustration arise without the ability to obtain or convey information within the most context-informed modality?