Multimodal design is the science of creating a user interface that includes multiple touchpoints using different modalities.
Let’s walk through a simple example of how multimodal design can improve a user’s experience for something like making a recipe. While it may seem simple on the surface, it involves multiple visual, audio, and touch interactions, sometimes simultaneously.
- User verbally asks their smart display for a recipe.
- Smart display verbalizes that it can help and provides visual recipe results.
- User manually scrolls through options on the smart display, taps for more details, and reads the recipe.
- User verbally updates the voice assistant’s shopping list for recipe ingredients.
- Smart display verbally and visually confirms the shopping list update.
- While driving to the store, the user remembers something else they need, uses an in-car smart assistant to verbally make an addition to their shopping list.
- At the grocery store, a user doesn’t want to disturb others, so they use the mobile app’s GUI to read their shopping list, tapping to check-off items as they go.
- Back at home, the smart display remembers the selected recipe and both verbally and visually explains the recipe steps.
- With their hands busy and messy, the user can verbally ask the smart display to repeat a step, set a timer, play music, etc
Think of the roadblocks this same user would experience through a rigidly audio or visual-only interface. Where might frustration arise without the ability to obtain or convey information within the most context-informed modality?