Augmented Perception Lab

Simulating Human Audiovisual Search Behavior

Hyunsung Cho, Xuejing Luo, Byungjoo Lee, David Lindlbauer, Antti Oulasvirta.
Published at ACM CHI 2026
Teaser image

Abstract

Locating a target based on auditory and visual cues---such as finding a car in a crowded parking lot or identifying a speaker in a virtual meeting---requires balancing effort, time, and accuracy under uncertainty. Existing models of audiovisual search often treat perception and action in isolation, overlooking how people adaptively coordinate movement and sensory strategies. We present Sensonaut, a computational model of embodied audiovisual search. The core assumption is that people deploy their body and sensory systems in ways they believe will most efficiently improve their chances of locating a target, trading off time and effort under perceptual constraints. Our model formulates this as a resource-rational decision-making problem under partial observability. We validate the model against newly collected human data, showing that it reproduces both adaptive scaling of search time and effort under task complexity, occlusion, and distraction, and characteristic human errors. Our simulation of human-like resource-rational search informs the design of audiovisual interfaces that minimize search cost and cognitive load.
Sensonaut thumbnail

Simulating Embodied Search

Sensonaut simulates how humans search for audiovisual targets by actively turning and moving in space, enabling realistic embodied search behavior.

To locate the highlighted target, humans and Sensonaut agents choose actions based on the evidence gathered through visual and auditory perception. In the top-down view, the pink and green stars indicate each agent's estimate of the target location.

3rd-person view 1st-person view Top-down view

Sensonaut replicates embodied search, taking actions like turning and moving to gather audio and visual evidence before committing to a target.

Humans can make search errors when the available evidence points to a plausible estimate, so they do not spend enough effort turning or moving to check further. Sensonaut simulates these human-like failure modes. In the top-down view, the pink and green stars indicate each agent's estimate of the target location.

3rd-person view 1st-person view Top-down view

In this case, another black car occludes the true target, so both the human and Sensonaut settle on the wrong estimate.

Applications

Audiovisual search is everywhere: finding a phone, your car in a parking lot, a person in a crowd. And it is increasingly central to HCI — in XR, in drone interaction, and in human-robot collaboration with distributed audiovisual sources. If we want to design and evaluate assistive systems for these tasks, today we still mostly rely on human-subject studies.

Applications of embodied audiovisual search in HCI
A model of audiovisual search enables:
  • Making verifiable predictions about embodied interaction
    • Time: How long will search take?
    • Physical effort: How much will it require users to move?
    • Search error: Will users succeed? What are potential sources of error?
  • Explaining why people behave that way

By modeling human-like search behavior, Sensonaut can inform the design of audiovisual interfaces that better support search, reduce effort, and avoid costly failures.

Additional applications of embodied audiovisual search in HCI

Code & Dataset

Source code and dataset available at https://github.com/choch-o/Sensonaut

Materials

Bibtex

@inproceedings {Cho2026Sensonaut, 
 author = {Cho, Hyunsung and Luo, Xuejing and Lee, Byungjoo and Lindlbauer, David and Oulasvirta, Antti}, 
 title = {Simulating Human Audiovisual Search Behavior}, 
 year = {2026}, 
 publisher = {Association for Computing Machinery}, 
 address = {New York, NY, USA}, 
 doi = {10.1145/3772318.3790614},keywords = {Computational behavior modeling; user simulation; multimodal perception; computational rationality; reinforcement learning}, 
 location = {Barcelona, Spain}, 
 series = {CHI '26} 
 }