Combining Shape-Changing Interfaces and Spatial Augmented Reality Enables Extended Object Appearance

We propose combining shape-changing interfaces and spatial augmented reality for extending the space of appearances and interactions of actuated interfaces. While shape-changing interfaces can dynamically alter the physical appearance of objects, the integration of spatial augmented reality additionally allows for dynamically changing objects' optical appearance with high detail. This way, devices can render currently challenging features such as high frequency texture or fast motion. We frame this combination in the context of computer graphics with analogies to established techniques for increasing the realism of 3D objects such as bump mapping. This extensible framework helps us identify challenges of the two techniques and benefits of their combination. We utilize our prototype shape-changing device enriched with spatial augmented reality through projection mapping to demonstrate the concept. We present a novel mechanical distance-fields algorithm for real-time fitting of mechanically constrained shape-changing devices to arbitrary 3D graphics. Furthermore, we present a technique for increasing effective screen real estate for spatial augmented reality through view-dependent shape change.

@inbook{Lindlbauer16b, 
 author = {Lindlbauer, David and Grønbæk, Jens Emil and Birk, Morten and Halskov, Kim and Alexa, Marc and M"{u}ller, J"{o}rg}, 
 title = {Combining Shape-Changing Interfaces and Spatial Augmented Reality Enables Extended Object Appearance}, 
 year = {2016}, 
 isbn = {9781450333627}, 
 publisher = {Association for Computing Machinery}, 
 address = {New York, NY, USA}, 
 url = {https://doi.org/10.1145/2858036.2858457}, 
 booktitle = {Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems}, 
 pages = {791–802}, 
 numpages = {12} 
 }